Skip to main content

Graceful Shutdown in Backend Systems

Real-World Problem Scenario​

  • Example:

    • Server restarts during a payment transaction

  • Risks:

    • Transaction loss
    • Double charging
    • Data inconsistency

Goal​

  • Ensure:

    • No data corruption
    • Smooth user experience
    • Safe deployment transitions

What is Graceful Shutdown?​

  • Graceful shutdown = controlled stopping of a server

Definition​

  • Instead of stopping abruptly:

    • Finish ongoing work
    • Clean resources
    • Exit safely

Analogy​

  • Like closing a restaurant:

    • Stop new customers
    • Let existing ones finish
    • Clean up
    • Close

Why It Matters​

  • Prevent:

    • Data corruption
    • Incomplete requests
    • System inconsistencies

  • Improves:

    • Reliability
    • User experience

Process Lifecycle Management

What is a Process?​

  • Backend runs as a process in OS

  • Every process has:

    • Start
    • Execution
    • Termination

Lifecycle Stages​

  • Start → Running → Termination

OS Communication with Process​

  • OS does NOT kill process immediately
  • Uses signals to communicate

Signals (Unix Systems)

Definition​

  • Signals = messages between OS and process

  • Used for:

    • Inter-Process Communication (IPC)

Signal Handlers​

  • Backend registers handlers:

    • Wait for signals
    • Execute shutdown logic

Types of Signals

1. SIGTERM (Graceful Signal)​

  • Meaning: Request to terminate politely

Behavior​

  • OS says:

    • “Finish your work and stop”

What App Does​

  • Finish ongoing requests
  • Clean resources
  • Exit

Used By​

  • Deployment systems
  • Kubernetes
  • Process managers (PM2, systemd)

2. SIGINT (Interrupt Signal)​

  • Triggered by:

    • Ctrl + C

Behavior​

  • User-initiated shutdown

Key Point​

  • Should be handled same as SIGTERM

3. SIGKILL (Force Kill)​

  • Meaning: Immediate termination

Behavior​

  • Cannot be:

    • Caught
    • Ignored

  • Process:

    • Stops instantly

Analogy​

  • SIGTERM → polite request
  • SIGINT → user interruption
  • SIGKILL → pulling power plug

Key Insight​

  • If you ignore graceful signals:

    • System eventually uses SIGKILL

Steps in Graceful Shutdown

1. Finish Existing Requests (Connection Draining)​

Definition​

  • Handle in-flight requests before stopping

Steps​

  1. Stop accepting new requests
  2. Allow ongoing requests to finish
  3. Close connections

Analogy​

  • Restaurant:

    • Stop new customers
    • Let current customers finish

Applies To​

  • HTTP servers → finish API requests
  • Databases → complete transactions
  • WebSockets → notify before closing

Timeout Mechanism​

  • Cannot wait indefinitely

Typical Values​

  • 30–60 seconds

Trade-off​

  • Too short → interrupt requests
  • Too long → slow deployments

2. Resource Cleanup​

Definition​

  • Release all system resources

Types of Resources​

  • File handles
  • Network connections
  • Database connections
  • Cache / temp files

Why Important​

  • Prevent:

    • Memory leaks
    • Connection exhaustion
    • Deadlocks

Database Example​

  • Must:

    • Commit OR rollback transactions

Network Example​

  • Close open connections properly

Cleanup Order Rule​

  • Reverse order of acquisition

Why?​

  • Avoid dependency issues

Graceful Shutdown Flow (Summary)

  1. Receive signal (SIGTERM / SIGINT)
  2. Stop accepting new requests
  3. Complete in-flight requests
  4. Clean up resources
  5. Exit process

Practical Backend Example (Conceptual)

Steps Performed​

  • Register signal handler

  • On signal:

    • Shutdown HTTP server
    • Close DB connections
    • Stop background jobs (e.g., Redis workers)

Internal Behavior​

  • HTTP server:

    • Stops new requests
    • Finishes current ones

  • DB:

    • Closes TCP connections

  • Background jobs:

    • Finish queued tasks

Key Concepts to Remember

Connection Draining​

  • Stop new traffic
  • Finish existing work

Resource Cleanup​

  • Release everything properly

Signals​

  • SIGTERM / SIGINT → graceful
  • SIGKILL → forceful

Design Considerations

  • Choose proper timeout

  • Handle all signals consistently

  • Coordinate with:

    • Load balancers
    • Service discovery

Final Takeaways

  • Graceful shutdown is critical in production systems

  • Ensures:

    • Data safety
    • System stability
    • Smooth deployments

Mental Model​

  • Don’t “kill” your server
  • Teach it to exit politely

➡️ Finish work → Clean up → Exit