A Common Pitfall in Containerized Applications: Process Management

Ever had that moment when you're reviewing container logs of your stopped pod's container in Kubernetes and noticed something odd? Your application isn't shutting down gracefully, even though you're absolutely sure you implemented graceful shutdown in your code. You've tested it locally, it works perfectly fine, but somehow in the container... it just gets killed abruptly.

When we containerize applications, we often oversimplify process management. We think, "one container, one process" – but reality isn't that simple. Sometimes we should be able to spawn child processes, handle signals, manage resources, and need proper cleanup using the application itself.

Understanding Process Hierarchy

Linux Process Basics

Before diving into the problem, let's refresh our memory about processes in Linux. A process is just a running instance of a program. When you start a program, the operating system creates a process and assigns it a unique Process ID (PID). Each process has a parent process (except for the init process, PID 1). This creates a hierarchy.

Here's a typical process tree:

$ pstree -p
systemd(1)─┬─systemd-journal(473)
           ├─systemd-udevd(503)
           ├─sshd(1285)─┬─sshd(5279)───bash(5280)───pstree(6093)
           └─nginx(1340)─┬─nginx(1341)
                        ├─nginx(1342)
                        └─nginx(1343)

See how everything stems from PID 1 (systemd in this case)? That's the init process, and it's incredibly important. Here's why:

• It's the first userspace process started by the kernel
• It's responsible for starting and supervising other system processes
• It handles orphaned processes (adopts them when their parent dies)
• It must reap zombie processes to prevent resource leaks
• It's responsible for proper signal handling and propagation

The Container Process Model

Now here's where it gets interesting.

In a standard Linux system, systemd (or another init system) starts as PID 1 and manages all other processes. But containers work differently: the process specified in your ENTRYPOINT (or  CMD if ENTRYPOINT isn't specified) becomes the container's init process (PID 1).

This is a crucial difference because now your entrypoint process inherits all the responsibilities of an init system, whether it's designed for this role or not.

Let's look at different ways to start an application and what becomes PID 1:

# Case 1: Shell application becomes the init process which it's not designed for
ENTRYPOINT ["go", "run", "main.go"]
# PID 1 = /bin/sh  <- This is incorrect

# Case 2: Direct application executable becomes the init process
RUN go build -o myapp
ENTRYPOINT ["./myapp"]
# PID 1 = ./myapp

# Case 3: Using shell form (don't do this)
RUN go build -o myapp
ENTRYPOINT ./myapp
# PID 1 = /bin/sh -c "./myapp"

Signal Handling Analysis

Let's analyze what happens in each case after running docker stop:

Case 1 (ENTRYPOINT ["go", "run", "main.go"]):

• The shell becomes PID 1 and your Go application runs as a child process
• When docker stop sends SIGTERM, it goes to the shell (PID 1)
• The shell doesn't forward signals to child processes by default
• Your application never receives the signal to shut down gracefully
• After the grace period(default: 10 seconds in Docker), SIGKILL forcefully terminates everything

Case 2 (ENTRYPOINT ["./myapp"]):

• Your Go application is PID 1
• It directly receives SIGTERM when docker stop is called
• If your application handles SIGTERM properly, it can:
  • Stop accepting new requests
  • Complete in-flight requests
  • Clean up resources
  • Exit gracefully
• If your application doesn't handle SIGTERM, it'll be killed after the grace period

Case 3 (ENTRYPOINT ./myapp):

• Same issues as Case 1, with additional shell interpretation overhead

Why this matters?

Besides graceful shutdowns, proper process management is crucial for several reasons. Let's examine a common scenario that illustrates these challenges:

Imagine you're running a web application that processes uploaded files:

1. User uploads a large file
2. Main application (PID 1) spawns a process to handle the conversion
3. The conversion process creates temporary files and spawns additional processes (like ImageMagick for image processing)
4. If the user cancels the upload or the operation times out:
   • Main application tries to kill the conversion process
   • But the child processes (ImageMagick) become orphaned
   • These orphaned processes continue running
   • When they eventually finish, they become zombies
   • The main application, acting as PID 1, doesn't know how to reap these zombies
   • System resources are gradually consumed by accumulated zombie processes

This isn't just theoretical. Here's a demonstration of a simple C program that'll show you the child process being adopted by the init process and continue to consume resources.

// ./main.c
#include <stdio.h>
#include <unistd.h>

int main() {
    pid_t pid = fork();
    
    if (pid == 0) {
        // Child process
        printf("Child PID: %d, Parent PID: %d\n", getpid(), getppid());
    } else {
        // Parent process
        printf("Parent PID: %d, Child PID: %d\n", getpid(), pid);
    }
    return 0;
}

When we run this:

./main
Parent PID: 12645, Child PID: 12646
Child PID: 12646, Parent PID: 1

The child process gets orphaned and adopted by PID 1, potentially leading to resource leaks if not properly managed.

Solution to the Problem

The most common and best-practice solution is to use a lightweight init system designed specifically for containers. Here are the two most popular options:

Tini

Tini is a tiny but valid init process specifically designed for containers. It's so reliable that Docker has integrated it as the default init process when you use the --init flag.

Dumb-init

Dumb-init is another lightweight process supervisor and init system designed by Yelp. It's similar to Tini but with some additional features.

Conclusion

Process management in containers extends far beyond the basic "one container, one process" paradigm. These aren't just implementation details - they're fundamental to building production-ready containerized applications.

When we invest time in understanding and implementing proper process management, we prevent subtle but critical issues: memory leaks from zombie processes, incomplete cleanup during container termination, and broken graceful shutdown mechanisms.

In modern cloud environments, where applications need to scale efficiently and handle unexpected terminations gracefully, proper process management isn't optional - it's essential for building truly production-grade containerized systems.

Happy containerizing! 🐋