Kubernetes Exit Code 127: What It Means and How to Fix It

When running workloads in Kubernetes, containers may occasionally fail within specific exit codes. These codes are essential to understand why a container crashed or failed to start. One of the most errors that a developer encounters is Exit Code 127. This code usually indicates a command not found issues inside a container, which is often caused by a missing binary, misconfigured Dockerfile, or incorrect volume mount. 

In this guide, we will break down what Exit Code 127 in Kubernetes is about, explore common causes, and show how to troubleshoot it effectively. 

Understanding Linux Exit Codes in Kubernetes

Kubernetes relies on standard Linux exit codes to report why a container has stopped. These exit codes are passed from the container runtime, such as Docker or containerd to the Kubernetes control plane, which helps operators identify and resolve issues. 

• Exit Code 0: The process completed successfully.
• Exit Code > 0: An error occurred. The number indicates the type of error.

Exit Code 127 specifically means:

Command not found — the shell or entrypoint tried to run a command that doesn’t exist in the container’s file system or environment.

Understanding the behavior is crucial for diagnosing container startup issues to ensure that your Kubernetes workloads run easily. 

Common Causes of Exit Code 127

Exit Code 127 in Kubernetes is mostly to point out an issue where a container is trying to run a command that cannot be found. Here are some of the most common reasons why this might happen in a Kubernetes environment. 

Missing Executable or Command

The most common cause for Exit code 127 is that the “specified command does not exist” inside the container image. This often happens when: 

• The command is not installed in the image. 
• A script is calling another tool or binary that is not yet included. 
• The PATH environment variable is not included in the command location. 

Example:

If your container tries to run the command “python” but “python3” is installed, the container will instantly fail with the Exit Code 127.

Fix:

Ensure that all the required commands and dependencies are properly installed in the image and verify that the command name matches the available ones in the system. 

Misconfigured Dockerfile or EntryPoint

Errors in the Dockerfile, particularly around the ENTRYPOINT or CMD instructions, which can lead to Exit Code 127. 

Related Article: Breaking Down Docker vs Terraform: How to Pick The Right DevOps Tool?

Common issues include:

• Specifying a shell style command without the sh -c or using incorrect syntax. 
• Using an ENTRYPOINT that generally refers to a non-existent script or binary. 
• Relying on build-time tools that are not present in the final image stage in multi-stage builds. 

Fix:

Double check the Dockerfile to make sure that the ENTRYPOINT and CMD that you specified are referenced are valid and existing. When in doubt, test the container that is locally using docker run before deploying it to the Kubernetes. 

Incorrect Volume Mounts

In Kubernetes, volume mounts can sometimes override the paths inside the container. If a volume is mounted over a directory that contains the essential binaries or scripts, the original content might be hidden, leading to the Exit Code 127. 

Example:

You mount a volume at the /app but the image’s ENTRYPOINT references a script inside the /app/start.sh. If the mounted volume does not include the start.sh, the container will fail. 

Fix:

Ensure that your volume mounts do not unintentionally overwrite critical directories or files needed to start the container. Validate that mounted volumes contain the expected content.

How to Troubleshoot Exit Code 127

When you come across Exit Code 127 in Kubernetes, the goal is to figure out what command is missing or misconfigured. Here is a structured approach to debugging the issue. 

Check the Container Logs

Start by inspecting the logs of the failed pod to check out the error code message was before exiting the container: 

kubectl logs <pod-name> –previous

Look for messages like:

/bin/sh: <command>: not found

This tells you exactly which command the container tried to execute and couldn’t find.

Inspect the Dockerfile of Image

Next, review the Dockerfile that is used to build the image: 

• Is the command you’re trying to run included in the image?
• Does the ENTRYPOINT or CMD refer to a file that exists and is executable?
• If you’re using a multi-stage build, is the final stage copying over the necessary binaries?

You can also inspect the image locally using Docker:

docker run -it –entrypoint /bin/sh <your-image>

Once you are inside the container, try running the commands manually. 

Verify File Paths and Commands

Sometimes the issue is not in the absence of the command, but in the form of an incorrect path or missing variables. For example: 

• Referencing ./start.sh when the working directory isn’t set.
• Expecting binaries in /usr/bin that are actually in /usr/local/bin.
• Typos in script or command names.

Checklist:

• Double-check command names.
• Confirm paths are absolute or correctly relative.
• Ensure scripts have executable permissions (chmod +x).

Use Debugging Pods or BusyBox

If the issue is not clear from the logs or Docker inspection, you can use a temporary debug pod with tools like BusyBox or Alpine to explore the container environment. 

Example using BusyBox:

kubectl run debug –rm -it –image=busybox — bash

Best Practices to Prevent Exit Code 127

Best Practice	Description
Use Explicit File Paths	Avoid relying on $PATH resolution by specifying full paths to scripts or binaries in your Dockerfile and Kubernetes YAML.
Validate Dockerfile Locally	Before pushing to a registry, run and test the container locally using docker run to catch errors early.
Ensure Commands Are Installed	Confirm all dependencies, tools, and commands are installed in the image, especially in minimal base images like Alpine or Debian-slim.
Use Executable Permissions	Scripts and binaries should have executable permissions (chmod +x script.sh) to avoid command not found errors.
Avoid Overwriting Critical Paths with Volumes	Be careful not to mount volumes over directories that contain important scripts or binaries inside the container.
Leverage Liveness/Readiness Probes Carefully	Ensure your probes call valid scripts or binaries that are present in the final container image.
Log Early in Startup Scripts	Print debug info at the start of your entrypoint or shell scripts to catch missing paths or misconfigurations quickly.

Conclusion: Resolve and Avoid Exit Code 127 in Your Workloads

Exit Code 127 in Kubernetes is one of the most common issues that you come across. This is mostly due to missing commands or variables. While it can be frustrating, it is definitely one of the easiest to resolve! 

FAQs