File: Dockerfile

...
HEALTHCHECK --interval=2h --timeout=8m --retries=2 CMD ./docker-healthcheck || exit 1
ENTRYPOINT ["./docker-entrypoint"]

What are Healthchecks?

Docker healthchecks provide docker image authors a way to define their own evaluation of the health status of their containers that goes beyond merely checking for an exit code in the entrypoint command.

From the Docker docs:

The HEALTHCHECK instruction tells Docker how to test a container to check that it is still working. This can detect cases such as a web server that is stuck in an infinite loop and unable to handle new connections, even though the server process is still running.

Seems like a useful feature, right?

What about logging?

The default behavior for logging health check output is inadequate for production use cases. Again from the official docs:

To help debug failing probes, any output text (UTF-8 encoded) that the command writes on stdout or stderr will be stored in the health status and can be queried with docker inspect. Such output should be kept short (only the first 4096 bytes are stored currently).

I do think the docker devs were right to not try to kludge the healthcheck stdout and stderr into the main process stdout and stderr, but it does mean a bit of extra boilerplate for anyone who wants to see more details on why a healthcheck might have failed.

The Rsyslog Approach

Install rsyslog in your image and then forward the output of the health check to the host machine’s syslog stream.

You’ll need to configure your docker container to have network access to the host. To do this, I set the container networking mode to host.

Here’s what my healthcheck implementation looks like:

File: Dockerfile

...
RUN apt-get install -y rsyslog
ADD ./forward_syslog.conf /etc/rsyslog.d/forward_syslog.conf
...
HEALTHCHECK --interval=2h --timeout=8m --retries=2 CMD ./docker-healthcheck || exit 1
...

Note: I default to an error exit code, just in case my healthcheck script is missing or can’t be run.

File: docker-healthcheck

#!/bin/bash

exec nodejs healthcheck.js | logger -t my-app ; test ${PIPESTATUS[0]} -eq 0

And then you can see in the check itself I output my healthcheck.js stdout and stderr to logger, a utility for pipping output to syslog. You’ll also notice that I then have to use bash’s ${PIPESTATUS} env var array to look up the original exit code of the script, because of course I don’t want to use the logger command’s exit code to determine if the healthcheck was successful or not.

Also notice my use of exec to invoke the nodes healthcheck. This is important in all Docker executable hooks, not only in health checks, to make sure that the process of the node script is the one to receieve signals (like SIGTERM) from the docker runtime, not the bash process that is the parent of the, in my case, nodejs process.

File: forward_syslog.conf

$template RFC3164fmt,"<%PRI%>%TIMESTAMP% %fromhost% %syslogtag%%msg%"

if $syslogtag startswith 'my-app'\
then @@cec-logger:5001;RFC3164fmt

This format is just my own production log format, you can of course use a more standard one.

Forward

I do wish the Docker healthcheck protocol was more widely embraced by the container community (in Kubernetes, for example) and that Ansible’s docker_container_module implemented the --health-start-period option.