Centralizing Docker container logs

The state of Docker container logging is evolving; with options out there such as logging to a volume, running a log collection agent in each container or reading the raw Docker log files after some clunky configuration of program X.

For me the solution lies with Docker: soon they will solve the container logging question, so we need a tool to read those logs and forward them to a central point. Currently the Docker way is to log directly to stderr/stdout.

So before going any further; all of your containers need to be configured to write their logs there.

You probably should also set up log rotation on your Docker log files.

$ cat /etc/logrotate.d/docker 
/var/lib/docker/containers/*/*.log {
  rotate 7
  daily
  compress
  delaycompress
  copytruncate
  sharedscripts
}

Logging: central collection point

I use logentries.com for my log centralization and management. They already have various agents and tools for log collection, but nothing that worked directly with Docker, or that ran nicely within a Docker container. But they do have a very easy to use API, so building a tool to interact with the service doesn’t really require too much thinking.

Introducing dle

dle is a tool that can sit inside a Docker container, requires very little configuration and forwards all Docker logs to logentries. It could also no doubt also be use against other providers.

It gets the logs from Docker itself (it doesn’t read the raw log files), and picks up any required configuration from the each container’s environment. So you can hopefully just leave it running in the background.

Requirements

  • Docker API v1.14 (if you use something older, then release tag 0.0.2 in Git/Docker’s hub is something for you)
  • Containers must log to stdout/stderr (at least while we wait for Docker to solve the container logging question)

Downloading

You can get a copy of dle from the Docker registry:

$ docker pull justadam/dle:0.0.3

Or you can build from source.

Getting started

You need an account at logentries.com: if you sign up because of this article, let me know so I could claim you as a referral :)

Log into your account and click on + Add new in Hosts & Logs.

Choose Manual configuration from the bottom and fill in the form. This first log entry we are creating is a catch all log for all containers which do not get configured which a specific log token.

Give the log a name (perhaps Docker default) and select Token TCP.

Now click Register new log, and you will see a message similar too:

✓ New log successfully registered. 
The token is:
475f2c34-7dfb-53e1-8bb3-45befe357814

Assuming you are using the Docker container, starting dle is as follows:

$ docker run -d --name dle -e DOCKER_ENDPOINT=unix:///tmp/docker.sock -e DLE_DEFAULT_TOKEN=YOUR_DEFAULT_TOKEN -e DLE_LOG_LEVEL=warn -v /var/run/docker.sock:/tmp/docker.sock justadam/dle:0.0.3

Replace DLE_DEFAULT_TOKEN=YOUR_DEFAULT_TOKEN with the token you got after following the steps above.

You can also pass -e DLE_IGNORE=true to ignore this container’s logging, or -e DLE_TOKEN=NEW_TOKEN to send the log to a specific log at logentries (create a new token by following the steps above again).

Configuration

Once dle starts it will find all the containers which are active and start forwarding their logs to the default token used above. It will continue to monitor Docker for all new container starts and then forward their logs.

Having every container log to the same logentry is probably not a good idea, so dle can be told which logentry to send the logs to by defining the environment variable DLE_TOKEN in the container. You can also request that a container’s logs are ignored by defining the environment variable DLE_IGNORE=true.

Getting another token

Create another logentry at logentries.com by going into your newly created host and clicking + Add new log. Follow the same steps as above; Manual configuration, give the log a name, Token TCP, select the host from the pulldown list and then click Register new log. Copy the token generated and start your container:

$ docker run -d --name YOUR_SERVICE -e DLE_TOKEN=the-generated-token you/yourservice:latest

And those logs will now be appearing at logentries.com in the new log you created.

Rince and repeat for all containers, and you now have a centralized point for your Docker container logs.


What is missing

We’re an early version and there is still a bit of work todo:

  • Dropped connection handling & notification (and what we do with the log output during downtime)
  • No log seeking options (sending missed entries)
  • Unit tests
  • Actual long term testing and usage in real world scenarios
  • Profiling and performance analysis

But this should be enough to get started and see how this works.

Adam Bell-Hanssen

maybe, someday .. just another code and ops guy

Oslo, Norway