Docker Compose healthcheck

"Life is all about sharing. If we are good at something, pass it on." - Mary Berry

Docker Compose healthcheck

2021-09-09

Categories: DevOps

The most important thing when running integration test using docker-compose is ensured that one container is started completely before others.

Sometime wait-for-it is not enough:

 1  cassandra:
 2    image: bitnami/cassandra:latest
 3    ports:
 4      - '7000:7000'
 5      - '9042:9042'
 6    volumes:
 7      - /path/to/init-scripts:/docker-entrypoint-initdb.d
 8
 9  wait-for-cassandra:
10    image: willwill/wait-for-it
11    command: cassandra:9042 -t 60
12    depends_on:
13      - cassandra:

We need a way to check if cassandra is really healthy.

Since I CREATE KEYSPACE in the init script:

1CREATE KEYSPACE IF NOT EXISTS test WITH replication = {'class':'SimpleStrategy', 'replication_factor' : 1};

I want to make sure that keyspace ‘test’ exist before starting others:

 1  cassandra:
 2    image: bitnami/cassandra:latest
 3    ports:
 4      - '7000:7000'
 5      - '9042:9042'
 6    volumes:
 7      - /path/to/init-scripts:/docker-entrypoint-initdb.d
 8    healthcheck:
 9      test: [ "CMD", "cqlsh", "-u cassandra", "-p cassandra" ,"-k test" ]
10      interval: 5s
11      timeout: 10s
12      retries: 6
13
14  wait-for-cassandra:
15    image: willwill/wait-for-it
16    command: cassandra:9042 -t 60
17    depends_on:
18      cassandra:
19        condition: service_healthy

Try to bring up the project:

1ERROR: for wait-for-cassandra  Container "d5a87a961ba1" is unhealthy.

Why cassandra is unhealthy? Is there a way to view docker-compose healthcheck logs?

Run docker inspect --format "{{ json .State.Health }}" cassandra_1 | jq and I got:

1    {
2      "Start": "2021-09-09T00:16:41.5668221Z",
3      "End": "2021-09-09T00:16:42.1117382Z",
4      "ExitCode": 1,
5      "Output": "Connection error: ('Unable to connect to any servers', {'127.0.0.1': AuthenticationFailed('Failed to authenticate to 127.0.0.1: Error from server: code=0100 [Bad credentials] message=\"Provided username  cassandra and/or password are incorrect\"',)})\n"
6    }

Why authentication failed? Turned out that there is an extra space before cassandra in the error message:

1Provided username  cassandra and/or password are incorrect

So, either you have to remove that space:

1    healthcheck:
2      test: [ "CMD", "cqlsh", "-ucassandra", "-pcassandra", "-ktest" ]
3      interval: 5s
4      timeout: 10s
5      retries: 6

or use the long options:

1    healthcheck:
2      test: [ "CMD", "cqlsh", "--username", "cassandra", "--password", "cassandra", "--keyspace", "test" ]
3      interval: 5s
4      timeout: 10s
5      retries: 6

or use the shell form (if the container has a shell /bin/sh):

1    healthcheck:
2      test: cqlsh -u cassandra -p cassandra -k test
3      interval: 5s
4      timeout: 10s
5      retries: 6

Tags: docker-compose healthcheck docker

Edit on GitHub