Docker rootless keeps restarting?
As some of you may know, this blog is hosted on Raspberry Pi. To monitor its status, I wrote a script, which you can find here.
Recently, I decided to switch the Docker daemon to run in rootless mode. However, after making this change, I started receiving notifications indicating that the blog was frequently going offline.
Whenever this happens, I ssh into my Pi and run the command
docker ps to list the running containers.
Strangely, I noticed that the container was only up for less than a second:
1$ docker ps 2CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 39bd3ad610e97 ghcr.io/quantonganh/blog:master "./blog -config conf…" 3 hours ago Up Less than a second 80/tcp
Additionally, I checked the status of the Docker service using the following command:
1● docker.service - Docker Application Container Engine (Rootless) 2 Loaded: loaded (/home/pi/.config/systemd/user/docker.service; enabled; vendor preset: enabled) 3 Active: active (running) since Wed 2023-07-12 09:33:17 BST; 56s ago 4 Docs: https://docs.docker.com/go/rootless/ 5 Main PID: 2313 (rootlesskit) 6 Tasks: 214 7 CPU: 13.395s
I was surprised by the fact that the Docker service showed an active state only 56s ago.
While analyzing the logs, I paid particular attention to these lines:
1Jul 12 09:31:06 raspberrypi systemd: Stopping User Manager for UID 1000... 2Jul 12 09:31:06 raspberrypi systemd: Stopping libcontainer container e0296500d5f75ef63f66686559c5122ed7bd4da2e88210c19f9ab0b352155554. 3Jul 12 09:31:06 raspberrypi systemd: Stopping Docker Application Container Engine (Rootless)... 4Jul 12 09:31:06 raspberrypi dockerd-rootless.sh: time="2023-07-12T09:31:06.720404072+01:00" level=info msg="Processing signal 'terminated'"
These logs indicated that the
systemd had stopped the User Manager for my UID and sent a SIGTERM signal to the Docker daemon?
I wondered why this was happening.
After some research, I came across the Arch Linux wiki page on systemd/User instances:
The systemd user instance is started after the first login of a user and killed after the last session of the user is closed.
This led me to suspect that
docker.service was being terminated when my SSH session is ended.
To confirm my hypothesis, I checked the linger status using the following command:
1$ loginctl user-status $(whoami) 2pi (1000) 3 Since: Wed 2023-07-12 09:33:10 BST; 57min ago 4 State: active 5 Sessions: 14 9 8 *4 6 Linger: no
Indeed, the output showed that lingering was disabled for my user. Therefore, I enabled lingering for my user:
1$ loginctl enable-linger $(whoami) 2$ loginctl user-status $(whoami) 3pi (1000) 4 Since: Wed 2023-07-12 09:33:10 BST; 1h 0min ago 5 State: active 6 Sessions: 14 9 8 *4 7 Linger: yes
Enabling lingering ensured that the Docker service would continue running even after my SSH session was closed.
Tags: docker rootless systemd
- Docker Compose healthcheck
- condition form of depends_on in docker-compose version 3
- Let's Encrypt too many certificates already issued
- plugins/docker failed to resolve Keycloak hostname?
- Why my golang docker container exits immediately (code 127)?