Shutting down Docker containers via supervisor

Question

I'm unable to shut down Docker containers that were launched by supervisor through supervisorctl stop all. Even through supervisorctl status shows the containers are down, docker ps and ps indicate that they are in fact still running.

Consulting the supervisor documentation on the action for supervisorctl stop <name> reveals that SIGTERM is sent to processes followed by SIGKILL if still running after some grace period. I tried to do this manually and found that

• SIGTERM sent to a docker run process doesn't do anything
• SIGKILL does kill the process, but doesn't actually update docker. docker ps shows this container is still running
• Supervisor's SIGKILL doesn't shut down the container

The questions is: How do I properly shut down a Docker container by supervisor?

---

Here's the result of my experiments simulating supervisor:

Starting position: foo-1 and bar-1 are running (I left the GCE containers in, in case they make a difference). ps aux and docker ps are in sync.

me@devenv:~$ sudo docker ps
CONTAINER ID        IMAGE                   COMMAND                CREATED             STATUS              PORTS                    NAMES
5ba70bf8937f        me/app:foo              "/bin/sh -c 'supervi   5 minutes ago       Up 5 minutes                                 foo-1
e1a684bcfceb        me/app:bar              "/bin/sh -c 'supervi   5 minutes ago       Up 5 minutes                                 bar-1
fce5db0517df        google/cadvisor:0.8.0   "/usr/bin/cadvisor"    35 minutes ago      Up 35 minutes                                bbbb 
db677eed47ef        kubernetes/pause:go     "/pause"               35 minutes ago      Up 35 minutes       0.0.0.0:4194->8080/tcp   aaaa

me@devenv:~$ ps aux | grep "docker run"
root     23358  0.0  0.1 124092 11856 pts/0    Sl   02:05   0:00 docker run --rm --name foo-1 ... -i me/app:foo
root     23365  0.0  0.1 124092 11928 pts/0    Sl   02:05   0:00 docker run --rm --name bar-1 ... -i me/app:bar

Simulate supervisorctl stop foo-1 by sending SIGTERM to the process. Result: process still active.

me@devenv:~$ sudo kill -SIGTERM 23358

... <waiting> ...

me@devenv:~$ ps aux | grep "docker run"
root     23358  0.0  0.1 124092 11856 pts/0    Sl   02:05   0:00 docker run --rm --name foo-1 ... -i me/app:foo
root     23365  0.0  0.1 124092 11928 pts/0    Sl   02:05   0:00 docker run --rm --name bar-1 ... -i me/app:bar

me@devenv:~$ sudo docker ps
CONTAINER ID        IMAGE                   COMMAND                CREATED             STATUS              PORTS                    NAMES
5ba70bf8937f        me/app:foo              "/bin/sh -c 'supervi   6 minutes ago       Up 6 minutes                                 foo-1
e1a684bcfceb        me/app:bar              "/bin/sh -c 'supervi   6 minutes ago       Up 6 minutes                                 bar-1
fce5db0517df        google/cadvisor:0.8.0   "/usr/bin/cadvisor"    36 minutes ago      Up 36 minutes                                bbbb 
db677eed47ef        kubernetes/pause:go     "/pause"               36 minutes ago      Up 36 minutes       0.0.0.0:4194->8080/tcp   aaaa

Next thing supervisor would do is issuing SIGKILL. Result: Process is killed (ps aux), but still showing as running docker process (docker ps).

me@devenv:~$ sudo kill -SIGKILL 23358
me@devenv:~$ ps aux | grep "docker run"
root     23365  0.0  0.1 124092 11928 pts/0    Sl   02:05   0:00 docker run --rm --name bar-1 ... -i me/app:bar

me@devenv:~$ sudo docker ps
CONTAINER ID        IMAGE                   COMMAND                CREATED             STATUS              PORTS                    NAMES
5ba70bf8937f        me/app:foo              "/bin/sh -c 'supervi   19 minutes ago      Up 19 minutes                                foo-1
e1a684bcfceb        me/app:bar              "/bin/sh -c 'supervi   19 minutes ago      Up 19 minutes                                bar-1
fce5db0517df        google/cadvisor:0.8.0   "/usr/bin/cadvisor"    49 minutes ago      Up 49 minutes                                bbbb 
db677eed47ef        kubernetes/pause:go     "/pause"               49 minutes ago      Up 49 minutes       0.0.0.0:4194->8080/tcp   aaaa

Supervisor was shut down during the above experiment (to avoid its autostart behaviour to interfere). The result for explicitly sending SIGKILL to the process could not be achieved by supervisor; the process was still alive (even though the supervisor logs state otherwise). docker stop <container_id> did however stop the container.

Update

Inside the Docker containers is also a supervisord process running which manages certain processes. Perhaps the problem is that signals aren't propagated and thus it won't shut down...

Update 2

I narrowed down the problem. I was able to directly start a containers process from the Dockerfile instead of going through starting supervisord and it makes a difference. I'm able to control this container through supervisor (the one outside of the docker containers which controls the containers).

Update 3

Setting stopasgroup=true as suggested here does not change anything for me.

Update 4

I was able to solve one of the problems: supervisorctl not being able to shut down the process. The problem was that I was launching the docker containers in the supervisor configuration file with command=sudo docker run... which created a sudo docker run... and a docker run... process. supervisorctl stop... just terminated the sudo docker run... process while the actual docker process was still running. When I omit the sudo command, only 1 process per supervisor program is started and supervisorctl stop terminates the process.

One problem remains which is that docker ps shows the container is still running while ps aux doesn't. Oddly enough the containers still seem to be active as they respond to requests. A quick look at the process list confirms that all processes spawned by the docker container are still active, yet the docker run... process is missing from the process list.

Update 5

Sending SIGTERM, SIGHUP or SIGQUIT to the docker run process doesn't seem to do anything to the process. Only SIGKILL terminates the docker process properly. Supervisor gets updated properly, but docker ps still shows the docker process running.

Answer 1

I think I found the problem. I didn't realise it, but there's multiple ways of starting a program when a docker container is fired up.

Apparently CMD myexec param1 param2 starts a shell which in turn starts myexec (in fact these two processes are visible in the container with /bin/sh -c myexec... at PID 1. A better way is to start the program directly (in my case supervisord).

On the other hand, CMD ["/usr/bin/python", "/usr/local/bin/supervisord", "-c", "/root/supervisord.conf", "--nodaemon"] worked fine. I am now able to start and stop the docker container through supervisor.

Here's the relevant section in the docker docs:

The CMD instruction has three forms:

CMD ["executable","param1","param2"] (exec form, this is the preferred form)

CMD ["param1","param2"] (as default parameters to ENTRYPOINT)

CMD command param1 param2 (shell form)

Update

Example supervisor file (inside Docker container):

[program:app]
command=python run_web_server.py
stdout_logfile=/var/log/app/app.log
directory=/opt/app
autostart=true
autorestart=false
stopsignal=INT
redirect_stderr=true
startretries=0
stopasgroup=true
killasgroup=true


[unix_http_server]
file=/var/run/supervisor.sock
chmod=0700

[supervisord]
logfile=/var/log/supervisor/supervisord.log
pidfile=/var/run/supervisord.pid
childlogdir=/var/log/supervisor

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run/supervisor.sock

The mako template to generate the Docker (outside) supervisor file:

[program:container]
command=docker run --rm --name ${name} \
% if container_links is not UNDEFINED:
% for host in container_hosts:
--add-host ${host['name']}:${host['ip']} \
% endfor
% endif
% if container_links is not UNDEFINED:
% for link in container_links:
--link ${link}:${link} \
% endfor
% endif
% if port_mappings is not UNDEFINED:
% for ext in port_mappings:
-p ${ext}:${port_mappings[ext]} \
% endfor
% endif
-e "INSTANCE_NAME=${name}" \
-e "TZ=${timezone}" \
% if environ is not UNDEFINED:
% for k in environ:
-e "${k}=${environ[k]}" \
% endfor
% endif
-v ${deployment_dir}/tmp:${deployment_dir}/app/tmp \
... more -v
-i foo/app-${version}:${type}
stdout_logfile=${deployment_dir}/log/${name}.log
redirect_stderr=true
autostart=false
autorestart=false
% if priority is not UNDEFINED:
priority=${priority}
% endif
startretries=0
# stopasgroup=true
# killasgroup=true

Answer 2

Hello I have managed that issue by using wrapper and trap signal.

Wrapper.sh:

function cleanup()
{
    docker kill ${CONTAINER_NAME}
}

trap cleanup INT

docker run -i --rm --name ${CONTAINER_NAME} ${IMAGE}&
wait

supervisord config:

[program:coolapp]
command=/opt/coolapp/wrapper.sh
directory=/opt/coolapp
autostart=true
autorestart=true
stderr_logfile=/var/log/coolapp/error.log
stderr_logfile_maxbytes=0
stdout_logfile=/var/log/coolapp/stdouot.log
stdout_logfile_maxbytes=0
stopasgroup=true
stopsignal=INT

Some thoughts about the issue:

You probably want to use -it option to keep container running in forgeround and supervisor will control runtime. But if you try it you`ll get an error like this:

the input device is not a TTY

It can be fixed by using -i option instead of -it

When you use -i option you able to start the container but not able to send signals to it. So in this case when supervisor stops container it sends configured signal (ususally SIGINT) and waits for SIGCHLD from process but docker client (docker run ...) does not send responce. After that supervisor will send SIGKILL signal that will kill client but container itself remain running.

I have tried to create pty (virtual TTY) with script command:

scropt -qc 'docker run -i --rm --name ${CONTAINER_NAME} ${IMAGE}' /dev/null

but still no results, this time supervisor kills script process and container remains running