Running Docker in production gotcha — “cpuset.cpu”

I might say this can affect anyone running Docker in Swarm (as I am) or in Kubernetes.

Problem

From Github issue 40041:

Docker does not sees a new cpuset if hot added after provisioning a VM with less CPUs because it hard codes somehow during installation its original cpuset in /sys/fs/cgroup/cpuset/docker/cpuset.cpus

In other words:
Boxes that were provisioned with N CPUS and later you bumped the resources to N+x CPU cores, will experience the problem where Docker will only see N CPUs available in every container instead of N+x CPUs

I believe containers should use the value from docker info => CPUs: 8 (runtime) instead of hardcoding it during installation.

If you don’t want to keep reading thats Ok, short solution

The quick solution

First review you are affected by this problem:

# in the host OS
# print the number of processing units available
$ nproc
N
# check if containers are indeed seeing those available units
$ docker run -it centos:7 nproc
M
# (M == N)? "not affected" : "you are affected by it."
# Check cpuset.cpus specified to Docker.
$ cat /sys/fs/cgroup/cpuset/docker/cpuset.cpus
M

To fix it

echo "0-7" > /sys/fs/cgroup/cpuset/docker/cpuset.cpus
# or more "dynamically"
echo "0-$(( $(nproc) - 1 ))" >  /sys/fs/cgroup/cpuset/docker/cpuset.cpus

Context

How I got here?

We were seeing some strange behavior where our applications were stressing out only the first 4 CPUs (0–3) out of 8 CPUS, they were not able to see nor use the remaining 4 CPUS (4–7).

This was caused because our VMs were originally provisioned, ~1 year ago, with 4 CPUs, We installed Docker using community edition installation instructions, nothing fancy, nothing out of normal provisioning.

After some time we decided to increase the resources in our boxes because we started to see the need for it and we were deploying more and more applications to our clusters; We hot plugged more CPUs thinking that will relief the resource usage, but with no luck, the containers were still seeing 0–3 cpuset as the only available CPUs.

That is when things got interesting, we researched and found some good issues and articles:

• https://medium.com/@kasunmaduraeng/docker-namespace-and-cgroups-dece27c209c7
• https://stackoverflow.com/questions/49151296/how-many-cpus-does-my-docker-container-have
• https://github.com/docker/for-linux/issues/536

Which led me to write this article and share it with you all.