Docker swarm stacks/services resource limits and reservations

Photo by Fancycrave on Unsplash

According to this documentation in order to run your application in production you should limit container’s resource limits:

By default, a container has no resource constraints and can use as much of a given resource as the host’s kernel scheduler allows. Docker provides ways to control how much memory, CPU, or block IO a container can use, setting runtime configuration flags of the docker run command. This section provides details on when you should set such limits and the possible implications of setting them

The reasons behind it are pretty well explained in that documentation, so I will not re explain them.

What about swarm services or stacks resource limitation ? There’s not so much explanation given, so you are forced to go back to container resource’s limitation, which it has nothing wrong, in my case however I was looking for something more “swarm wise”.

But if you are like me, and you are already running (on production, stage, home etc):

• A (few) swarm cluster(s)
• A (few) stack (s)

But you’ll like to apply some of this advices to your already running setup, given the following example:

1. Your cluster is 1 GB of RAM x 1CPU .
2. Your app only uses 50MB (in “idle” mode) but it can burst up to 320MB of memory usage.
3. Lets assume host OS consumes 0 MB and 0 CPU %.

4. Your docker-compose.yml has the following description.

deploy:
  resources:
    limits:
      cpus: '0.5'
      memory: 350M
    reservations:
      cpus: '0.1'
      memory: 100M

It will:

• Indeed limit your app/container to only use up to 350M. (Runtime)
• Limit your app/container uses 50% of the CPU every second. (Runtime)
• Reserve (this is where it gets tricky, this is at scheduled-time), the Swarm scheduler will schedule a new task only if it can reserve at least 100M.

So if you will like to scale your app to more than 10 replicas (in “idle” mode), Guess what?

No, swarm will not scale nor schedule your app even when you have plenty memory (~ 500 MB free) as it will say:

overall progress: 3 out of 10 tasks 
1/10: running   [==================================================>] 
2/10: running   [==================================================>] 
3/10: running   [==================================================>] 
4/10: no suitable node (insufficient resources on 1nodes) 
5/10: no suitable node (insufficient resources on 1nodes) 
6/10: starting  [============================================>      ] 
7/10: no suitable node (insufficient resources on 1nodes) 
8/10: no suitable node (insufficient resources on 1nodes) 
9/10: starting  [============================================>      ] 
10/10: starting  [============================================>      ] 

So lesson of the day: be careful with reservations as this will require a bit more real resource usage of your app to find a sweet spot.

Further Reading: https://semaphoreci.com/community/tutorials/scheduling-services-on-a-docker-swarm-mode-cluster