You can also view throttling metrics in cpu.stat. Quota and period settings are in cpu.cfs_quota_us and cpu.cfs_period_us. When an application has used its allotted CPU quota for a given period, it gets throttled until the next period.Īll CPU metrics for a cgroup are located in /sys/fs/cgroup/cpu,cpuacct/. The CFS-Cgroup bandwidth control mechanism manages CPU allocation using two settings: quota and period. When hard CPU limits are set in a container orchestrator, the kernel uses Completely Fair Scheduler (CFS) Cgroup bandwidth control to enforce those limits.
Background: How container CPU constraints workĪlmost all container orchestrators rely on the kernel control group (cgroup) mechanisms to manage resource constraints. We eventually found the culprit, but first we had to understand the mechanisms at work. Normal CPU usage and high throttling shouldn’t have been possible.
Upon further investigation, it was clear that the incidence of high response times directly correlated to periods of high CPU throttling. We saw an increase in tail response times for our web applications, but when we looked at CPU usage, everything seemed fine. The issue began last year, shortly after the v4.18 release of the Linux kernel. In this two-part series, I’ll explain our journey to find the root cause and how we ultimately arrived at the solution. In doing so, we lowered worst-case response latency in one of Indeed’s applications from over two seconds to 30 milliseconds. This year, my teammates and I solved a CPU throttling issue that affects nearly every container orchestrator with hard limits, including Kubernetes, Docker, and Mesos. This post is the first in a two-part series.
0 kommentar(er)
