Managing and calculating resources used for Pods
we create those Pods without caring about how much CPU and memory they will need to run or are limited to only how much CPU and memory they can use. chief. But when running in reality, determining the resources of each Pod is very important, it will help our applications achieve good performance when running in the cluster. For example, we have 2 services, one handles notifications, the other handles transactions. We don't want the notify service to run and take up all the resources of the transaction processing service, because the transaction service is more important.
In this article we will see how to configure and manage resource requests and resource limits of a Pod in the most effective way.
Requesting resources
When writing a Pod's configuration, we can add properties to specify the total amount of CPU and memory of a container that it needs to run (called resource requests), and the total amount of CPU and memory that it can only handle. We can consume that much, but we cannot consume more than the total amount we have specified (called the resource limit).
We will specify properties for each container in the Pod, and the resouce requests - resource limits of a Pod will be added up from its containers.
Create Pod with resource requests
Now let's do an example to understand better, create a file named requests-pod.yaml with the following configuration:
In the file above, we use the resources.requests.cpu attribute to specify the amount of CPU this container needs, and resources.requests.memory to specify the amount of memory this container needs. Now we create a Pod and test its CPU with the top command:
The dd command running in the container can consume as many resources as it wants because we have not specified a resources limit. In the example above we see there are 6 cpu, each 12% is 120m, total is 720m.
What do resource requests mean?
If you think that when we specify the resources.requests attribute for the container in the Pod, kubernetes will keep that amount of server resources for the container to run, then that's not true.
Note that one important thing is that when we specify this value, as in the example above, kubernetes will not keep 200m cpu and 10Mi memory of the server for this container to use, but this value will be used during the process. A Pod is scheduled to a worker node. The scheduler will find which worker node still has unallocated CPU greater than 200m and unallocated memory greater than 10Mi to schedule Pod to that worker node. If we do not specify resources.requests , then when the Pod is scheduled to a worker node, the Scheduler will not consider the worker node's CPU and unallocated memory values ​​for the Pod. This means that if our Pod needs 200m CPU to run, but we do not specify it clearly, then a worker node only has 100m CPU, our Pod can also be scheduled to that worker node.
And another important thing is that the CPU and unallocated memory values ​​will be calculated by the Scheduler by subtracting the node's resources from the total resources.requests of the Pod (if any, or if the pod's current CPU if there is no index). resource requests, or the current CPU is larger than the resource requests because we do not use limit, the Scheduler will take the larger value) on a worker node, not the value of the worker's free and unused CPU and memory. node. Even if the worker node's free value meets Pod requests and the unallocated value does not, the Pod will not be scheduled there.
In the illustration above, the Pod will not be deployed to the worker node because the current CPU request is larger than the remaining unallocated CPU of the worker node, even though the actual free CPU is up to 45%. The scheduler will not consider actual free cpu but only unallocated cpu.
How Scheduler uses resource requests to choose the best node for Pod
In the post Kubernetes internals architecture , we talked about how the Scheduler will filter out the best nodes using an algorithm and then arrange them in order of priority to select the node with the highest priority. Then, resource requests will be used as a criterion to evaluate the priority order for the node, there are two methods: LeastRequestedPriority, MostRequestedPriority.
LeastRequestedPriorit: will choose the node with the least total resource request, meaning the most unallocated resources.
MostRequestedPriority: will choose the node with the highest total resource request, meaning the least unallocated resource.
The scheduler will be configured to choose one of the two methods above, by default the method used is LeastRequestedPriorit. We can configure Scheduler to use the MostRequestedPriority method instead, use MostRequestedPriority when we want our resources to be used as evenly as possible, very important in a cloud provider environment, we will want to use as few VMs as possible to Cut the cost.
Scheduler in action
Now we will create another pod to see if the Scheduler does it the way we said above. First we will look at the node's capacity.
There are two important properties we need to pay attention to: Capacity and Allocatable, with Capacity being the total resources of the worker node, and Allocatable being the resources that the Pod can use. We will see that it is not equal because there will be a few resources for the system to use. We shouldn't use it all. We have an allocatable cpu of 8 cores = 8000m, and an allocated cpu of 3475m. We will create a Pod that requests 3000m to see if our cpu allocation increases.
If you run kubelet version greater than 1.21, you will receive the following message:
If you encounter this error, create a config file and do not use cli. Now let's describe the node again:
We will see that now our CPU allocation has increased to 6475m. If we create another Pod with a request of 3000m, our Pod will not be able to deploy to the node.
Our Pod requests-pod-3 will be in pending state because it is not deployed to the pod. When we describe the pod, we will see the reason is that no worker node has enough CPU for the Pod:
Limiting resources
We have understood how to use resources requests, now we will see how to use resources limit, to avoid one Pod taking up the resources of other Pods. Create a file named limited-pod.yaml with the following configuration:
In the config file above, we use the resources.limits attribute to specify the resource limit of a container. If we do not specify the request attribute, by default it will be assigned equal to the value of limits. Create file and test:
We will see that above when we have not specified the limit, the cpu core we can use is 6. After we specify, the cpu core we can use is now only 1.
When a container exceeds limit
So what happens when one of our containers uses resources beyond the limit resource we specified. Then there will be two cases:
Regarding cpu, when our container needs more than the number of cpu we have specified in the limit, it cannot use more than this cpu, it can only use the cpu we have specified.
For memory, it will be different. When our container exceeds the allowed limit, this container will be killed. If we specify the restartPolicy mentioned in lesson 12 as Always or OnFailure, our container will automatically restart. .
QoS classes
Unlike cpu limit, our total memory limit when configuring can exceed 100% of the worker node's memory. So what happens if a Pod runs out of worker node's memory, will that Pod be killed or will another Pod be killed?
For example, we have two Pods, one A with limit 550Mi memory, one B with limit 600Mi memory, our worker node memory only has 1G RAM. Now guy A is running and taking up 500Mi, guy B is running and taking up 400Mi, then guy B is running and taking up 550Mi of memory, the two guys combined are 1050Mi, exceeding the memory of the worker node, while no one is running over the memory. Its limit is over, so who will be killed?
So kubernetes provides us with a way to rate the priority of Pods. Pods with lower priority will be killed first. Quality of Service (QoS) is how kubernetes prioritizes Pods, there are 3 QoS:
BestEffort (lowest priority)
Burstable
Guaranteed (highest priority)
QoS will be assigned to the container, then combining all the QoS of the container we will get the QoS of the Pod.
Determine the QoS of the container
BestEffort is the container with the lowest priority, this QoS class will be assigned to the container without specifying resources requests and limits attributes. If the worker node's memory exceeds 100%, the Pod with class BestEffort will be killed first.
Burstable is a container with medium priority. This QoS class will be assigned to containers that specify each resource request, or specify that the values ​​of the two attributes resources requests and limits are not equal, Pods with class Burstable will be killed. when no more Pods with class BestEffort exist.
Guaranteed is the container with the highest priority, killed last. This QoS class will be assigned to the Pod that:
Specify both resources requests and limits in the container. Or specify each resource limit, resource requests will default to the value of resource limits.
These two attributes must have equal values.
Determine the Pod's QoS
If all containers have the same QoS class, then that QoS will be the Pod's QoS, otherwise, the QoS will be combined according to the following table:
| Container 1 QoS class | Container 2 QoS class | Pod’s QoS class |
|---|---|---|
Best Effort | Best Effort | Best Effort |
Best Effort | Burstable | Burstable |
Best Effort | Guaranteed | Burstable |
Burstable | Burstable | Burstable |
Burstable | Guaranteed | Burstable |
Guaranteed | Guaranteed | Guaranteed |
Understand how Pods are killed when memory exceeds the worker node's limit
As we said above, when the worker node's memory exceeds 100%, the Pod with Qos as BestEffort will be killed first, followed by the Pod with Qos as Burstable, and finally Guaranteed.
How Pod with QoS will be killed
Each of our processes will have an OutOfMemory (OOM) score. When Pods have the same QoS class, the process with higher OOM will be killed first. OOM score will be calculated by the number of memory requests and the number of memory processes currently in use. Whoever is using the higher percentage of memory with the request will be killed first.
Configure default request and limit for namespace
We have learned how to use requests and limits, by specifying properties for each specific container. If we do not specify, the Pod will spend as much as it wants, so it would be good to have a way to configure requests and limits with a default value for a container when we do not specify its requests and limits.
LimitRange resource
Instead of having to configure each container separately, we can create a resource called LimitRange, to specify default requests and limits in a namespace for us.
LimitRange will have two uses, specifying the minimum and maximum of requests and limits, and the default value when we do not specify properties in the container.
When we create LimitRange, it will be configured inside the Admission control plugin that we talked about in lesson 11 .
Create LimitRange
Now we will create a LimitRange resource, create a file called limits.yaml with the following configuration:
In the config file above, we specify the limit for 3 components: Pod, Container, PersistentVolumeClaim. We specify the Pod's cpu minimun as 50m and memory as 5Mi, maximun as 1 cpu and 1Gi memory. For containers, we specify the default value of the request with the defaultRequest attribute and the default value of limit with the default attribute , and the max and min attributes are similar to those of the Pod. We can also specify a value for storage with type PersistentVolumeClaim.
We create and test whether LimitRange works as we want.
If it prints this error, then our LimitRange is working correctly, we only allow maximun cpu request to be 1 core, above we request 3 cores so we cannot create a Pod.
We check and see that LimitRange has correctly validated the min and max of a Pod that can be created. Now let's check if it specifies default requests and limits values ​​for us.
Check the terminal and we will see that our container has been assigned default requests and limits values. LimitRange already works as we want.
Limit the total number of resources of a namespace
We have seen the convenience when we use LimitRange. But you can notice that LimitRange only limits the resources of an object in a namespace, but cannot limit the total usable resources of a namespace. When working on a project with many different teams, we will want each team to only be able to use the amount of resources we assign to them, this team should not use other teams' resources. To do that, kubernetes provides us with a resource called ResourceQuota.
ResourceQuota
This is a resource that will help us limit the resources of a namespace. When we create a ResourceQuota resource, it will be configured in the Admission Control plugin like LimitRange. When a Pod is created, it must go through this plugin to check whether the Pod is valid or not. whether or not the resource limit we specified in ResourceQuota is used, if so, the API server will return an error.
ResourceQuota not only limits the resource usage of Pods and storage of PersistentVolumeClaims, but can also limit the number of Pods and PersistentVolumeClaims that can be created in a namespace.
ResourceQuota limits cpu and memory
Now we will look at an example to make it easier to understand, create a file named quota-cpu-memory.yaml with the following configuration:
In the config file above, we will specify the total request resource of the default namespace as 400m cpu and 200Mi memory. When we create a Pod, we cannot specify a request larger than this number, and the total resource it uses is limited. is 600M CPU and 500Mi memory. When we create this ResourceQuota, all Pods in the namespace can only use a maximum of 600m CPU and 500Mi memory. For example, if we need to create a Pod with a request of 200m cpu, we can only create a maximum of 3 Pods in the default namespace.
After we create a resource, we can see how many resources have been used within the quota with the describe command:
If we delete all the Pods created above, we will see that all 4 parameters are 0. Now we will create a Pod and check to see if this number increases:
We see that the value here has increased exactly as we want, now we will create a Pod that requires 400m cpu, this time it will report an error:
You will see the error that pods "quota-pod-1" is forbidden: exceeded quota: cpu-and-mem , meaning that Pod quota-pod-1 exceeds the limit of our current ResourceQuota, so the Pod cannot be created. go out.
ResourceQuota limits persistent storage
To limit persistent storage of a namespace, we create a file with the following configuration:
In the config file above, the request limit for all PersistentVolumeClaims in the default namespace is 500Gi. We can also clearly specify the request limit for each specific StorageClass, by prefixing the StorageClass name as follows <storageclass-name>.storageclass.storage.k8s.io/requests.storage.
ResourceQuota limits the number of resources that can be created
Besides resource and storage limits, we can also use ResourceQuota to specify the number of resource objects we can create in a namespace, for example:
In the config file above, we limit the number of pods that can be created in the default namespace to 10, replication controllers to 5, secrets to 10, etc. The resources that ResourceQuota can specify are:
Pods
ReplicationControllers
Secrets
ConfigMaps
PersistentVolumeClaims
Services: can specify the number of LoadBalancer Services and NodePort Services
Quota scope for Pod
For Pod, we can specify whether Quota is applied to it or not based on the following 4 properties:
BestEffort: only affects Pods with Qos class BestEffort
NotBestEffort: only affects Pods with Qos class Burstable and Guaranteed
Terminating: only affects Pods with activeDeadlineSeconds attribute
NotTerminated: only affects Pods that do not have the activeDeadlineSeconds attribute
For example:
In the above config file, we can only create 4 pods with Qos class BestEffort and no active deadline attribute specified.
Conclude
So we have learned about how to configure requests and limits for Pod. When running a real application on kubernetes cluster, we need to pay close attention to this index, to help our application run as best as possible. We also know how to use LimitRange to configure default requests and limits for Pods. Using LimitRange will be more convenient when we want all containers in our Pod to have requests and limits. Use ResourceQuota to limit resources for a namespace, dividing resources for each team appropriately. If you have any questions or need further clarification, you can ask in the comment section below.
Last updated
