pods
The total number of pods in a non-terminal state that can exist in the project.
replicationcontrollers
The total number of replication controllers that can exist in the project.
resourcequotas
The total number of resource quotas that can exist in the project.
services
The total number of services that can exist in the project.
secrets
The total number of secrets that can exist in the project.
configmaps
The total number of ConfigMap objects that can exist in the project.
persistentvolumeclaims
The total number of persistent volume claims that can exist in the project.
openshift.io/imagestreams
The total number of image streams that can exist in the project.
You can configure an object count quota for these standard namespaced resource types using the count/ . syntax while creating a quota.
1
is the name of the resource, and is the API group, if applicable. Use the kubectl api-resources command for a list of resources and their associated API groups.
Setting Resource Quota for Extended Resources
Overcommitment of resources is not allowed for extended resources, so you must specify requests and limits for the same extended resource in a quota. Currently, only quota items with the prefix requests. are allowed for extended resources. The following is an example scenario of how to set resource quota for the GPU resource nvidia.com/gpu .
Determine how many GPUs are available on a node in your cluster. For example:
In this example, 2 GPUs are available.
Set a quota in the namespace nvidia . In this example, the quota is 1 :
Create the quota:
Verify that the namespace has the correct quota set:
Run a pod that asks for a single GPU:
Verify that the pod is running:
Verify that the quota Used counter is correct:
Attempt to create a second GPU pod in the nvidia namespace. This is technically available on the node because it has 2 GPUs:
This Forbidden error message is expected because you have a quota of 1 GPU and this pod tried to allocate a second GPU, which exceeds its quota.
Quota Scopes
Each quota can have an associated set of scopes. A quota will only measure usage for a resource if it matches the intersection of enumerated scopes.
Adding a scope to a quota restricts the set of resources to which that quota can apply. Specifying a resource outside of the allowed set results in a validation error.
Terminating
Match pods where spec.activeDeadlineSeconds >= 0 .
NotTerminating
Match pods where spec.activeDeadlineSeconds is nil .
BestEffort
Match pods that have best effort quality of service for either cpu or memory .
NotBestEffort
Match pods that do not have best effort quality of service for cpu and memory .
A BestEffort scope restricts a quota to limiting the following resources:
pods
A Terminating, NotTerminating, and NotBestEffort scope restricts a quota to tracking the following resources:
pods
memory
requests.memory
limits.memory
cpu
requests.cpu
limits.cpu
ephemeral-storage
requests.ephemeral-storage
limits.ephemeral-storage
Ephemeral storage requests and limits apply only if you enabled the ephemeral storage technology preview. This feature is disabled by default.
Quota Enforcement
After a resource quota for a project is first created, the project restricts the ability to create any new resources that may violate a quota constraint until it has calculated updated usage statistics.
After a quota is created and usage statistics are updated, the project accepts the creation of new content. When you create or modify resources, your quota usage is incremented immediately upon the request to create or modify the resource.
When you delete a resource, your quota use is decremented during the next full recalculation of quota statistics for the project. A configurable amount of time determines how long it takes to reduce quota usage statistics to their current observed system value.
If project modifications exceed a quota usage limit, the server denies the action, and an appropriate error message is returned to the user explaining the quota constraint violated, and what their currently observed usage stats are in the system.
Requests Versus Limits
When allocating compute resources, each container may specify a request and a limit value each for CPU, memory, and ephemeral storage. Quotas can restrict any of these values.
If the quota has a value specified for requests.cpu or requests.memory , then it requires that every incoming container make an explicit request for those resources. If the quota has a value specified for limits.cpu or limits.memory , then it requires that every incoming container specify an explicit limit for those resources.
Sample Resource Quota Definitions
12345
The total number of ConfigMap objects that can exist in the project. |
The total number of persistent volume claims (PVCs) that can exist in the project. |
The total number of replication controllers that can exist in the project. |
The total number of secrets that can exist in the project. |
The total number of services that can exist in the project. |
1
The total number of image streams that can exist in the project.
1234567
The total number of pods in a non-terminal state that can exist in the project. |
Across all pods in a non-terminal state, the sum of CPU requests cannot exceed 1 core. |
Across all pods in a non-terminal state, the sum of memory requests cannot exceed 1Gi. |
Across all pods in a non-terminal state, the sum of ephemeral storage requests cannot exceed 2Gi. |
Across all pods in a non-terminal state, the sum of CPU limits cannot exceed 2 cores. |
Across all pods in a non-terminal state, the sum of memory limits cannot exceed 2Gi. |
Across all pods in a non-terminal state, the sum of ephemeral storage limits cannot exceed 4Gi. |
12
The total number of pods in a non-terminal state with BestEffort quality of service that can exist in the project. |
Restricts the quota to only matching pods that have BestEffort quality of service for either memory or CPU. |
12345
The total number of pods in a non-terminal state. |
Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this value. |
Across all pods in a non-terminal state, the sum of memory limits cannot exceed this value. |
Across all pods in a non-terminal state, the sum of ephemeral storage limits cannot exceed this value. |
Restricts the quota to only matching pods where spec.activeDeadlineSeconds is set to nil . Build pods will fall under NotTerminating unless the RestartNever policy is applied. |
12345
The total number of pods in a non-terminal state. |
Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this value. |
Across all pods in a non-terminal state, the sum of memory limits cannot exceed this value. |
Across all pods in a non-terminal state, the sum of ephemeral storage limits cannot exceed this value. |
Restricts the quota to only matching pods where spec.activeDeadlineSeconds >=0 . For example, this quota would charge for build or deployer pods, but not long running pods like a web server or database. |
1234567
The total number of persistent volume claims in a project |
Across all persistent volume claims in a project, the sum of storage requested cannot exceed this value. |
Across all persistent volume claims in a project, the sum of storage requested in the gold storage class cannot exceed this value. |
Across all persistent volume claims in a project, the sum of storage requested in the silver storage class cannot exceed this value. |
Across all persistent volume claims in a project, the total number of claims in the silver storage class cannot exceed this value. |
Across all persistent volume claims in a project, the sum of storage requested in the bronze storage class cannot exceed this value. When this is set to 0 , it means bronze storage class cannot request storage. |
Across all persistent volume claims in a project, the sum of storage requested in the bronze storage class cannot exceed this value. When this is set to 0 , it means bronze storage class cannot create claims. |
Creating a Quota
To create a quota, first define the quota in a file, such as the examples in Sample Resource Quota Definitions. Then, create using that file to apply it to a project:
Источник
Resource quotas per project
A resource quota, defined by a ResourceQuota object, provides constraints that limit aggregate resource consumption per project. It can limit the quantity of objects that can be created in a project by type, as well as the total amount of compute resources and storage that may be consumed by resources in that project.
This guide describes how resource quotas work, how cluster administrators can set and manage resource quotas on a per project basis, and how developers and cluster administrators can view them.
Resources managed by quotas
The following describes the set of compute resources and object types that can be managed by a quota.
A pod is in a terminal state if status.phase in (Failed, Succeeded) is true.
Table 1. Compute resources managed by quota
The sum of CPU requests across all pods in a non-terminal state cannot exceed this value. cpu and requests.cpu are the same value and can be used interchangeably.
The sum of memory requests across all pods in a non-terminal state cannot exceed this value. memory and requests.memory are the same value and can be used interchangeably.
The sum of local ephemeral storage requests across all pods in a non-terminal state cannot exceed this value. ephemeral-storage and requests.ephemeral-storage are the same value and can be used interchangeably. This resource is available only if you enabled the ephemeral storage technology preview. This feature is disabled by default.
The sum of CPU requests across all pods in a non-terminal state cannot exceed this value. cpu and requests.cpu are the same value and can be used interchangeably.
The sum of memory requests across all pods in a non-terminal state cannot exceed this value. memory and requests.memory are the same value and can be used interchangeably.
The sum of ephemeral storage requests across all pods in a non-terminal state cannot exceed this value. ephemeral-storage and requests.ephemeral-storage are the same value and can be used interchangeably. This resource is available only if you enabled the ephemeral storage technology preview. This feature is disabled by default.
The sum of CPU limits across all pods in a non-terminal state cannot exceed this value.
The sum of memory limits across all pods in a non-terminal state cannot exceed this value.
The sum of ephemeral storage limits across all pods in a non-terminal state cannot exceed this value. This resource is available only if you enabled the ephemeral storage technology preview. This feature is disabled by default.
Resource Name | Description |
---|---|
Table 2. Storage resources managed by quota
Resource Name | Description |
---|---|
The sum of storage requests across all persistent volume claims in any state cannot exceed this value. The total number of persistent volume claims that can exist in the project. The sum of storage requests across all persistent volume claims in any state that have a matching storage class, cannot exceed this value. The total number of persistent volume claims with a matching storage class that can exist in the project. |
Table 3. Object counts managed by quota
Resource Name | Description | |||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
The total number of pods in a non-terminal state that can exist in the project. The total number of ReplicationControllers that can exist in the project. The total number of resource quotas that can exist in the project. The total number of services that can exist in the project. The total number of services of type LoadBalancer that can exist in the project. The total number of services of type NodePort that can exist in the project. The total number of secrets that can exist in the project. The total number of ConfigMap objects that can exist in the project. The total number of persistent volume claims that can exist in the project. The total number of imagestreams that can exist in the project. Quota scopesEach quota can have an associated set of scopes. A quota only measures usage for a resource if it matches the intersection of enumerated scopes. Adding a scope to a quota restricts the set of resources to which that quota can apply. Specifying a resource outside of the allowed set results in a validation error. Match pods where spec.activeDeadlineSeconds >= 0 . Match pods where spec.activeDeadlineSeconds is nil . Match pods that have best effort quality of service for either cpu or memory . Match pods that do not have best effort quality of service for cpu and memory . A BestEffort scope restricts a quota to limiting the following resources: A Terminating , NotTerminating , and NotBestEffort scope restricts a quota to tracking the following resources: Ephemeral storage requests and limits apply only if you enabled the ephemeral storage technology preview. This feature is disabled by default. Quota enforcementAfter a resource quota for a project is first created, the project restricts the ability to create any new resources that may violate a quota constraint until it has calculated updated usage statistics. After a quota is created and usage statistics are updated, the project accepts the creation of new content. When you create or modify resources, your quota usage is incremented immediately upon the request to create or modify the resource. When you delete a resource, your quota use is decremented during the next full recalculation of quota statistics for the project. A configurable amount of time determines how long it takes to reduce quota usage statistics to their current observed system value. If project modifications exceed a quota usage limit, the server denies the action, and an appropriate error message is returned to the user explaining the quota constraint violated, and what their currently observed usage statistics are in the system. Requests versus limitsWhen allocating compute resources, each container might specify a request and a limit value each for CPU, memory, and ephemeral storage. Quotas can restrict any of these values. If the quota has a value specified for requests.cpu or requests.memory , then it requires that every incoming container make an explicit request for those resources. If the quota has a value specified for limits.cpu or limits.memory , then it requires that every incoming container specify an explicit limit for those resources. Sample resource quota definitions123456
1 The total number of imagestreams that can exist in the project. 1234567
12
12345
12345
1234567
Creating a quotaYou can create a quota to constrain resource usage in a given project. Define the quota in a file. Use the file to create the quota and apply it to a project: Creating object count quotasYou can create an object count quota for all OpenShift Container Platform standard namespaced resource types, such as BuildConfig , and DeploymentConfig . An object quota count places a defined quota on all standard namespaced resource types. When using a resource quota, an object is charged against the quota if it exists in server storage. These types of quotas are useful to protect against exhaustion of storage resources. To configure an object count quota for a resource: Run the following command: 1 is the name of the resource, and is the API group, if applicable. Use the oc api-resources command for a list of resources and their associated API groups. This example limits the listed resources to the hard limit in each project in the cluster. Verify that the quota was created: Setting resource quota for extended resourcesOvercommitment of resources is not allowed for extended resources, so you must specify requests and limits for the same extended resource in a quota. Currently, only quota items with the prefix requests. is allowed for extended resources. The following is an example scenario of how to set resource quota for the GPU resource nvidia.com/gpu . Determine how many GPUs are available on a node in your cluster. For example: In this example, 2 GPUs are available. Set a quota in the namespace nvidia . In this example, the quota is 1 : Create the quota: Verify that the namespace has the correct quota set: Run a pod that asks for a single GPU: Verify that the pod is running: Verify that the quota Used counter is correct: Attempt to create a second GPU pod in the nvidia namespace. This is technically available on the node because it has 2 GPUs: This Forbidden error message is expected because you have a quota of 1 GPU and this pod tried to allocate a second GPU, which exceeds its quota. Viewing a quotaYou can view usage statistics related to any hard limits defined in a project’s quota by navigating in the web console to the project’s Quota page. You can also use the CLI to view quota details. Get the list of quotas defined in the project. For example, for a project called demoproject : Describe the quota you are interested in, for example the core-object-counts quota: Configuring quota synchronization periodWhen a set of resources are deleted, but before quota usage is restored, a user might encounter problems when attempting to reuse the resources. The synchronization time frame of resources is determined by the resource-quota-sync-period setting, which can be configured by a cluster administrator. Adjusting the regeneration time can be helpful for creating resources and determining resource usage when automation is used. The resource-quota-sync-period setting is designed to balance system performance. Reducing the sync period can result in a heavy load on the master. To configure the quota synchronization period: Edit the Kubernetes controller manager. Change the unsupportedconfigOverrides field to have the following settings, specifying the amount of time, in seconds, for the resource-quota-sync-period field: Источник Читайте также: 403 error forbidden мтс Adblock |
If a project has resource quotas set, and it contains an invalid buildConfig which causes the build controller to create an invalid builder pod, the user will see a misleading «Quota limit has been reached» error message.
This issue has come up together with #15876, but I’m creating a new issue because I believe this is an unrelated error that may also present in different circumstances. The steps to reproduce are the same as with #15876, but this time a resource quota must also be set in the project before deploying the application.
Version
oc v1.5.1
kubernetes v1.5.2+43a9be4
features: Basic-Auth GSSAPI Kerberos SPNEGO
Server https://api.podspace.io:8443
openshift v1.5.1
kubernetes v1.5.2+43a9be4
Steps To Reproduce
- Create a new project and set up the following quota and resource limits inside the project:
$ oc create -f - <<EOF
apiVersion: v1
items:
- apiVersion: v1
kind: LimitRange
metadata:
creationTimestamp: null
name: compute-resources-limitrange
spec:
limits:
- default:
cpu: "1"
memory: 2Gi
defaultRequest:
cpu: 20m
memory: 64Mi
max:
cpu: "4"
memory: 8Gi
min:
cpu: 20m
memory: 64Mi
type: Container
kind: List
metadata: {}
EOF
$ oc create -f - <<EOF
apiVersion: v1
items:
- apiVersion: v1
kind: ResourceQuota
metadata:
creationTimestamp: null
name: compute-resources-quota
spec:
hard:
limits.cpu: "6"
limits.memory: 8Gi
pods: "32"
scopes:
- NotTerminating
status: {}
kind: List
metadata: {}
EOF
- Follow the steps to reproduce build fails to start if a name of a source secret contains a dot #15876 which will create a build config which fails to create a builder pod
Current Result
When the project is deployed after step 2., the user will immediately see a «Quota limit reached» error in the web console. The quota status page will show that the resource limit quotas are indeed depleted.
The project event log will contain these entries at this point:
LASTSEEN FIRSTSEEN COUNT NAME KIND SUBOBJECT TYPE REASON SOURCE MESSAGE
2s 29m 28 hello-1 Build Warning FailedCreate {build-controller } Error creating: Pod "hello-1-build" is invalid: [spec.volumes[2].name: Invalid value: "my.secret-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[2].name: Not found: "my.secret-source"]
2s 29m 28 hello-1 Build Warning HandleBuildError {build-controller } Build has error: failed to create build pod: Pod "hello-1-build" is invalid: [spec.volumes[2].name: Invalid value: "my.secret-source": must match the regex [a-z0-9]([-a-z0-9]*[a-z0-9])? (e.g. 'my-name' or '123-abc'), spec.containers[0].volumeMounts[2].name: Not found: "my.secret-source"]
0s 29m 1739 hello-1 Build Warning FailedCreate {build-controller } Error creating: pods "hello-1-build" is forbidden: exceeded quota: compute-resources-quota, requested: limits.memory=2Gi, used: limits.memory=8Gi, limited: limits.memory=8Gi
0s 29m 1739 hello-1 Build Warning HandleBuildError {build-controller } Build has error: failed to create build pod: pods "hello-1-build" is forbidden: exceeded quota: compute-resources-quota, requested: limits.memory=2Gi, used: limits.memory=8Gi, limited: limits.memory=8Gi
Expected Result
The quota should not be depleted when the build controller fails to create the builder pod. No quota depletion error should be shown to the user, only the real cause of the failure (invalid volume name in this case)
4.1. Resource quotas per project
A resource quota, defined by a ResourceQuota
object, provides constraints that limit aggregate resource consumption per project. It can limit the quantity of objects that can be created in a project by type, as well as the total amount of compute resources and storage that might be consumed by resources in that project.
This guide describes how resource quotas work, how cluster administrators can set and manage resource quotas on a per project basis, and how developers and cluster administrators can view them.
4.1.1. Resources managed by quotas
The following describes the set of compute resources and object types that can be managed by a quota.
A pod is in a terminal state if status.phase in (Failed, Succeeded)
is true.
Table 4.1. Compute resources managed by quota
Resource Name | Description |
---|---|
|
The sum of CPU requests across all pods in a non-terminal state cannot exceed this value. |
|
The sum of memory requests across all pods in a non-terminal state cannot exceed this value. |
|
The sum of local ephemeral storage requests across all pods in a non-terminal state cannot exceed this value. |
|
The sum of CPU requests across all pods in a non-terminal state cannot exceed this value. |
|
The sum of memory requests across all pods in a non-terminal state cannot exceed this value. |
|
The sum of ephemeral storage requests across all pods in a non-terminal state cannot exceed this value. |
|
The sum of CPU limits across all pods in a non-terminal state cannot exceed this value. |
|
The sum of memory limits across all pods in a non-terminal state cannot exceed this value. |
|
The sum of ephemeral storage limits across all pods in a non-terminal state cannot exceed this value. |
Table 4.2. Storage resources managed by quota
Resource Name | Description |
---|---|
|
The sum of storage requests across all persistent volume claims in any state cannot exceed this value. |
|
The total number of persistent volume claims that can exist in the project. |
|
The sum of storage requests across all persistent volume claims in any state that have a matching storage class, cannot exceed this value. |
|
The total number of persistent volume claims with a matching storage class that can exist in the project. |
Table 4.3. Object counts managed by quota
Resource Name | Description |
---|---|
|
The total number of pods in a non-terminal state that can exist in the project. |
|
The total number of ReplicationControllers that can exist in the project. |
|
The total number of resource quotas that can exist in the project. |
|
The total number of services that can exist in the project. |
|
The total number of services of type |
|
The total number of services of type |
|
The total number of secrets that can exist in the project. |
|
The total number of |
|
The total number of persistent volume claims that can exist in the project. |
|
The total number of imagestreams that can exist in the project. |
4.1.2. Quota scopes
Each quota can have an associated set of scopes. A quota only measures usage for a resource if it matches the intersection of enumerated scopes.
Adding a scope to a quota restricts the set of resources to which that quota can apply. Specifying a resource outside of the allowed set results in a validation error.
Scope |
Description |
|
Match pods where |
|
Match pods where |
|
Match pods that have best effort quality of service for either |
|
Match pods that do not have best effort quality of service for |
A BestEffort
scope restricts a quota to limiting the following resources:
-
pods
A Terminating
, NotTerminating
, and NotBestEffort
scope restricts a quota to tracking the following resources:
-
pods
-
memory
-
requests.memory
-
limits.memory
-
cpu
-
requests.cpu
-
limits.cpu
-
ephemeral-storage
-
requests.ephemeral-storage
-
limits.ephemeral-storage
4.1.3. Quota enforcement
After a resource quota for a project is first created, the project restricts the ability to create any new resources that may violate a quota constraint until it has calculated updated usage statistics.
After a quota is created and usage statistics are updated, the project accepts the creation of new content. When you create or modify resources, your quota usage is incremented immediately upon the request to create or modify the resource.
When you delete a resource, your quota use is decremented during the next full recalculation of quota statistics for the project. A configurable amount of time determines how long it takes to reduce quota usage statistics to their current observed system value.
If project modifications exceed a quota usage limit, the server denies the action, and an appropriate error message is returned to the user explaining the quota constraint violated, and what their currently observed usage statistics are in the system.
4.1.4. Requests versus limits
When allocating compute resources, each container might specify a request and a limit value each for CPU, memory, and ephemeral storage. Quotas can restrict any of these values.
If the quota has a value specified for requests.cpu
or requests.memory
, then it requires that every incoming container make an explicit request for those resources. If the quota has a value specified for limits.cpu
or limits.memory
, then it requires that every incoming container specify an explicit limit for those resources.
4.1.5. Sample resource quota definitions
core-object-counts.yaml
apiVersion: v1 kind: ResourceQuota metadata: name: core-object-counts spec: hard: configmaps: "10" 1 persistentvolumeclaims: "4" 2 replicationcontrollers: "20" 3 secrets: "10" 4 services: "10" 5 services.loadbalancers: "2" 6
- 1
-
The total number of
ConfigMap
objects that can exist in the project. - 2
-
The total number of persistent volume claims (PVCs) that can exist in the project.
- 3
-
The total number of replication controllers that can exist in the project.
- 4
-
The total number of secrets that can exist in the project.
- 5
-
The total number of services that can exist in the project.
- 6
-
The total number of services of type
LoadBalancer
that can exist in the project.
openshift-object-counts.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
name: openshift-object-counts
spec:
hard:
openshift.io/imagestreams: "10" 1
- 1
-
The total number of image streams that can exist in the project.
compute-resources.yaml
apiVersion: v1 kind: ResourceQuota metadata: name: compute-resources spec: hard: pods: "4" 1 requests.cpu: "1" 2 requests.memory: 1Gi 3 requests.ephemeral-storage: 2Gi 4 limits.cpu: "2" 5 limits.memory: 2Gi 6 limits.ephemeral-storage: 4Gi 7
- 1
-
The total number of pods in a non-terminal state that can exist in the project.
- 2
-
Across all pods in a non-terminal state, the sum of CPU requests cannot exceed 1 core.
- 3
-
Across all pods in a non-terminal state, the sum of memory requests cannot exceed 1Gi.
- 4
-
Across all pods in a non-terminal state, the sum of ephemeral storage requests cannot exceed 2Gi.
- 5
-
Across all pods in a non-terminal state, the sum of CPU limits cannot exceed 2 cores.
- 6
-
Across all pods in a non-terminal state, the sum of memory limits cannot exceed 2Gi.
- 7
-
Across all pods in a non-terminal state, the sum of ephemeral storage limits cannot exceed 4Gi.
besteffort.yaml
apiVersion: v1 kind: ResourceQuota metadata: name: besteffort spec: hard: pods: "1" 1 scopes: - BestEffort 2
- 1
-
The total number of pods in a non-terminal state with
BestEffort
quality of service that can exist in the project. - 2
-
Restricts the quota to only matching pods that have
BestEffort
quality of service for either memory or CPU.
compute-resources-long-running.yaml
apiVersion: v1 kind: ResourceQuota metadata: name: compute-resources-long-running spec: hard: pods: "4" 1 limits.cpu: "4" 2 limits.memory: "2Gi" 3 limits.ephemeral-storage: "4Gi" 4 scopes: - NotTerminating 5
- 1
-
The total number of pods in a non-terminal state.
- 2
-
Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this value.
- 3
-
Across all pods in a non-terminal state, the sum of memory limits cannot exceed this value.
- 4
-
Across all pods in a non-terminal state, the sum of ephemeral storage limits cannot exceed this value.
- 5
-
Restricts the quota to only matching pods where
spec.activeDeadlineSeconds
is set tonil
. Build pods will fall underNotTerminating
unless theRestartNever
policy is applied.
compute-resources-time-bound.yaml
apiVersion: v1 kind: ResourceQuota metadata: name: compute-resources-time-bound spec: hard: pods: "2" 1 limits.cpu: "1" 2 limits.memory: "1Gi" 3 limits.ephemeral-storage: "1Gi" 4 scopes: - Terminating 5
- 1
-
The total number of pods in a terminating state.
- 2
-
Across all pods in a terminating state, the sum of CPU limits cannot exceed this value.
- 3
-
Across all pods in a terminating state, the sum of memory limits cannot exceed this value.
- 4
-
Across all pods in a terminating state, the sum of ephemeral storage limits cannot exceed this value.
- 5
-
Restricts the quota to only matching pods where
spec.activeDeadlineSeconds >=0
. For example, this quota would charge for build or deployer pods, but not long running pods like a web server or database.
storage-consumption.yaml
apiVersion: v1 kind: ResourceQuota metadata: name: storage-consumption spec: hard: persistentvolumeclaims: "10" 1 requests.storage: "50Gi" 2 gold.storageclass.storage.k8s.io/requests.storage: "10Gi" 3 silver.storageclass.storage.k8s.io/requests.storage: "20Gi" 4 silver.storageclass.storage.k8s.io/persistentvolumeclaims: "5" 5 bronze.storageclass.storage.k8s.io/requests.storage: "0" 6 bronze.storageclass.storage.k8s.io/persistentvolumeclaims: "0" 7
- 1
-
The total number of persistent volume claims in a project
- 2
-
Across all persistent volume claims in a project, the sum of storage requested cannot exceed this value.
- 3
-
Across all persistent volume claims in a project, the sum of storage requested in the gold storage class cannot exceed this value.
- 4
-
Across all persistent volume claims in a project, the sum of storage requested in the silver storage class cannot exceed this value.
- 5
-
Across all persistent volume claims in a project, the total number of claims in the silver storage class cannot exceed this value.
- 6
-
Across all persistent volume claims in a project, the sum of storage requested in the bronze storage class cannot exceed this value. When this is set to
0
, it means bronze storage class cannot request storage. - 7
-
Across all persistent volume claims in a project, the sum of storage requested in the bronze storage class cannot exceed this value. When this is set to
0
, it means bronze storage class cannot create claims.
4.1.6. Creating a quota
You can create a quota to constrain resource usage in a given project.
Procedure
- Define the quota in a file.
-
Use the file to create the quota and apply it to a project:
$ oc create -f <file> [-n <project_name>]
For example:
$ oc create -f core-object-counts.yaml -n demoproject
4.1.6.1. Creating object count quotas
You can create an object count quota for all standard namespaced resource types on OpenShift Container Platform, such as BuildConfig
and DeploymentConfig
objects. An object quota count places a defined quota on all standard namespaced resource types.
When using a resource quota, an object is charged against the quota upon creation. These types of quotas are useful to protect against exhaustion of resources. The quota can only be created if there are enough spare resources within the project.
Procedure
To configure an object count quota for a resource:
-
Run the following command:
$ oc create quota <name> --hard=count/<resource>.<group>=<quota>,count/<resource>.<group>=<quota> 1
- 1
-
The
<resource>
variable is the name of the resource, and<group>
is the API group, if applicable. Use theoc api-resources
command for a list of resources and their associated API groups.
For example:
$ oc create quota test --hard=count/deployments.extensions=2,count/replicasets.extensions=4,count/pods=3,count/secrets=4
Example output
resourcequota "test" created
This example limits the listed resources to the hard limit in each project in the cluster.
-
Verify that the quota was created:
$ oc describe quota test
Example output
Name: test Namespace: quota Resource Used Hard -------- ---- ---- count/deployments.extensions 0 2 count/pods 0 3 count/replicasets.extensions 0 4 count/secrets 0 4
4.1.6.2. Setting resource quota for extended resources
Overcommitment of resources is not allowed for extended resources, so you must specify requests
and limits
for the same extended resource in a quota. Currently, only quota items with the prefix requests.
is allowed for extended resources. The following is an example scenario of how to set resource quota for the GPU resource nvidia.com/gpu
.
Procedure
-
Determine how many GPUs are available on a node in your cluster. For example:
# oc describe node ip-172-31-27-209.us-west-2.compute.internal | egrep 'Capacity|Allocatable|gpu'
Example output
openshift.com/gpu-accelerator=true Capacity: nvidia.com/gpu: 2 Allocatable: nvidia.com/gpu: 2 nvidia.com/gpu 0 0
In this example, 2 GPUs are available.
-
Set a quota in the namespace
nvidia
. In this example, the quota is1
:# cat gpu-quota.yaml
Example output
apiVersion: v1 kind: ResourceQuota metadata: name: gpu-quota namespace: nvidia spec: hard: requests.nvidia.com/gpu: 1
-
Create the quota:
# oc create -f gpu-quota.yaml
Example output
resourcequota/gpu-quota created
-
Verify that the namespace has the correct quota set:
# oc describe quota gpu-quota -n nvidia
Example output
Name: gpu-quota Namespace: nvidia Resource Used Hard -------- ---- ---- requests.nvidia.com/gpu 0 1
-
Define a pod that asks for a single GPU. The following example definition file is called
gpu-pod.yaml
:apiVersion: v1 kind: Pod metadata: generateName: gpu-pod- namespace: nvidia spec: restartPolicy: OnFailure containers: - name: rhel7-gpu-pod image: rhel7 env: - name: NVIDIA_VISIBLE_DEVICES value: all - name: NVIDIA_DRIVER_CAPABILITIES value: "compute,utility" - name: NVIDIA_REQUIRE_CUDA value: "cuda>=5.0" command: ["sleep"] args: ["infinity"] resources: limits: nvidia.com/gpu: 1
-
Create the pod:
# oc create -f gpu-pod.yaml
-
Verify that the pod is running:
# oc get pods
Example output
NAME READY STATUS RESTARTS AGE gpu-pod-s46h7 1/1 Running 0 1m
-
Verify that the quota
Used
counter is correct:# oc describe quota gpu-quota -n nvidia
Example output
Name: gpu-quota Namespace: nvidia Resource Used Hard -------- ---- ---- requests.nvidia.com/gpu 1 1
-
Attempt to create a second GPU pod in the
nvidia
namespace. This is technically available on the node because it has 2 GPUs:# oc create -f gpu-pod.yaml
Example output
Error from server (Forbidden): error when creating "gpu-pod.yaml": pods "gpu-pod-f7z2w" is forbidden: exceeded quota: gpu-quota, requested: requests.nvidia.com/gpu=1, used: requests.nvidia.com/gpu=1, limited: requests.nvidia.com/gpu=1
This Forbidden error message is expected because you have a quota of 1 GPU and this pod tried to allocate a second GPU, which exceeds its quota.
4.1.7. Viewing a quota
You can view usage statistics related to any hard limits defined in a project’s quota by navigating in the web console to the project’s Quota page.
You can also use the CLI to view quota details.
Procedure
-
Get the list of quotas defined in the project. For example, for a project called
demoproject
:$ oc get quota -n demoproject
Example output
NAME AGE besteffort 11m compute-resources 2m core-object-counts 29m
-
Describe the quota you are interested in, for example the
core-object-counts
quota:$ oc describe quota core-object-counts -n demoproject
Example output
Name: core-object-counts Namespace: demoproject Resource Used Hard -------- ---- ---- configmaps 3 10 persistentvolumeclaims 0 4 replicationcontrollers 3 20 secrets 9 10 services 2 10
4.1.8. Configuring explicit resource quotas
Configure explicit resource quotas in a project request template to apply specific resource quotas in new projects.
Prerequisites
- Access to the cluster as a user with the cluster-admin role.
-
Install the OpenShift CLI (
oc
).
Procedure
-
Add a resource quota definition to a project request template:
-
If a project request template does not exist in a cluster:
-
Create a bootstrap project template and output it to a file called
template.yaml
:$ oc adm create-bootstrap-project-template -o yaml > template.yaml
-
Add a resource quota definition to
template.yaml
. The following example defines a resource quota named ‘storage-consumption’. The definition must be added before theparameters:
section in the template:- apiVersion: v1 kind: ResourceQuota metadata: name: storage-consumption spec: hard: persistentvolumeclaims: "10" 1 requests.storage: "50Gi" 2 gold.storageclass.storage.k8s.io/requests.storage: "10Gi" 3 silver.storageclass.storage.k8s.io/requests.storage: "20Gi" 4 silver.storageclass.storage.k8s.io/persistentvolumeclaims: "5" 5 bronze.storageclass.storage.k8s.io/requests.storage: "0" 6 bronze.storageclass.storage.k8s.io/persistentvolumeclaims: "0" 7
- 1
-
The total number of persistent volume claims in a project.
- 2
-
Across all persistent volume claims in a project, the sum of storage requested cannot exceed this value.
- 3
-
Across all persistent volume claims in a project, the sum of storage requested in the gold storage class cannot exceed this value.
- 4
-
Across all persistent volume claims in a project, the sum of storage requested in the silver storage class cannot exceed this value.
- 5
-
Across all persistent volume claims in a project, the total number of claims in the silver storage class cannot exceed this value.
- 6
-
Across all persistent volume claims in a project, the sum of storage requested in the bronze storage class cannot exceed this value. When this value is set to
0
, the bronze storage class cannot request storage. - 7
-
Across all persistent volume claims in a project, the sum of storage requested in the bronze storage class cannot exceed this value. When this value is set to
0
, the bronze storage class cannot create claims.
-
Create a project request template from the modified
template.yaml
file in theopenshift-config
namespace:$ oc create -f template.yaml -n openshift-config
To include the configuration as a
kubectl.kubernetes.io/last-applied-configuration
annotation, add the--save-config
option to theoc create
command.By default, the template is called
project-request
.
-
-
If a project request template already exists within a cluster:
If you declaratively or imperatively manage objects within your cluster by using configuration files, edit the existing project request template through those files instead.
-
List templates in the
openshift-config
namespace:$ oc get templates -n openshift-config
-
Edit an existing project request template:
$ oc edit template <project_request_template> -n openshift-config
-
Add a resource quota definition, such as the preceding
storage-consumption
example, into the existing template. The definition must be added before theparameters:
section in the template.
-
-
-
If you created a project request template, reference it in the cluster’s project configuration resource:
-
Access the project configuration resource for editing:
-
By using the web console:
- Navigate to the Administration → Cluster Settings page.
- Click Global Configuration to view all configuration resources.
- Find the entry for Project and click Edit YAML.
-
By using the CLI:
-
Edit the
project.config.openshift.io/cluster
resource:$ oc edit project.config.openshift.io/cluster
-
-
-
Update the
spec
section of the project configuration resource to include theprojectRequestTemplate
andname
parameters. The following example references the default project request template nameproject-request
:apiVersion: config.openshift.io/v1 kind: Project metadata: ... spec: projectRequestTemplate: name: project-request
-
-
Verify that the resource quota is applied when projects are created:
-
Create a project:
$ oc new-project <project_name>
-
List the project’s resource quotas:
$ oc get resourcequotas
-
Describe the resource quota in detail:
$ oc describe resourcequotas <resource_quota_name>
-
4.2. Resource quotas across multiple projects
A multi-project quota, defined by a ClusterResourceQuota
object, allows quotas to be shared across multiple projects. Resources used in each selected project are aggregated and that aggregate is used to limit resources across all the selected projects.
This guide describes how cluster administrators can set and manage resource quotas across multiple projects.
4.2.1. Selecting multiple projects during quota creation
When creating quotas, you can select multiple projects based on annotation selection, label selection, or both.
Procedure
-
To select projects based on annotations, run the following command:
$ oc create clusterquota for-user --project-annotation-selector openshift.io/requester=<user_name> --hard pods=10 --hard secrets=20
This creates the following
ClusterResourceQuota
object:apiVersion: v1 kind: ClusterResourceQuota metadata: name: for-user spec: quota: 1 hard: pods: "10" secrets: "20" selector: annotations: 2 openshift.io/requester: <user_name> labels: null 3 status: namespaces: 4 - namespace: ns-one status: hard: pods: "10" secrets: "20" used: pods: "1" secrets: "9" total: 5 hard: pods: "10" secrets: "20" used: pods: "1" secrets: "9"
- 1
-
The
ResourceQuotaSpec
object that will be enforced over the selected projects. - 2
-
A simple key-value selector for annotations.
- 3
-
A label selector that can be used to select projects.
- 4
-
A per-namespace map that describes current quota usage in each selected project.
- 5
-
The aggregate usage across all selected projects.
This multi-project quota document controls all projects requested by
<user_name>
using the default project request endpoint. You are limited to 10 pods and 20 secrets. -
Similarly, to select projects based on labels, run this command:
$ oc create clusterresourcequota for-name 1 --project-label-selector=name=frontend 2 --hard=pods=10 --hard=secrets=20
- 1
-
Both
clusterresourcequota
andclusterquota
are aliases of the same command.for-name
is the name of theClusterResourceQuota
object. - 2
-
To select projects by label, provide a key-value pair by using the format
--project-label-selector=key=value
.
This creates the following
ClusterResourceQuota
object definition:apiVersion: v1 kind: ClusterResourceQuota metadata: creationTimestamp: null name: for-name spec: quota: hard: pods: "10" secrets: "20" selector: annotations: null labels: matchLabels: name: frontend
4.2.2. Viewing applicable cluster resource quotas
A project administrator is not allowed to create or modify the multi-project quota that limits his or her project, but the administrator is allowed to view the multi-project quota documents that are applied to his or her project. The project administrator can do this via the AppliedClusterResourceQuota
resource.
Procedure
-
To view quotas applied to a project, run:
$ oc describe AppliedClusterResourceQuota
Example output
Name: for-user Namespace: <none> Created: 19 hours ago Labels: <none> Annotations: <none> Label Selector: <null> AnnotationSelector: map[openshift.io/requester:<user-name>] Resource Used Hard -------- ---- ---- pods 1 10 secrets 9 20
4.2.3. Selection granularity
Because of the locking consideration when claiming quota allocations, the number of active projects selected by a multi-project quota is an important consideration. Selecting more than 100 projects under a single multi-project quota can have detrimental effects on API server responsiveness in those projects.
In this post we will be diving in to quotas and limits. Quotas and limits can be used to allocate, limit en reserve resources within your cluster for one or multiple namepaces and can limit the usage of CPU and Memory for workloads. Limits can also be used to gurante (by reserving) resources for your workloads.
Understanding and using quotas and limits from day one can help plan the resource growth of your cluster and prevents unwanted resource depletion. Using these API objects helps your cluster to more effectively schedule workloads across your cluster while also making sure they get their piece of resources when they need it.
As always, these concepts apply to both Kubernetes and Openshift. We will try to do everything from the oc
command line.
Controlling resources allocation
We have options to control how many resources namespaces and pods are allowed to use or create in a cluster. This comes in handy to prevent unwanted growth of a namespace and prevent depletion of vital compute resources.
Let’s start with a overview:
- Limits (api: can set the limits and request of a workload. With a limit we can limit a workload to use a maximum amount of CPU or request a minimum amount of CPU. You can find limits under
pods.spec.containers.resource
- Limit is the maximum amount of a resource that can be used. Found under
pods.spec.containers.resource.limits
- Request is the amount of a resource that is reserved. Requests are set under
pods.spec.containers.resource.requests
- Limit is the maximum amount of a resource that can be used. Found under
- Limit Ranges are great because they assign default limits and requests to pods that don’t have any and can set a threshold on how much every individual pod can use. This gives you finer control on resource allocation then a Quota because a quota looks at all requests in a namespace. Limit ranges use the API object
limitrange
- Quotas are used to set how many resources a namespace can create and/or request. We can limit the amount of pods that can run in a single namespace. We can also use a quota to limit the amount of CPU all workloads can use by enforcing the usage of a CPU limit on each pod or the amount of memory the workloads can request. A quota is created of the type
resourcequota
- Cluster Quotas can be used across the Openshift cluster and are not bound to a single namespace. A Cluster Qouta of the object
clusterresourcequota
What do the units mean?
When we request or limit compute resources we do this on Memory
and CPU
. The amount that we want is expressed in units in different ways in yaml
:
- CPU A single virtual CPU is always a
1
inyaml
or1000m
. We can use the smaller unit to divide and share the CPU computing power between workloads. The actual clock speed of the CPU in this case is not relevant but could have an impact when you move to another cluster or CPU type 1. - Memory Unlike CPU memory is expressed in bytes. You can use a fixed number like
1500M
or a power-of-two equivalent like512Mi
[^MemoryInKubernetes]. We will use the later in our examples.
Understanding Limits
A limit can consist of an upper value (limit) and lower (request) threshold that you set on a pod or container. Limits on individual pods are usually baked in to the yaml
by most applications (you’ll see a lot of cpu: 10m
requests in containers, see below for why). Most of the time we will set a limit for a complete deployment.
Limits on a pod can look like this on a deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
....
spec:
....
spec:
containers:
....
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 10m
memory: 128Mi
The request part
A part of a limit is that you can also set a request. This tells the scheduler that in order to run your workload you’ll need a certain amount of CPU or Memory. The scheduler will place your pods on a node that has this amount free and the amount you request is held in reserver for you. A lot of pods have a CPU request of 10m
. So thats 1/100 of a vCPU. The reason for this is that they often don’t use a lot of CPU but in the case of a high load on a node they would still receive some CPU from the node.
Some things to keep in mind:
- If you set no requests on a pod and the node on which the pod is running is getting busy your pod might not receive any CPU or Memory. Why? Because you didn’t tell the cluster you needed any
- If you set only a limit and no request of a resource type the cluster will take your limit and turn it in to the request. So if you set a high limit, be sure to set the right request so that your pod wont eat up all resources 2.
- If you set a request but no limit the cluster will try to default to the values set in a
limitrange
or the cluster default setting 3.
Hitting a limit
So we set a limit. Everything is running but then al of sudden cHaOs. Pods go wild, workloads go crazy! Your whole namespace is on fire! 🔥 Resources are being eaten up by different pods. CPU and memory usage is climbing! What will happen now?
- Hitting the CPU Limit No worries, hitting a CPU limit is not cause for panic. Your pods will be throttled by the cluster. This might have some unforseen impact on perfomance but in most cases your fine
- Hitting the Memory Limit Now this is a bit more tricky. When you hit your memory limit in a cluster the cluster will kill the workout with the classic line
OOM
(Out Of Memory). Even if the node has48Gi
of Memory and is not using half of it. If your pod hits your defined Memory limit it is killed 4.
Understanding Limit Ranges
A limit
applies a limit (or request) of resources on a pod. A limitrange
can apply a minimum, maximum or default value for limits and requests of a pod or container in your namespace. If for example you want to have all pods in a namespace request a minimum of 128Mi
of Memory you can create a limitrange
to do this.
Limits on a whole Namespaces
As we said before. We set limits on pods (using the deployment) or on individual containers. But how do we control the total usage of CPU and Memory of a namespaces? But how do we do this? By using a quota!
Understanding Quotas
Quotas can be used to limit the amount of resources that can be created, requested or used. Usually quotas are set on a namespace basis (so they apply to the sum of the resources in the namespace) but they can also be set on a group of namespaces or even on a per user basis.
Limits can for example:
- Control the maximum amount of pods that can run
- Limit the amount of routes used by a namespace
- Control how many services can be created
- Set the maximum amount of CPU all pods can request
- Set a maximum amount of Memory that can be used at once
In short “Quotas can be used to apply limits”.
A quota can look like this
apiVersion: v1
kind: ResourceQuota
metadata:
name: my-quota
namespace: a-quota-for-me
spec:
hard:
cpu: "1"
memory: 1G
persistentvolumeclaims: "10"
pods: "2"
replicationcontrollers: "2"
resourcequotas: "1"
secrets: "5"
services: "3"
cpu
and memory
in this example relate to requests. To set a limit use limits.cpu
and limits.memory
. You can also go 100% correct and use requests.cpu
and requests.memory
and save yourself some troubleshooting time in the future.
Understanding quota ranges
A quota can be crated with two different scopes:
resourcequota
Is created withoc create quota
clusterresourcequota
Is created withoc create clusterresourcequota
So what are the differences?
Range of resourcequota
A quota of the type resourcequota
is designed to apply a quota of resources to one project or namespace. You could add this resource to the default template of your Openshift projects creation to apply a default quota to all new projects that are created.
The quota is created on a project/namespace level.
Range of clusterresourcequota
As the name suggets, cluster quotas (clusterresourcequota
) are created and designed to work across you cluster. The way they are applied is by using selectors that matches to a label on a project or by using the openshift.io/requester
annotation to link them to a project owner. This can come in handy when you are also using the cluster for developers to experiment on and don’t want them to eat up all the resources. Be sure to update the openshift.io/requester
value on other projects.
Cumulative Quotas
You can create (or apply) multible quota’s to a single namespace (but this is not recomendend). If you do, the effect is cummulative (meaning all quota’s will be merged in to one). If you specify the quota for a specifiek resourcetype (like pods) in multible quota’s the lowest value will be used. For example:
$ oc get resourcequotas
NAME AGE REQUEST LIMIT
quota-1 3m pods: 5/10, secrets: 9/10
quota-2 3m configmaps: 2/10, pods: 5/5
2 Quota’s are set: quota-1
and quota-2
. One specifies a limit on secrets and the other on configmaps. This is added up, you an create 10 configmaps and secrets.
Both quota’s specify a maximum number of running pods (max: 5
and max: 10
). The lowest one takes precedence. So a maximum of 5 pods can run.
Failure due to quotas
When you set a quota and you try to start things up that exceed that quota some strange things might happen, like:
- When you set a quota of 1 pod per namespace and you try to do a rolling deployment you might encounter some issues. This is because for a rolling deployment to work two (or more depending on the amount of replicas) are required. This is not allowed by your quota so it will get stuck
- If you try to start up a pod with no requests or limits but you have set resource limits in your quota the pods will not start (because it has no limits set)
- When exceeding the amount of Memory that is allowed by your quota Openshift will give you some time to adjust in stead of killing your pod.
Creating and viewing Limits and Requests
Lets get bussy on the command line and create some limits and request
Limits on Pods
Lets create a namespace called limit-the-sky
with a demo app called airplane
:
$ oc new-project limit-the-sky
$ oc new-app --name airplane --image bitnami/nginx
Now lets do something crazy. Let’s set a request thats way to high for our current cluster:
$ oc set resources deployment airplane
--requests cpu=12
deployment.apps/airplane resource requirements updated
What did we do? We set a request for our app for 12 whole vCPU’s. Let’s see if that flies:
$ oc get pods
NAME READY STATUS RESTARTS AGE
airplane-5d8d87c8d5-lsqsd 0/1 Pending 0 58s
airplane-779789c8cc-2sfqz 1/1 Running 0 2m32s
$ oc describe pod airplane-5d8d87c8d5-lsqsd
....
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 24s (x2 over 91s) default-scheduler 0/1 nodes are available: 1 Insufficient cpu.
As we can see that just won’t fly in the current cluster. Now lets set something more realistlcy. We are going to ask our cluster for 10m
CPU and 128Mi
of memory and we are going to set a upper limit of a half vCPU (500m
or 0.5
) and a maximum amount of Memory of 512Mi
:
$ oc set resources deployment airplane
--requests cpu=10m,memory=128Mi
--limits cpu=500m,memory=512Mi
deployment.apps/airplane resource requirements updated
That flies:
$ oc get pods
NAME READY STATUS RESTARTS AGE
airplane-59cbf468f7-w52xq 1/1 Running 0 18s
We can always check the limit of a deployment using the oc describe
command:
$ oc describe deployment airplane
....
Pod Template:
Labels: deployment=airplane
Annotations: openshift.io/generated-by: OpenShiftNewApp
Containers:
airplane:
Image: bitnami/nginx@sha256:78cb209a82fca83aee2f2d71f7115165f911acf1fcc6ce48e1c8bddeb5191049
Ports: 8080/TCP, 8443/TCP
Host Ports: 0/TCP, 0/TCP
Limits:
cpu: 500m
memory: 512Mi
Requests:
cpu: 10m
memory: 128Mi
Environment: <none>
Mounts: <none>
Volumes: <none>
Setting default, min and max with a limitrange
Now, you don’t want to manually add limits and requests to everything. We can take care of this by creating a limitrange
. A limitrange can’t entirly be created from CLI but the setup is pretty easy (tip: use oc explain limitrange
to see the available fields):
# limit-range.yaml
apiVersion: v1
kind: LimitRange
metadata:
name: a-limit-range
namespace: limit-the-sky
spec:
limits:
- type: Container
default:
cpu: 50m
max:
cpu: 500m
min:
cpu: 10m
$ oc apply -f limit-range.yaml
limitrange/a-limit-range created
And let’s have a look at it:
$ oc describe limitranges a-limit-range
Name: a-limit-range
Namespace: limit-the-sky
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Container cpu 10m 500m 50m 50m
And now, when we crate a new deployment that has no CPU limits:
$ oc new-app --name zeplin --image bitnami/nginx
And let’s have a look:
$ oc get pods zeplin-b79b4dcf6-jgtkh -o yaml | grep cpu
kubernetes.io/limit-ranger: 'LimitRanger plugin set: cpu request for container
zeplin; cpu limit for container zeplin'
cpu: 50m
cpu: 50m
What happens if we try to go above (or below) the limit?
$ oc set resources deployment/zeplin --limits cpu=1000m
deployment.apps/zeplin resource requirements updated
That’s strange? We set the mac allowed CPU to 500m
right? So, why no error? So, there is actually an error but it’s hard to find. You can see the error using oc get events
, by doing a oc get replicasets
or by using oc describe deployment zeplin
. You will see that the cluster is refusing to rollout the latest replicaset because of the limitrange:
$ oc get events | grep cpu
....
7m33s Warning FailedCreate replicaset/zeplin-7dc7446698 Error creating: pods "zeplin-7dc7446698-cpqdh" is forbidden: maximum cpu usage per Container is 500m, but limit is 1k
7m32s Warning FailedCreate replicaset/zeplin-7dc7446698 Error creating: pods "zeplin-7dc7446698-cdxdm" is forbidden: maximum cpu usage per Container is 500m, but limit is 1k
6m11s Warning FailedCreate replicaset/zeplin-7dc7446698 (combined from similar events): Error creating: pods "zeplin-7dc7446698-95g7s" is forbidden: maximum cpu usage per Container is 500m, but limit is 1k
Or in the deployment:
$ oc describe deployment zeplin
....
Conditions:
Type Status Reason
---- ------ ------
ReplicaFailure True FailedCreate
Available True MinimumReplicasAvailable
Progressing True ReplicaSetUpdated
OldReplicaSets: zeplin-b79b4dcf6 (1/1 replicas created)
NewReplicaSet: zeplin-6f58fb66cd (0/1 replicas created)
Creating and viewing Quotas
Now lets take a step back and look at a the bigger picture. It’s great to set limits and requests on pods but this is a cluster we are using. So lets step it up. We are going to create a namespace a-quota-for-me
with a our quota called my-quota
:
$ oc new-project a-quota-for-me
$ oc create quota my-quota
--hard=cpu=1,memory=1G,pods=2,secrets=1
resourcequota/my-quota created
Let’s have a look at our quota:
$ oc describe resourcequotas my-quota
Name: my-quota
Namespace: a-quota-for-me
Resource Used Hard
-------- ---- ----
cpu 0 1
memory 0 1G
pods 0 2
secrets 9 1
Thats strange! 9 secrets in use? But we set a limit to 1? This can happen because we set the quota after the creation of these secrets. A quota enforces its restrictions on new resources and not on existing. For example, when we try to create a secret now:
$ oc create secret generic just-try-it --from-literal key1=password
error: failed to create secret secrets "just-try-it" is forbidden: exceeded quota: my-quota, requested: secrets=1, used: secrets=9, limited: secrets=1
Applying quota to a deployment
Now lets run a demo application called you-can-quota-me-on-that
:
$ oc new-app
--name you-can-quota-me-on-that
--image bitnami/nginx
--as-deployment-config
But our pod is nowhere to be found:
$ oc get pods
No resources found in a-quota-for-me namespace
Why? Because we set a resource limit in the quota (a CPU and Memory max) and our pod does not have one:
$ oc describe deploymentconfigs you-can-quota-me-on-that
....
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal DeploymentCreated 4m20s deploymentconfig-controller Created new replication controller "you-can-quota-me-on-that-1" for version 1
Warning FailedRetry 2m42s deployer-controller Stop retrying: couldn't create deployer pod for "a-quota-for-me/you-can-quota-me-on-that-1": pods "you-can-quota-me-on-that-1-deploy" is forbidden: failed quota: my-quota: must specify cpu,memory
Warning FailedCreate 95s (x24 over 4m20s) deployer-controller Error creating deployer pod: pods "you-can-quota-me-on-that-1-deploy" is forbidden: failed quota: my-quota: must specify cpu,memory
Or:
$ oc get events
61s Warning FailedCreate deploymentconfig/you-can-quota-me-on-that Error creating deployer pod: pods "you-can-quota-me-on-that-1-deploy" is forbidden: failed quota: my-quota: must specify cpu,memory
So lets set a request like in our previous example:
$ oc set resources deploymentconfig you-can-quota-me-on-that
--requests cpu=10m,memory=56Mi
deploymentconfig.apps.openshift.io/you-can-quota-me-on-that resource requirements updated
But that won’t do the trick. Why? Because we set the requests on the pod and the error message is telling us the deployer pod is no allowed. So lets edit that by hand. To do that we add the requests under the deployment strategy:
$ oc edit deploymentconfigs you-can-quota-me-on-that
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
deploymentconfig: you-can-quota-me-on-that
strategy:
activeDeadlineSeconds: 21600
resources: {}
rollingParams:
intervalSeconds: 1
maxSurge: 25%
maxUnavailable: 25%
timeoutSeconds: 600
updatePeriodSeconds: 1
type: Rolling
resources:
requests:
cpu: 10m
memory: 56Mi
Now lets rollout the latest version:
$ oc rollout latest you-can-quota-me-on-that
deploymentconfig.apps.openshift.io/you-can-quota-me-on-that rolled out
NAME READY STATUS RESTARTS AGE
you-can-quota-me-on-that-3-deploy 0/1 Completed 0 24s
you-can-quota-me-on-that-3-k2pl7 1/1 Running 0 18s
Hitting quota limits
Let’s scale that up to 4 replicas:
$ oc scale deploymentconfig you-can-quota-me-on-that --replicas 4
deploymentconfig.apps.openshift.io/you-can-quota-me-on-that scaled
$ oc get pods
NAME READY STATUS RESTARTS AGE
you-can-quota-me-on-that-3-deploy 0/1 Completed 0 70s
you-can-quota-me-on-that-3-k2pl7 1/1 Running 0 64s
you-can-quota-me-on-that-3-v5xnp 1/1 Running 0 10s
That’s stange. Where are the pods? As mentioned before. A quota denies deployment of resources that exceed the quota. We have requested four pods to be created but the cluster will only give use two.
We can see the error in the replicationcontroller
$ oc describe replicationcontrollers you-can-quota-me-on-that-3
....
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 2m35s replication-controller Error creating: pods "you-can-quota-me-on-that-3-tcn57" is forbidden: exceeded quota: my-quota, requested: pods=1, used: pods=2, limited: pods=2
Warning FailedCreate 2m35s replication-controller Error creating: pods "you-can-quota-me-on-that-3-w9wp6" is forbidden: exceeded quota: my-quota, requested: pods=1, used: pods=2, limited: pods=2
Warning FailedCreate 25s (x8 over 2m34s) replication-controller (combined from similar events): Error creating: pods "you-can-quota-me-on-that-3-zmp82" is forbidden: exceeded quota: my-quota, requested: pods=1, used: pods=2,
Creating a clusterresourcequota
We can create a cluster quota that wil target our user anna with the following command5:
$ oc create clusterquota for-user-anna
--project-annotation-selector openshift.io/requester=anna
--hard pods=10
--hard secrets=20
To see a resourcequota that is applied to the namespace use:
$ oc describe AppliedClusterResourceQuota
Name: for-user-anna
Created: 2 minutes ago
Labels: <none>
Annotations: <none>
Namespace Selector: ["annas-project"]
Label Selector:
AnnotationSelector: map[openshift.io/requester:anna]
Resource Used Hard
-------- ---- ----
pods 10 10
secrets 9 20
Wrapping it up
This post turned out rather long! Lets review the information with some examples:
- We use
Limits
to set a limit (the maximum amount of a resource that a pod can use) and requests (the amount of resources a pod needs and is ensured) on containers and pods.- You can use a
limit
to set the maximum amount of CPU usage by a pod or container to 1 CPU - You can ensure a pod or container always gets
512Mi
of Memory using arequest
- You can use a
- We use
LimitRanges
to set a default, minimum or maximum of the resources each pod can get.- We can use a
LimitRange
to set a default Memory request for all pods - A
LimitRange
can limit the maximum amount of CPU a pod can request to200m
- We can use a
- We use
ResourceQuota
to set the maximum number of objects in a namespace or to enforce the usage of a limit and the total amount of resources a namespace can use- A quota in a namespace can limit the sum of all CPU requests to 2 CPU’s (
2000m
or2
) - You can limit the amount of routes you can have in a namespace to 1
- You can force pod’s to have a set limit or request
- A quota in a namespace can limit the sum of all CPU requests to 2 CPU’s (
- And we can use a
ClusterResourceQuota
to span a quota over multiple namespaces based on a field tag or the requester (owner) of a project.- With a cluster quota we can set a limit of 20 pods that can be running by a user
- We can limit the amount of CPU used to 4 (
4000m
) of several namespaces that we have tagd asproduction=true
All in all, pretty usefull stuf. Be sure to practice with this and run in to strange situations!
Do you want to see more of my EX280 post’s? Checkout my EX280 page
ResourceQuota is an object in Kubernetes that enables administrators to restrict cluster tenants’ resource usage per namespace. Namespaces create virtual clusters within a physical Kubernetes cluster to help users avoid resource naming conflicts and manage capacity, among others.
In this article, we explore Kubernetes resource quota configuration requirements to govern resource consumption by namespace. We have included demo codes illustrating how to use .yaml files with kubectl to configure constraints on hardware usage.
Resource Quota Overview
A Kubernetes cluster has a limited amount of available hardware resources. Hardware resources are measured based on worker nodes with a specific number of CPU cores or RAM allocation.
In a shared Kubernetes environment, it is important to pre-define the allocation of resources for each tenant to avoid unintended resource contention and depletion.
By utilizing this method, each tenant in a shared Kubernetes cluster can be configured with specific settings for maximum allowed resource consumption.
To put resource limits on individual containers, use the Kubernetes resource requests and limits associated with pods within each namespace.
Namespace Resource Quotas vs. Pod Requests & Limits
A cluster can contain multiple namespaces intended for administrative separation. Each namespace can in turn contain multiple pods of containers. The namespace and the pod each have their own resource configuration files. The namespace resource quota governs the maximum use of computing resources by the namespace, while the pod request and limits govern the use of computing resources by the containers within each pod.
A Kubernetes resource request establishes the amount of computing resources a pod is reserving for itself at creation time. This configuration is important because it helps the Kubernetes scheduler make smart decisions by placing an appropriate number of pods on each node. The resource limit is the upper bound on the CPU or RAM usage a pod can possibly use.
A pod quota can be defined based on resource request and limit values in a YAML file referenced by the Kubernetes JSON API.
How Resource Quota Limits Work
Requests for the creation, deletion, and update of system resources go through the Kubernetes API server. There are different admission controllers that can view and filter the requests. The quota operates until the resource limit is reached or violated.
Once the resource quota object is defined on a namespace by the Kubernetes cluster administrator, the Kubernetes quota admission controller watches for the new objects created in that namespace. Then it will keep monitoring and tracking resource usage.
If a user or process tries to acquire more of that resource, the Kubernetes admission controller will throw an error or exception and will not allow that operation.
INFOGRAPHIC POSTER
Do you know the 12 risks of Kubernetes resource management?
Kubernetes architects and engineers immediately find value in seeing the spread of resource risks. Download the 12 Risks of K8s Resources poster now!
Practical Usage of Resource Quotas
The following points summarize the main steps to understanding how Kubernetes resource quota configuration works in practice:
- User teams are assigned to different namespaces to deploy resources.
- The administrator creates one ResourceQuota object for each namespace.
- Users create resources such as pods and services in their assigned namespace.
- The quota system tracks total system resource usage to ensure it does not exceed the hard resource limits defined in the ResourceQuota object.
- If an API request tries to create or update a resource that violates a quota, the request will fail with a message explaining the constraint violation.
- If the quota is enabled in a namespace for computing resources like CPU and RAM, authorized users must specify requests or limits for those values.
- Without user authorization, the quota system will not allow pod creation.
ResourceQuota object support is enabled by default on most Kubernetes distributions. It can be enabled manually by setting the API server. For example, the command --enable-admission-plugins= flag
has the ResourceQuota object as its target argument.
A resource quota is enforced in a particular namespace when there is a ResourceQuota object in that namespace. This file can be utilized in different ways in configuration.
How to use the Resource Quota
In this section, we go through a demo example of how to create and define the CPU resource quota on a namespace with requests and limits.
We will then try to exceed the quota through web traffic spikes to show that once the defined object limit is reached, no more resources using that quota can be created.
Demo Code
First, check to see if you have an active Kubernetes cluster up and running. We have some worker nodes available for the code. At least one worker node is required to perform this demo.
C02W84XMHTD5:terraform-dev iahmad$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
autoscale-concourse-vmxh Ready <none> 4m v1.21.2
autoscale-default-vm37 Ready <none> 56m v1.21.2
Next, create a namespace to test Kubernetes resource quotas, quota-demo:
C02W84XMHTD5:terraform-dev iahmad$ kubectl create namespace quota-demo
namespace/quota-demo created
Then, define and create the CPU quota on a namespace:
C02W84XMHTD5:terraform-dev iahmad$ cat cpu-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
name: test-cpu-quota
namespace: quota-demo
spec:
hard:
requests.cpu: "200m"
limits.cpu: "300m"
C02W84XMHTD5:terraform-dev iahmad$
C02W84XMHTD5:terraform-dev iahmad$ kubectl create -f cpu-quota.yaml
resourcequota/test-cpu-quota created
You can verify the ResourceQuota object has been created. Note: the “used” column, as initially no quota is used in the configuration and the namespace is populated as empty.
Automated, Intelligent Container Sizing
Kubernetes Vertical Pod Autoscaling doesn’t recommend pod limit values or consider I/O. Densify identifies mis-provisioned containers at a glance and prescribes the optimal configuration.
Densify has partnered with Intel to offer one year of free resource optimization software licensing to qualified companies.
C02W84XMHTD5:terraform-dev iahmad$ kubectl describe resourcequota/test-cpu-quota --namespace quota-demo
Name: test-cpu-quota
Namespace: quota-demo
Resource Used Hard
-------- ---- ----
limits.cpu 0 300m
requests.cpu 0 200m
Thereafter, you can create a test pod with requests and limits defined as shown below:
C02W84XMHTD5:terraform-dev iahmad$ kubectl create -n quota-demo -f- <<EOF
apiVersion: v1
kind: Pod
metadata:
name: testpod1
spec:
containers:
- name: quota-test
image: busybox
imagePullPolicy: IfNotPresent
command: ['sh', '-c', 'echo Pod is Running ; sleep 5000']
resources:
requests:
cpu: "100m"
limits:
cpu: "200m"
restartPolicy: Never
EOF
pod/testpod1 created
See the updated settings on the namespace and note the data displayed in the “used” column. You will now notice a difference, with the new pod just created having used some of the quota:
C02W84XMHTD5:terraform-dev iahmad$ kubectl describe resourcequota/test-cpu-quota --namespace quota-demo
Name: test-cpu-quota
Namespace: quota-demo
Resource Used Hard
-------- ---- ----
limits.cpu 200m 300m
requests.cpu 100m 200m
To continue the process, create another pod and observe the remaining quota values:
C02W84XMHTD5:terraform-dev iahmad$ kubectl create -n quota-demo -f- <<EOF
apiVersion: v1
kind: Pod
metadata:
name: testpod2
spec:
containers:
- name: quota-test
image: busybox
imagePullPolicy: IfNotPresent
command: ['sh', '-c', 'echo Pod is Running ; sleep 5000']
resources:
requests:
cpu: "10m"
limits:
cpu: "20m"
restartPolicy: Never
EOF
pod/testpod2 created
C02W84XMHTD5:terraform-dev iahmad$
C02W84XMHTD5:terraform-dev iahmad$
C02W84XMHTD5:terraform-dev iahmad$
C02W84XMHTD5:terraform-dev iahmad$ kubectl describe resourcequota/test-cpu-quota --namespace quota-demo
Name: test-cpu-quota
Namespace: quota-demo
Resource Used Hard
-------- ---- ----
limits.cpu 220m 300m
requests.cpu 110m 200m
As noted above, the used column is now updated again. The new pod is now listed as having consumed more of the available quota limits for the total allocated CPU resources.
If we try to create a pod requesting more resources than what’s available in quota, we will now receive an error message that states we don’t have enough quota left to create the new pod:
C02W84XMHTD5:terraform-dev iahmad$ kubectl create -n quota-demo -f- <<EOF
apiVersion: v1
kind: Pod
metadata:
name: testpod3
spec:
containers:
- name: quota-test
image: busybox
imagePullPolicy: IfNotPresent
command: ['sh', '-c', 'echo Pod is Running ; sleep 5000']
resources:
requests:
cpu: "100m"
limits:
cpu: "200m"
restartPolicy: Never
EOF
Error from server (Forbidden): error when creating "STDIN": pods "testpod3" is forbidden: exceeded quota: test-cpu-quota, requested: limits.cpu=200m,requests.cpu=100m, used: limits.cpu=220m,requests.cpu=110m, limited: limits.cpu=300m,requests.cpu=200m
C02W84XMHTD5:terraform-dev iahmad$
Finally, do the clean-up of the installation and configuration files by deleting the namespace and resources contained in it. You can use the command below with your configuration variables:
C02W84XMHTD5:terraform-dev iahmad$ kubectl delete ns quota-demo --cascade
namespace "quota-demo" deleted
Best Practices
- If you follow the best practice of defining and putting resource requests and limits into ResourceQuota object files to establish configuration settings, the Kubernetes cluster will be more stable and there will be fewer disruptions
- Developers and DevOps people on your team should perform CPU and memory profiling to test application resources requirements in advance. Then communicate those numbers to the cluster-admin so that optimal values for requests, limits, and quotas can be configured for Kubernetes runtimes.
- DevOps teams should monitor the actual usage of resources as compared to the allocated (or reserved) capacity to ensure that resources aren’t wasted.
Automated, Intelligent Container Sizing
Kubernetes Vertical Pod Autoscaling doesn’t recommend pod limit values or consider I/O. Densify identifies mis-provisioned containers at a glance and prescribes the optimal configuration.
Densify has partnered with Intel to offer one year of free resource optimization software licensing to qualified companies.