4 ms·

For "Maximum pods per node", it shows GKE at 100, AKS at 110 but still puts GKE in green. How come?

cobookman 8y ago
Does anyone even have more than 20 pods / node? I've not personally seen anything near the 100 pods / node limit in the real world.
- sofaofthedamned 8y ago
  I didn't have 100, but I did have around 60 of my own plus all the associated K8s pods (about 10 of them). It's a bad idea to have that many if they're all Apache or whatever. But when they're simple Golang containers that handle APIs it's ok.
- regnerba 8y ago
  We don't have 100, but definitely more than 20. A quick look at a random node from our cluster: - 3x ingress pods (1 for an internal load balancer, 2 for an external load balancer) - 1x cert-manager - 2x production version of internal app01 - 2x staging version of internal app02 - 1x fluentd - 1x elastalert - 1x kubewatch - 1x prometheus node exporter - 1x redis for sentry - 1x sentry web - 2x sentry worker - 4x sourcegraph language servers (there are 9 language servers running across the 4 nodes, this node seems to especially like them) - 1x thelounge - 1x staging version of app02 - 2x production version of app02 - 1x kube proxy - 1x kubedns - 1x couchdb (small footprint just for testing some things) the node is just over half provisioned and we are due to add another node to the cluster soon.
  - deepsun 8y ago
    What does prometheus node exporter collects from within a pod? I was under impression that to collect node stats you absolutely need to be "outside" any container, i.e. using prometheus-k8s integration so that it pulls nodes stats from k8s api, not nodes themselves).
    - regnerba 8y ago
      Not an expert, and this may have changed since I deployed this, but I based most of our deployment on the Prometheus helm chart. The helm charts include this: https://github.com/helm/charts/tree/master/stable/prometheus-node-exporter https://github.com/helm/charts/tree/master/stable/prometheus...
    - halbritt 8y ago
      The kube-state-metrics service runs as part of the kubernetes API and provides container metrics. It replaces heapster. The node exporter runs as a daemonset on each node and provides node specific metrics like CPU, memory, disk IO, network IO, etc. The node metrics comes from the node itself.
- smarterclayton 8y ago
  A lot of large clusters I see in the enterprise are on big VMs or metal (48 core boxes) and so can range from 100-400 for a lot of common medium sized workloads. But at that density you also have to have a workloads with low IO requirements and think about network density. Also, very dense multi tenant dev clusters can easily get above 100 for test environments or when lots of people have created demo apps that mostly sit idle.
  - halbritt 8y ago
    I have >100 development environments, each of which is comprised of ~23 pods or so. I think I'm running 70 16-core nodes.
- diek 8y ago
  Try explaining to people that Kubernetes may not be the best option for large-scale hosting. I'm currently dealing with people who want to have a 'one pod per site' deployment model for 500,000 customers. Do the math on how many nodes that is with a 30 or 100 pod limit per node.
  - regnerba 8y ago
    Out of curiosity what sort of resource utilization is each deployment?
    - diek 8y ago
      Varies quite a bit. Some deployments are high CPU, high I/O, some are basically idle. All are running the same docker image.
- halbritt 8y ago
  My org has hit the 100 pods/node limit and it bugs me immensely. We have a fair number of idle pods and we're using n1-highmem-16 nodes. I'd rather be using much larger nodes, but I'd have a hard time warming the cores and getting my money's worth. It'd be nice to use larger nodes and have more overhead, but I suspect kubelet would have a hard time.
- knoxa2511 8y ago
  I've worked with 200-300 K8s customers and 30-40 openshift customers. On K8s I typically see somewhere in the low 20s as the number of pods per node. Openshift I'll see high 20s or low 30s as the average number of pods across most openshift customers. But we're seeing some crazy numbers for some larger enterprise customers. 100,400,2500 pods per node. This seems to be driven by the way openshift is licensing The latter is an absolute nightmare to support, and they seem to have trouble organizing internally as well.

Does anyone even have more than 20 pods / node? I've not personally seen anything near the 100 pods / node limit in the real world.

sofaofthedamned 8y ago
I didn't have 100, but I did have around 60 of my own plus all the associated K8s pods (about 10 of them). It's a bad idea to have that many if they're all Apache or whatever. But when they're simple Golang containers that handle APIs it's ok.
regnerba 8y ago
We don't have 100, but definitely more than 20. A quick look at a random node from our cluster: - 3x ingress pods (1 for an internal load balancer, 2 for an external load balancer) - 1x cert-manager - 2x production version of internal app01 - 2x staging version of internal app02 - 1x fluentd - 1x elastalert - 1x kubewatch - 1x prometheus node exporter - 1x redis for sentry - 1x sentry web - 2x sentry worker - 4x sourcegraph language servers (there are 9 language servers running across the 4 nodes, this node seems to especially like them) - 1x thelounge - 1x staging version of app02 - 2x production version of app02 - 1x kube proxy - 1x kubedns - 1x couchdb (small footprint just for testing some things) the node is just over half provisioned and we are due to add another node to the cluster soon.
- deepsun 8y ago
  What does prometheus node exporter collects from within a pod? I was under impression that to collect node stats you absolutely need to be "outside" any container, i.e. using prometheus-k8s integration so that it pulls nodes stats from k8s api, not nodes themselves).
  - regnerba 8y ago
    Not an expert, and this may have changed since I deployed this, but I based most of our deployment on the Prometheus helm chart. The helm charts include this: https://github.com/helm/charts/tree/master/stable/prometheus-node-exporter https://github.com/helm/charts/tree/master/stable/prometheus...
  - halbritt 8y ago
    The kube-state-metrics service runs as part of the kubernetes API and provides container metrics. It replaces heapster. The node exporter runs as a daemonset on each node and provides node specific metrics like CPU, memory, disk IO, network IO, etc. The node metrics comes from the node itself.
smarterclayton 8y ago
A lot of large clusters I see in the enterprise are on big VMs or metal (48 core boxes) and so can range from 100-400 for a lot of common medium sized workloads. But at that density you also have to have a workloads with low IO requirements and think about network density. Also, very dense multi tenant dev clusters can easily get above 100 for test environments or when lots of people have created demo apps that mostly sit idle.
- halbritt 8y ago
  I have >100 development environments, each of which is comprised of ~23 pods or so. I think I'm running 70 16-core nodes.
diek 8y ago
Try explaining to people that Kubernetes may not be the best option for large-scale hosting. I'm currently dealing with people who want to have a 'one pod per site' deployment model for 500,000 customers. Do the math on how many nodes that is with a 30 or 100 pod limit per node.
- regnerba 8y ago
  Out of curiosity what sort of resource utilization is each deployment?
  - diek 8y ago
    Varies quite a bit. Some deployments are high CPU, high I/O, some are basically idle. All are running the same docker image.
halbritt 8y ago
My org has hit the 100 pods/node limit and it bugs me immensely. We have a fair number of idle pods and we're using n1-highmem-16 nodes. I'd rather be using much larger nodes, but I'd have a hard time warming the cores and getting my money's worth. It'd be nice to use larger nodes and have more overhead, but I suspect kubelet would have a hard time.
knoxa2511 8y ago
I've worked with 200-300 K8s customers and 30-40 openshift customers. On K8s I typically see somewhere in the low 20s as the number of pods per node. Openshift I'll see high 20s or low 30s as the average number of pods across most openshift customers. But we're seeing some crazy numbers for some larger enterprise customers. 100,400,2500 pods per node. This seems to be driven by the way openshift is licensing The latter is an absolute nightmare to support, and they seem to have trouble organizing internally as well.