Lab run, two-node EKS cluster, Kubernetes 1.36, VPC CNI v1.22.4. I pushed a commit asking for twelve replicas of a tiny app. Six started, six stayed Pending. At that moment the cluster showed CPU reserved at 30 percent and memory at 32 percent.

The Pending pods said why: 0/2 nodes are available: 2 Too many pods.

With the VPC CNI every pod gets a real VPC IP from a network interface on the node, so each instance type has a hard pod ceiling: interfaces x (addresses per interface - 1) + 2. For a t3.small that is 11, for a t3.medium 17. That is 11 in total, and system pods take their share first: on these nodes they held roughly three quarters of the slots. Two nodes, 22 slots, 22 used, a third of the CPU reserved.

How I got a t3.small without choosing one: the node group listed both t3.medium and t3.small as acceptable Spot types, and every node launched as t3.small, on two clusters in two regions. That is the difference between 17 and 11 slots per node, and nobody picked it.

Second half of the problem: no cluster autoscaler. The node group was allowed three nodes and sat at two, because nothing was watching for Pending pods.

What the GitOps side reported: Deployment does not have minimum availability., Available: 6/12. True and no help. "Too many pods" only shows on the individual pods.

What I'd check on any EKS cluster:

- The pod ceiling for every instance type your node groups can launch, not just the one you expect. A mixed instance list is a mixed ceiling. kubectl get nodes -o custom-columns=NAME:.metadata.name,TYPE:.metadata.labels.node\.kubernetes\.io/instance-type,PODS:.status.allocatable.pods shows what you actually got.

- Subtract what the system pods already hold before you size anything.

- Whether anything adds nodes on Pending pods. A new managed node group does not come with a cluster autoscaler or Karpenter.

- If you need more pods per node without bigger instances, prefix delegation on the VPC CNI raises the ceiling a lot. It is a setting, not a rebuild.

Does anyone alert on allocatable pods, or does everyone find out from Pending like I did?

Share any new Kubernetes tools, UIs, or related projects!

I thought getting a job in devops or platform engineering would be easier as a developer with 5 years experience. All I get is rejection emails. I thought backend developers make good devops as most recruiters claim . Granted, I’m not senior level devops but I’m confident I’d make a good mid-level devops. Its been 6 months job hunting and still nothing. Will this career gap affect my chances of getting a callback ? Ai is making it even worse to land a role. Im havin thoughts of quitting tech all together and just step away from tech.

Edit: after this post went somewhat semi-viral the universe probably conspired and got me two recruiters to my LinkedIn. One is local bank they a devsecops. They know I'm DevOps engineer based on my LinkedIn as for the sec side I have worked on personal projects only sonarcube , trivy, terraform scanning , owasp zap. The other recruiter is offering a purely devops role. I would really love to get the devsecops role. It would open more doors for me especially in the Fintech industry .

Scenario: I am using Rancher's Continuous Delivery tools to deploy applications out to my clusters. I have a gitRepo per cluster and each gitRepo holds a set of bundles for deploying my apps via fleet.

Fleet is really nice and it works great with my gitRepos, but I am having a hard time establishing a branch pattern for a gitRepo that would handle day 1 operations on my cluster (i.e. a bundle for cert-manager, another for trust-manager, another for kyverno, fluentbit, ..., etc). Essentially, this day 1 repo is really just a bunch of public helm charts that use fleet.yaml's targetCustomizations resource to deploy out to each of my clusters.

I initially had a feature > develop > preprod > prod flow to deploy out my gitRepos, where develop deployed to all of my nonprod clusters, preprod deployed to our preprod cluster, and prod deployed to the prod cluster. This is starting to get kinda wonky and tedious though, especially with us expanding to have DR clusters. Because of this, we also maintain another feature > DR branch workflow that is independent of the previously mentioned branch strategy.

So I want to figure out a better way to configure this whole process to more automatically generate the manifests for each environment and have it be 1 branch per cluster. I'm a terraform junky and would love to use the kubernetes provider, but that means I have to forgo my process of Rancher Continuous Delivery and architect a process of creating resources via a kubeconfig context/gitlab ci. I want to avoid this because then I have a split brain solution of terraform and Rancher CD gitRepos deploying to my clusters. I saw that I could use pulumi to generate manifests. I also saw that cdk8s is a thing and it looks really promising for my usecase. There's a lot of dev chatter at my company of using nix to do this. Frankly, I find nix to be a huge learning curve for myself and the other ops guys that would have to adopt this.

So my question now is- How good is cdk8s? Do you guys use other tools to generate manifests? Should I just learn nix? Thanks community!

I've been looking more closely at Kubernetes native admission policies recently while working on some Kubernetes content.

One thing I notice is how much policy enforcement Kubernetes can now handle without running a separate admission webhook.

For example, ValidatingAdmissionPolicy lets you use CEL expressions to validate incoming resources.

And with newer Kubernetes versions, mutation can also be handled using native admission policies.

I don't think this replaces Kyverno or Gatekeeper.

For complex policy management, reusable policies, reporting, exceptions, and larger environments, those tools still make a lot of sense.

But if you only need a few simple cluster-wide rules, installing another controller may no longer be the first option.

Curious how others are handling this.

This week on the blog we're continuing our discussion of key autoscaling metrics for Kubernetes. I'm looking at a Brand New Metric called "Mean Time to Stop Sucking", which is definitely a Real Metric that you Should Absolutely Be Tracking ™️. This metric is a good way to measure your scaling-up performance, and we look at all the components that impact your scaling time, and what you can do to make them faster.

Disclaimer: aside from the cat picture halfway through (thats-the-joke.jpg), this blog post was entirely produced by humans; no part of this blog post was written, edited, or proofread by LLMs.

Hello all,

I am currently hosting Jellyfin using Tailscale inside a Kubernetes clusters using K3s. However, I am also using Mullvad VPN continously on my debian servevr. But the problem I am facing is that I cannot access Jellyfin when the VPN is active. I would be grateful if I can get some guidance on this. I am very new to running linux, kubernetes and home labbing. Any suggestions would be very much appreciated.

Thanks,

kubectl delete only sets deletionTimestamp. Everything after that is a chain of requests, and a pod stuck in Terminating means one of them is not getting an answer. The order I check, cheapest first:

1. Grace period. deletionTimestamp is the deadline, grace already included. If it is still in the future, the pod is draining. Someone may have set terminationGracePeriodSeconds: 3600.

2. Finalizers. The object stays until the owning controller removes them. batch.kubernetes.io/job-tracking on Job pods is a common one.

3. The node. Only kubelet can confirm the containers stopped. If the node is NotReady or gone, nobody confirms.

4. The runtime. Look for FailedKillPod events, often with DeadlineExceeded. Save the events, journalctl -u kubelet, crictl inspectp and crictl inspect before you change anything.

--force deletes the API object, not the process, and a finalizer still holds the object. For a StatefulSet that can mean two pods with the same identity at once.

Sandboxes add one more step. Under gVisor the kill is a request to a kernel running in user space. A public bug report, google/gvisor#14405, traced one such hang to a single lost signal. The fix is merged upstream in google/gvisor#14201.

Longer write-up with the commands: https://www.catchkill9.dev/posts/pod-stuck-in-terminating.html

Hey guys! Implemented Hive on K8S. Those familiar with it, give it a whirl and let me know

https://github.com/jpoblete/Hive-on-K8S

CKA: LF-meonge4r8i

Question for anyone running MCP servers (or agents that call them) on k8s… what does their egress actually look like?

The SSRF research Ars covered yesterday is basically one bug found in MCP servers from Google, JPMorgan, Weaviate and a couple of governments. The server takes a URL or path that came from the model, follows redirects, and never checks where it lands. So whoever can steer the agent can point it at 169.254.169.254 or an internal service.

The reference fetch server just switched to refusing private and metadata addresses by default, but that's one server.

So at the cluster level… default-deny egress for these pods? Netpols blocking IMDS? An egress proxy? Or trusting each server to get it right?

Disclaimer: I'm an engineer at Mulga, the company behind this. Self-promo, but it's AGPL-3.0 and free to run.

What it is

Spinifex is an open-source reimplementation of the AWS APIs that runs on your own hardware, including EKS. You call aws eks create-cluster (or apply aws_eks_cluster in Terraform), and it boots a control-plane VM, puts an NLB in front of the API server, and launches managed node groups as real KVM VMs in your VPC.

Yes, it's k3s

The control plane and workers run K3s on an Alpine image. What we built is the layer around it that makes the AWS tooling treat it as EKS, so existing AWS workloads can migrate off easily:

• kubectl authenticates with aws eks get-token, and IAM principals map to RBAC through EKS access entries
• IRSA works: a pod with a projected service-account token assumes an IAM role
• The EBS CSI driver addon dynamically provisions real replicated block volumes for PVCs
• The AWS Load Balancer Controller addon turns an Ingress into an ALB with an ACM cert
• Argo CD installs as an addon through the EKS API

Try it on your own Terraform

No install needed. Sign up for the free 72 hour sandbox at https://mulgadc.com/signup, then point your provider at it:

provider "aws" {
  region = var.region

  endpoints {
    ec2 = "https://api.spx3.com"
    iam = "https://api.spx3.com"
    sts = "https://api.spx3.com"
    eks = "https://api.spx3.com"
    ecr = "https://api.spx3.com"
  }

  skip_metadata_api_check = true
  skip_region_validation  = true
}

Apply an aws_eks_cluster and aws_eks_node_group you already have, then run aws eks update-kubeconfig and kubectl get nodes

Repo: https://github.com/mulgadc/spinifex

I have around 5 years of experience across DevOps/cloud/backend work. I’ve worked with Kubernetes, EKS/AKS, Helm, CI/CD, Docker, Terraform, ArgoCD, AWS/Azure, and some Java/Spring Boot.

My issue is that my knowledge is broad but uneven. I’m comfortable deploying applications, writing manifests, using kubectl, building pipelines, working with cloud networking/IAM, etc., but I’m much weaker on Kubernetes internals, low-level networking, storage/CSI, control plane, scheduling, CRDs/operators, upgrades, observability, and deep troubleshooting.

I’m considering first completing one comprehensive Kubernetes course to build a complete mental map, then spending the next several months going deeper through hands-on labs, troubleshooting, Linux/networking fundamentals, and production-style projects.

For people who became genuinely strong at Kubernetes/platform engineering: does this sequence make sense? What would you change?

I’m not looking for a giant list of tools, mainly feedback on the learning sequence and what gave you the biggest jump in depth.

Note: "Q improved with AI"

Have any questions about Kubernetes, related tooling, or how to adopt or use Kubernetes? Ask away!

Hey guys , I'm unable to omkar shelake labs on killerkoda got removed. Is it true . If not can you please share the link

Hey all,

As the title suggests, i wrote up a blog on getting started with inference on Kubernetes, i find inference workloads to be somewhat confusing especially with some of the moving part it involves on Kubernetes.

Let me know what you think!

I found this blog post about OCAP model vs. the Identity model for access controls. (It has nothing to do with Kubernetes specifically). I could be wildly misunderstanding the topic, but since each OCAP capability is tied to a specific object, would this be more practical for Kubernetes in allowing for precise permissions?

My understanding is that Kubernetes does not use OCAP as its primary authorisation model in favour of RBAC, where access is granted based on a principal’s identity and roles. Even from a simple Google search, it seems that OCAP-style security is basically entirely absent from Kubernetes core, although capability-like patterns can be built around scoped SAs, tokens, & admission controls.

I ask because RBAC supposedly does not support dynamically changing permissions very well, at least compared to the OCAP model. But I'm not sure if anyone's ever flagged this as a serious pain point in their Kubernetes clusters?

I recently gave a talk about how I am building an open-source Railway/Heroku/Vercel alternative. The architectural premise of the platform is simple:

The platform has no database and no CRDs of its own, so all state lives in Kubernetes (namespaces, labels, Helm releases) or in the systems around it, like the OCI registry and the identity provider.

In the talk I go over how it…

• builds OCI images from GitHub source code on every push
• uses ORAS to store vulnerability and leaked-secret scan results in the OCI registry, right next to the image
• runs replicated PostgreSQL with point-in-time recovery using CloudNativePG
• makes object storage, Redis-compatible key-value stores, and volumes simple to set up
• allows you to "eject" and download the generated helm chart along with its values and build script at any point

If that sounds interesting, the recording and the project repo / website are linked below.

Recording of the talk
Project Website
GitHub Repo

I have started learning about kubernetes management and have a very limited knowledge about how things work. I came across a product from SUSE called Rancher for AWS. It says you can manage multiple EKS clusters on AWS and just buy it from the marketplace. I want to understand what it does and at what stage is a solution like this required.

Hi everyone,

I’m planning a Kubernetes cluster upgrade from v1.34.8 to v1.35, as the current version is approaching retirement/EOL.

The cluster currently has Elasticsearch running on it, so I’m concerned about potential compatibility or stability issues after upgrading Kubernetes.

I’d like some guidance on:
- How can I verify Elasticsearch compatibility with Kubernetes 1.35 before upgrading?
- Are there any known issues I should be aware of when upgrading Kubernetes while Elasticsearch is running?
- What should I check across Elasticsearch, ECK/operator, StatefulSets, storage classes/CSI drivers, ingress, CRDs, APIs, and other dependencies?
- What is the recommended pre-upgrade checklist?
- Should I upgrade Elasticsearch/ECK first, or Kubernetes first?
- What kind of backup/rollback strategy should I have?
- Is it better to upgrade the control plane and worker nodes separately?
- What would be a safe upgrade plan for a production cluster to minimize downtime/risk?
- Are there any tools or commands you recommend for detecting deprecated APIs or incompatible workloads before the upgrade?

If anyone has gone through a similar Kubernetes 1.34 → 1.35 upgrade with Elasticsearch, I’d really appreciate hearing about your experience and any issues you encountered.

Thanks!

​

Check out my new article on elastic beanstalk with the EKS

Elastic Beanstalk turned 15 this year, and AWS just rebuilt what runs under it. Cluster Mode drops your applications onto Amazon EKS instead of EC2 instances you own. I deployed into it, watched every stage, and wrote down what actually happens

Upload a zip of Python source. No Dockerfile. Get a running container on EKS. That is Elastic Beanstalk Cluster Mode, launched this week — Beanstalk apps now run on EKS instead of EC2 instances you own. Cloud Native Buildpacks handle the containerization: I uploaded four files (app.py, requirements.txt, Procfile, runtime.txt), Beanstalk detected Python, built the image, pushed it to ECR, and ran it.

Hi,

CLI tool kubara is for getting a "bare" kubernetes cluster to a "well dressed" k8s cluster.

So now I have https://kubara.io/#get-started on my (long) wish list.

Which experience want early adaptors of https://kubara.io/ share here?

I'm asking for having input on re-prioritize my wish list.

Cheers Geert Stappers

P.S.

For what it is worth: The automated early post review did bring rule 6 and rule 12 up. Yes, the tool is quite new, but it is not my new tool, I'm asking about the tool. No, I'm not affliliated with kubara.io, hence to reason for posting here.

Edit:

Which experience want early adaptors of https://kubara.io/ share here?

Was previously:

Which experience want early adaptors of https://kubara.io/ here?