I write stuff here.

Kubectl

January 14, 2024

Kubectl is the all powerful command-line tool that you can use to manage Kubernetes resources and cluster- everything from creating, deleting resources, switching contexts, accessing containers- and more!

Not sure how to pronounce kubectl? Check out kubectl: The definitive pronunciation guide. I’m a monster and pronounce it “kube-see-tee-l”, but I also pronounce systemctl as “system-see-tee-l” so at least I’m consistent. Hopefully this also inspires you to just set up the k alias and never think about the etymology of cli tools again.

Anyway…

There’s a small bug I picked up to fix how kubectl handles in-cluster configuration a.k.a. Issue 93474.

In order to figure out what was actually happening, I learned a couple neat kubectl tricks.

First, --v is not just verbose. You can actually set the log level and get a bunch of interesting outputs. Here’s the kubectl command with no client side configuration working:

I have no name!@kubectl-deployment-8557b8bcb-5nxtw:/$ kubectl get pods --v=8
I1226 18:34:53.023563     117 merged_client_builder.go:163] Using in-cluster namespace
I1226 18:34:53.023718     117 merged_client_builder.go:121] Using in-cluster configuration
I1226 18:34:53.026928     117 merged_client_builder.go:121] Using in-cluster configuration
I1226 18:34:53.027091     117 round_trippers.go:463] GET https://10.96.0.1:443/api/v1/namespaces/default/pods?limit=500
I1226 18:34:53.027104     117 round_trippers.go:469] Request Headers:
I1226 18:34:53.027108     117 round_trippers.go:473]     Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json
I1226 18:34:53.027112     117 round_trippers.go:473]     User-Agent: kubectl/v1.29.0 (linux/arm64) kubernetes/3f7a50f
I1226 18:34:53.027125     117 round_trippers.go:473]     Authorization: Bearer <masked>
I1226 18:34:53.031586     117 round_trippers.go:574] Response Status: 200 OK in 4 milliseconds
I1226 18:34:53.031603     117 round_trippers.go:577] Response Headers:
I1226 18:34:53.031609     117 round_trippers.go:580]     Audit-Id: e6632157-089c-4d76-b785-5d5cf17ce21d
I1226 18:34:53.031613     117 round_trippers.go:580]     Cache-Control: no-cache, private
I1226 18:34:53.031616     117 round_trippers.go:580]     Content-Type: application/json
I1226 18:34:53.031619     117 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: e89a4367-5d31-4ef4-8ce4-0ab965b47216
I1226 18:34:53.031623     117 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 00dde4db-4ddd-4725-8f4f-7a5187672b4a
I1226 18:34:53.031626     117 round_trippers.go:580]     Date: Tue, 26 Dec 2023 18:34:53 GMT
I1226 18:34:53.031719     117 request.go:1212] Response Body: {"kind":"Table","apiVersion":"meta.k8s.io/v1","metadata":{"resourceVersion":"73373"},"columnDefinitions":[{"name":"Name","type":"string","format":"name","description":"Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names#names","priority":0},{"name":"Ready","type":"string","format":"","description":"The aggregate readiness state of this pod for accepting traffic.","priority":0},{"name":"Status","type":"string","format":"","description":"The aggregate status of the containers in this pod.","priority":0},{"name":"Restarts","type":"string","format":"","description":"The number of times the containers in this pod have been restarted and when the last container in this pod has restarted.","priority":0},{" [truncated 4077 chars]
NAME                                 READY   STATUS    RESTARTS   AGE
kubectl-deployment-8557b8bcb-5nxtw   1/1     Running   0          14m

And here is kubectl failing miserably with a --timeout client-side config:

I have no name!@kubectl-deployment-8557b8bcb-5nxtw:/$ kubectl get pods --request-timeout 30 --v=8
I1226 18:35:14.592315     128 merged_client_builder.go:163] Using in-cluster namespace
I1226 18:35:14.592601     128 round_trippers.go:463] GET http://localhost:8080/api?timeout=30s
I1226 18:35:14.592612     128 round_trippers.go:469] Request Headers:
I1226 18:35:14.592618     128 round_trippers.go:473]     Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json
I1226 18:35:14.592622     128 round_trippers.go:473]     User-Agent: kubectl/v1.29.0 (linux/arm64) kubernetes/3f7a50f
I1226 18:35:14.593293     128 round_trippers.go:574] Response Status:  in 0 milliseconds
I1226 18:35:14.593335     128 round_trippers.go:577] Response Headers:
E1226 18:35:14.593422     128 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=30s": dial tcp [::1]:8080: connect: connection refused
I1226 18:35:14.593462     128 cached_discovery.go:120] skipped caching discovery info due to Get "http://localhost:8080/api?timeout=30s": dial tcp [::1]:8080: connect: connection refused

Notice anything different?

Yep! This line GET http://localhost:8080/api?timeout=30s is very very wrong for an in-cluster setup. It should be:

https://10.96.0.1:443/api/v1/namespaces/default/pods?limit=500timeout=30s

So what’s going on?

Currently kubectl –request-timeout (or any other client configuration flag) does not work with in-cluster configuration because isDefaultConfig returns false when checking for equality on this line and then the mergedConfig is used instead of the correct in cluster config.

TL;DR Merging was correctly handled in the in-cluster case so client side config (–timeout, –as) wasn’t being correctly applied.

Another neat kubectl trick I learned from debugging- there’s a --raw flag you can use to directly send the request!

❯ kubectl get --raw="https://127.0.0.1:51532/api/v1/pods?limit=1"
{"kind":"PodList","apiVersion":"v1","metadata" .... lots of random pod info }

I usually don’t run kubectl from an in-cluster pod, so it was fun figuring out the best way to reproduce the issue. First roadblock- my work laptop is a very fancy arm Mac, so I needed to figure out how to cross compile kubectl. Digging into the makefile, I eventually figured out the right magic encantation and we were cooking:

build/run.sh make kubectl KUBE_BUILD_PLATFORMS=linux/amd64

Next- I needed some image with my custom build kubectl to make sure the fix actually worked. Easy, peasy, container-eezy, just needed a small dockerfile:

FROM ubuntu:20.04

WORKDIR /app

COPY kubectl /usr/local/bin/
RUN chmod +x /usr/local/bin/kubectl

USER 1001
ENTRYPOINT [ "kubectl" ]
CMD [ "--help" ]

This docker copies the kubectl I built above that get spit out at kubernetes/_output/dockerized/bin/linux/amd64 and puts it in the new container’s /usr/local/bin/ and then makes it execuctable. Building this dockerfile is super quick because it has nothing in other than the base ubuntu:20.04 and kubectl. I just ran this where I defined the file:

docker build -t nina-kubectl -f test.Dockerfile .

Next, I needed a basic kind cluster. It didn’t need to be super fancy (no metallb, single node, no special cluster config changes). Basically kind create cluster just did the trick.

I created a basic Deployment that used my new docker image:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubectl-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubectl
  template:
    metadata:
      labels:
        app: kubectl
    spec:
      serviceAccountName: my-service-account
      containers:
      - name: kubectl-container
        image: nina-kubectl:latest
        command: [ "/bin/sh", "-c", "--" ]
        args: [ "while true; do sleep 30; done;" ]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-service-account

I execed into the pod that got spun up from my deployment, checked kubectl was there, and then… I realized I needed some RBAC permissions to actually list the pods… Ughhhh….

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["*"]
  verbs: ["list"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: read-pods
subjects:
- kind: ServiceAccount
  name: my-service-account
  namespace: default
roleRef:
  kind: ClusterRole
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: default
subjects:
- kind: ServiceAccount
  name: my-service-account
  namespace: default
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

So in order to validate the fix actually worked, I execed into pod and test kubectl behavior with and without client config with fix:

I have no name!@kubectl-deployment-6874f6db87-9s9qb:/app$ kubectl get pods
NAME                                    READY   STATUS        RESTARTS   AGE
kubectl-deployment-6874f6db87-9s9qb     1/1     Running       0          15s
kubectl-deployment-6874f6db87-mfskr     1/1     Terminating   0          61s
pod-identity-webhook-8565f74769-2772d   1/1     Running       0          27h
I have no name!@kubectl-deployment-6874f6db87-9s9qb:/app$ kubectl get pods  --request-timeout 30
NAME                                    READY   STATUS    RESTARTS   AGE
kubectl-deployment-6874f6db87-9s9qb     1/1     Running   0          35s
pod-identity-webhook-8565f74769-2772d   1/1     Running   0          27h

Nice! So I wrote up some unit tests, updated an e2e test with client config and put my PR.

I thought I was done…

Alas…

Before you can run the e2e test locally, you need a couple things:

  1. Gingko v2 which you can grab with go install github.com/onsi/^Cnkgo/v2/ginkgo@latest
  2. Build the e2e test frame work with: make all WHAT=test/e2e/e2e.test vendor/github.com/onsi/ginkgo/v2/ginkgo
  3. Make sure you have a kind cluster running (Nothing to fancy, kind create cluster will work)

Now in order to run the kubectl e2e tests, you need to dig up this line from the test logs:

 KUBERNETES_PROVIDER=skeleton ./hack/ginkgo-e2e.sh --ginkgo.focus="should handle in-cluster config" --ginkgo.skip='\[Serial\]|\[Deprecated\]'

But for all my cross-compiling magic, it turned out the e2e kubectl tests don’t run on arm based on this comment I should have read earlier…

// TODO: Find a way to download and copy the appropriate kubectl binary, or maybe a multi-arch kubectl image
// for now this only works on amd64
e2eskipper.SkipUnlessNodeOSArchIs("amd64")

Luckily they worked just fine on my very ancient undergrad ubuntu 18.04 laptop.

But… when I ran they e2e test disaster struck again! The e2e tests use the local kubectl set in your env as the kubectl in the tests:

  I0226 14:27:02.844380 462679 builder.go:146] stderr: ""
  I0226 14:27:02.844416 462679 builder.go:147] stdout: ""
  I0226 14:27:02.844558 462679 kubectl.go:645] copying configmap manifests to the httpd pod
  I0226 14:27:02.844614 462679 builder.go:121] Running '/usr/local/bin/kubectl --server=https://127.0.0.1:32947 --kubeconfig=/home/npolshak/.kube/config --namespace=kubectl-4371 cp /tmp/icc-override3336876710/invalid-configmap-with-namespace.yaml kubectl-4371/httpd:/tmp/'

Add to the PATH:

export PATH="/home/npolshak/kubernetes/_output/dockerized/bin/linux/amd64:$PATH"

To be continued…

Other posts