Pitfalls of YAML and shell-less containers

October 28, 2023

Recently I stumbled across some unexpected issues setting up a simple Pod in a Kubernetes cluster.

The Origin

I tried to adapt a Pod like this one, using a Bash shell to run a single command:

apiVersion: v1
kind: Pod
metadata:
  name: my-proxy-pod
  namespace: kube-system
spec:
  containers:
    - name: proxy
      image: quay.io/kubermatic/util:2.3.0
      command:
        - /bin/bash
      args:
        - -c
        - |-
          set -euo pipefail

          echo "Starting kubectl proxy on port $PROXY_PORT..."

          kubectl proxy \
            --address=0.0.0.0 \
            --port=$PROXY_PORT \
            --accept-hosts='^.*'          
      env:
        - name: KUBECONFIG
          value: /opt/kubeconfig/kubeconfig
        - name: PROXY_PORT
          value: "8001"
      ports:
        - containerPort: 8001
          name: http
          protocol: TCP
      # yadda yadda …

Simplifying the PodSpec

I noticed that since I only run a single command, I could skip the /bin/bash entrypoint and instead use kubectl directly. My new Pod then looked like this:

#
# This Pod will not work, do not copy&paste it.
# Scroll down further on this page to find the working example.
#

apiVersion: v1
kind: Pod
metadata:
  name: my-proxy-pod
  namespace: kube-system
spec:
  containers:
    - name: proxy
      image: quay.io/kubermatic/util:2.3.0
      command:
        - kubectl
      args:
        - proxy
        - --address='0.0.0.0'
        - --accept-hosts='^.*'
      env:
        - name: KUBECONFIG
          value: /opt/kubeconfig/kubeconfig
      ports:
        - containerPort: 8001
          name: http
          protocol: TCP
      # yadda yadda …

I deployed this pod (kubectl apply --filename mypod.yaml) and tried it out, only to receive

$ curl http://localhost:8001/metrics
Forbidden

WTF is happening? Why am I suddenly not authenticated? It felt like kubectl proxy was simply ignoring my CLI flags, or that the filtering must be broken. I also wondered if the proxy sub command can simply only forward to the REST API and not any arbitrary endpoint of the Kubernetes server.

I spent an hour trying different flags. I forgot to set --accept-paths='^.*', but even with this flag added, it still wouldn’t work. The only thing to make it truly work was to add --disable-filter, but this is not what I wanted.

I even checked inside the container whether or not the flags were not properly propagated to kubectl proxy, but ps auxf clearly showed

Looks sane, right? But it still wouldn’t work.

The Solution

Long story short: quotes. Without a shell wrapper, Go (kubectl) has to parse the flag values as '...' and Go will simply not strip the single quotes. Meaning that my --accept-paths flag defined a Regex that would never ever match anything.

So to solve this, simply remove the quotes or quote differnently, like so:

apiVersion: v1
kind: Pod
metadata:
  name: my-proxy-pod
  namespace: kube-system
spec:
  containers:
    - name: proxy
      image: quay.io/kubermatic/util:2.3.0
      command:
        - kubectl
      args:
        - proxy
        - "--address=0.0.0.0"
        - "--accept-hosts=^.*"
      env:
        - name: KUBECONFIG
          value: /opt/kubeconfig/kubeconfig
      ports:
        - containerPort: 8001
          name: http
          protocol: TCP
      # yadda yadda …

Og der går du – now the flag values are parsed correctly, kubectl understands what I meant and the proxy works like expected.

Another valid approach would have been

      args:
        - proxy
        - --address
        - 0.0.0.0
        - --accept-hosts
        - ^.*

The Lesson

If you’re not using a shell anymore, pay attention to how strings are quoted and make sure you’re not getting stray quotes anywhere. But at the same time, make sure your YAML is still quoted correctly.

There’s really no good way to debug this, unless the underlying application (kubectl in this example) would explicitly print the value it understood for every single CLI flag. Something I have rarely seen apps do.