Talking to the API with client-go

Build a Kubernetes client from in-cluster credentials and list or change cluster resources from Go.

Syntaxpods, err := clientset.CoreV1().Pods(namespace).List(ctx, metav1.ListOptions{})

Every piece of Kubernetes talks to one HTTP service called the API server. It holds all cluster state: which pods exist, what each Deployment wants, every secret. Nothing else holds that state. If your program wants to know what is running, or wants to change it, it asks the API server.

client-go is the official Go library for those requests. It gives you Go types and methods instead of hand-written HTTP.

Credentials

A program running inside a pod gets a token file mounted at /var/run/secrets/kubernetes.io/serviceaccount/token, plus the API server's address in environment variables. rest.InClusterConfig reads all of that and returns a *rest.Config. Outside a cluster, clientcmd.BuildConfigFromFlags("", path) reads your ~/.kube/config instead. Most programs try the first and fall back to the second.

Clientsets

kubernetes.NewForConfig(cfg) turns a config into a *Clientset. A clientset is one object with a method per API group. client.CoreV1().Pods("shop") narrows down to pods in the namespace shop. A namespace is a named compartment that cluster objects live in. From there, List, Get, Create, Update and Delete each take a context.Context as their first argument and an options struct as their last.

Permission

Having a token is not having permission. Kubernetes checks a Role (what verbs on what kinds) bound to your pod's ServiceAccount. Without a matching rule the call returns 403 Forbidden, not an empty list.

The same thing, written two ways

kubernetes.NewForConfigOrDie(cfg) is shorthand for calling NewForConfig and panicking on error. Identical otherwise. Use it only in main, where there is nothing better to do with the error.

Example

Example · go
package main

import (
	"context"
	"fmt"
	"sort"

	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/kubernetes/fake"
)

// podNames takes kubernetes.Interface, not the concrete *Clientset, so a real
// client and a fake one both fit. That is what makes this testable.
func podNames(ctx context.Context, client kubernetes.Interface, ns string) ([]string, error) {
	list, err := client.CoreV1().Pods(ns).List(ctx, metav1.ListOptions{})
	if err != nil {
		return nil, err
	}
	names := make([]string, 0, len(list.Items))
	for _, p := range list.Items {
		names = append(names, p.Name)
	}
	sort.Strings(names) // the API server promises no particular order
	return names, nil
}

func main() {
	// In a pod this is kubernetes.NewForConfig(rest.InClusterConfig()) instead.
	client := fake.NewSimpleClientset(
		&corev1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "web-2", Namespace: "shop"}},
		&corev1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "web-1", Namespace: "shop"}},
	)
	names, err := podNames(context.Background(), client, "shop")
	if err != nil {
		fmt.Println("list failed:", err)
		return
	}
	fmt.Println(names)
}

// Output:
// [web-1 web-2]

When to use it

  • Write a small command-line tool that reports which pods in a namespace are not running.
  • Have a service scale its own worker Deployment up when a queue backlog grows.
  • Read a ConfigMap at startup from inside the cluster instead of mounting it as a file.
  • Build an admin endpoint that lists the cluster's Deployments and their image tags.

More examples

One client that works inside and outside the cluster

Put this in every cluster-aware tool you write. Without the fallback you cannot run the program on your own machine to debug it, and without InClusterConfig it cannot run as a pod.

Example · go
package main

import (
	"fmt"
	"os"
	"path/filepath"

	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/rest"
	"k8s.io/client-go/tools/clientcmd"
)

// newClient prefers the pod's own credentials and falls back to the developer's
// kubeconfig file, so the same binary runs in both places.
func newClient() (*kubernetes.Clientset, error) {
	cfg, err := rest.InClusterConfig()
	if err != nil {
		path := os.Getenv("KUBECONFIG")
		if path == "" {
			home, herr := os.UserHomeDir()
			if herr != nil {
				return nil, herr
			}
			path = filepath.Join(home, ".kube", "config")
		}
		// The first argument is an API server URL; empty means take it from the file.
		cfg, err = clientcmd.BuildConfigFromFlags("", path)
		if err != nil {
			return nil, err
		}
	}
	return kubernetes.NewForConfig(cfg)
}

func main() {
	client, err := newClient()
	if err != nil {
		fmt.Fprintln(os.Stderr, "no cluster connection:", err)
		os.Exit(1)
	}
	_ = client
	fmt.Println("connected")
}

The RBAC rules the program above needs

Go code alone cannot read the cluster. This is the other half. A 403 Forbidden from List almost always means the Role is missing a verb, or the Deployment forgot serviceAccountName and is using the default account.

Example · yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: pod-reader
  namespace: shop
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: shop
rules:
  # Pods are in the core group, written as the empty string.
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-reader
  namespace: shop
subjects:
  - kind: ServiceAccount
    name: pod-reader
    namespace: shop
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io
---
# The Deployment must name the ServiceAccount, or it gets "default", which can read nothing.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: watcher
  namespace: shop
spec:
  replicas: 1
  selector:
    matchLabels: {app: watcher}
  template:
    metadata:
      labels: {app: watcher}
    spec:
      serviceAccountName: pod-reader
      containers:
        - name: watcher
          image: registry.example.com/watcher:v1

Changing an object without losing someone else's change

Reach for this whenever you write to the cluster. The Get must be inside the retry function: retrying an Update with the stale object you already fetched will conflict forever. Many spec fields are pointers so that "zero" and "not specified" stay distinguishable.

Example · go
package main

import (
	"context"
	"fmt"

	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/util/retry"
)

// scale sets a Deployment's replica count. Update sends the whole object, so it
// fails with a conflict if anything changed since the Get. RetryOnConflict runs
// the whole Get-modify-Update body again on exactly that error.
func scale(ctx context.Context, client kubernetes.Interface, ns, name string, replicas int32) error {
	return retry.RetryOnConflict(retry.DefaultRetry, func() error {
		dep, err := client.AppsV1().Deployments(ns).Get(ctx, name, metav1.GetOptions{})
		if err != nil {
			return err
		}
		dep.Spec.Replicas = &replicas // Replicas is a *int32: nil means "unset"
		_, err = client.AppsV1().Deployments(ns).Update(ctx, dep, metav1.UpdateOptions{})
		return err
	})
}

func main() {
	fmt.Println("call scale(ctx, client, \"shop\", \"web\", 3)")
}

Related

go k8s informersgo k8s operatorsgo k8s manifestsgo context timeoutgo interface basics

Discussion

  • Be the first to comment on this lesson.