Custom Resources

Teach the Kubernetes API a new kind of object with a CRD, and write the Go structs that represent it.

Syntaxtype Database struct { metav1.TypeMeta; metav1.ObjectMeta; Spec DatabaseSpec; Status DatabaseStatus }

Kubernetes knows about Pods, Deployments and Secrets. You can add your own kind. A CustomResourceDefinition, or CRD, is itself a Kubernetes object, and submitting one tells the API server to accept a new kind of object: validate it, store it, serve it, and let kubectl get list it. After that, kubectl apply -f my-database.yaml works for a kind nobody shipped.

The CRD gives the new kind a group (a DNS-style namespace such as db.example.com), a version (v1alpha1), a kind (Database) and an OpenAPI schema describing its fields. Nothing is implemented yet. A CRD only creates a place to store intent; a controller makes the intent real.

Spec and status

Every Kubernetes object splits in two. spec is what the user asked for; only the user writes it. status is what the controller observed; only the controller writes it. Enabling the status subresource in the CRD enforces that split, and means a status write does not change metadata.generation.

The Go side

Each kind is a struct embedding metav1.TypeMeta (apiVersion and kind) and metav1.ObjectMeta (name, namespace, labels), plus your Spec and Status structs. The JSON field tags are the field names users will type in YAML. Each kind also needs a List type and a generated DeepCopyObject method.

The same thing, written two ways

You can write the CRD's OpenAPI schema by hand. The shorthand is a // +kubebuilder:validation: comment above each Go field; controller-gen reads those comments and writes the same YAML. It also writes the DeepCopyObject methods, which is why no one types them.

Example

package main

import (
	"encoding/json"
	"fmt"

	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// Database is one object of the custom kind. TypeMeta carries apiVersion and
// kind; ObjectMeta carries name, namespace and labels. Every kind embeds both.
type Database struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   DatabaseSpec   `json:"spec,omitempty"`
	Status DatabaseStatus `json:"status,omitempty"`
}

// DatabaseSpec is the half the user writes. The json tags are the names that
// appear in YAML, so changing one is a breaking API change.
type DatabaseSpec struct {
	Engine    string `json:"engine"`
	SizeGB    int32  `json:"sizeGB"`
	Retention string `json:"retention,omitempty"`
}

// DatabaseStatus is the half the controller writes.
type DatabaseStatus struct {
	Phase string `json:"phase,omitempty"`
}

func main() {
	db := Database{
		TypeMeta:   metav1.TypeMeta{APIVersion: "db.example.com/v1alpha1", Kind: "Database"},
		ObjectMeta: metav1.ObjectMeta{Name: "orders", Namespace: "shop"},
		Spec:       DatabaseSpec{Engine: "postgres", SizeGB: 20},
	}
	out, err := json.Marshal(db.Spec)
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(db.Kind, db.Name, string(out))
}

// Output:
// Database orders {"engine":"postgres","sizeGB":20}

When to use it

Let application teams request a database with a short YAML file instead of filing a ticket.
Represent a tenant, a feature flag set or a scheduled report as a first-class cluster object that kubectl can list.
Store configuration that a controller watches, with validation enforced by the API server rather than by your code.
Give an internal platform an API that works with existing Kubernetes tooling, RBAC and audit logs for free.

More examples

The CRD that makes the kind exist

Apply this and kubectl get databases starts working, with no controller running at all. The schema is the only validation users get, so anything you do not constrain here reaches your Go code unchecked. additionalPrinterColumns is what turns kubectl get db from a list of names into something readable.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # The name must be <plural>.<group>, exactly.
  name: databases.db.example.com
spec:
  group: db.example.com
  scope: Namespaced
  names:
    kind: Database
    plural: databases
    singular: database
    shortNames: [db]
  versions:
    - name: v1alpha1
      served: true
      storage: true   # exactly one version may be the storage version
      subresources:
        status: {}    # without this, status writes count as spec writes
      additionalPrinterColumns:
        - name: Engine
          type: string
          jsonPath: .spec.engine
        - name: Phase
          type: string
          jsonPath: .status.phase
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [engine, sizeGB]
              properties:
                engine:
                  type: string
                  enum: [postgres, mysql]
                sizeGB:
                  type: integer
                  minimum: 1
                  maximum: 1000
                retention:
                  type: string
            status:
              type: object
              properties:
                phase:
                  type: string

Markers that generate the schema and the deepcopy code

This file plus `controller-gen object crd` produces both the deepcopy methods and the YAML from the previous example, so the schema cannot drift from the structs. Forgetting to register a kind with the scheme gives a confusing "no kind registered" error at the first client call rather than at compile time.

// Package v1alpha1 holds the Database API types.
// +groupName=db.example.com
package v1alpha1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime/schema"
	"sigs.k8s.io/controller-runtime/pkg/scheme"
)

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:resource:shortName=db
// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
type Database struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   DatabaseSpec   `json:"spec,omitempty"`
	Status DatabaseStatus `json:"status,omitempty"`
}

type DatabaseSpec struct {
	// These comments become the enum and range in the CRD's OpenAPI schema.
	// +kubebuilder:validation:Enum=postgres;mysql
	Engine string `json:"engine"`
	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:validation:Maximum=1000
	SizeGB int32 `json:"sizeGB"`
}

type DatabaseStatus struct {
	Phase      string             `json:"phase,omitempty"`
	Conditions []metav1.Condition `json:"conditions,omitempty"`
}

// +kubebuilder:object:root=true

// DatabaseList is required: the API server returns a List kind from any list call.
type DatabaseList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []Database `json:"items"`
}

// GroupVersion identifies this API. AddToScheme registers both kinds with a
// client so it knows how to encode and decode them.
var (
	GroupVersion  = schema.GroupVersion{Group: "db.example.com", Version: "v1alpha1"}
	SchemeBuilder = &scheme.Builder{GroupVersion: GroupVersion}
	AddToScheme   = SchemeBuilder.AddToScheme
)

func init() {
	SchemeBuilder.Register(&Database{}, &DatabaseList{})
}

Reading and updating your own kind

Reach for this once the CRD and types exist and you need the behaviour behind them. The three things to copy are the NotFound check, the early return when there is nothing to do, and Status().Update instead of Update.

package main

import (
	"context"

	apierrors "k8s.io/apimachinery/pkg/api/errors"
	"sigs.k8s.io/controller-runtime/pkg/client"
	ctrl "sigs.k8s.io/controller-runtime"

	dbv1 "example.com/db/api/v1alpha1"
)

type Reconciler struct {
	client.Client
}

// Reconcile is called with the name of one Database. ctrl.Request carries only
// namespace and name, so the object is always re-read from the cache here.
func (r *Reconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	var db dbv1.Database
	if err := r.Get(ctx, req.NamespacedName, &db); err != nil {
		if apierrors.IsNotFound(err) {
			return ctrl.Result{}, nil // deleted; owner references clean up the children
		}
		return ctrl.Result{}, err
	}

	if db.Status.Phase == "Ready" {
		return ctrl.Result{}, nil // nothing to do: this is what idempotence looks like
	}

	db.Status.Phase = "Ready"
	// Status() writes the status subresource. A plain Update would also send spec
	// and could overwrite a change the user made a moment ago.
	return ctrl.Result{}, r.Status().Update(ctx, &db)
}

Discussion

Be the first to comment on this lesson.

Spec and status

The Go side

The same thing, written two ways

Example

When to use it

More examples

The CRD that makes the kind exist

Markers that generate the schema and the deepcopy code

Reading and updating your own kind

Related

Discussion