Managing existing clusters Enterprise

Managing non-capi clusters

Any kubernetes cluster whether capi or not can be added to Weave Gitops Enterprise. The only thing we need is a secret containing a valid kubeconfig.

Existing kubeconfig

If you already have a kubeconfig stored in a secret in your management cluster, continue below to create a GitopsCluster.

If you have a kubeconfig, you can load in into the cluster like so:

kubectl create secret generic demo-01-kubeconfig \
--from-file=value=./demo-01-kubeconfig

Create a kubeconfig for a ServiceAccount

How to create a kubeconfig secret using a service account

1. Create a new service account on the remote cluster:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: demo-01
  namespace: default

2. Add RBAC permissions for the service account

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: impersonate-user-groups
subjects:
  - kind: ServiceAccount
    name: demo-01
    namespace: default
roleRef:
  kind: ClusterRole
  name: user-groups-impersonator
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: user-groups-impersonator
rules:
  - apiGroups: [""]
    resources: ["users", "groups"]
    verbs: ["impersonate"]
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list"]

This will allow WGE to introspect the cluster for available namespaces.

Once we know what namespaces are available we can test whether the logged in user can access them via impersonation.

3. Get the token of the service account

First get the list of secrets of the service accounts by running the following command:

kubectl get secrets --field-selector type=kubernetes.io/service-account-token
NAME                      TYPE                                  DATA   AGE
default-token-lsjz4       kubernetes.io/service-account-token   3      13d
demo-01-token-gqz7p       kubernetes.io/service-account-token   3      99m

demo-01-token-gqz7p is the secret that holds the token for demo-01 service account

To get the token of the service account run the following command:

TOKEN=$(kubectl get secret demo-01-token-gqz7p -o jsonpath={.data.token} | base64 -d)

4. Create a kubeconfig secret

We'll use a helper script to generate the kubeconfig, save this into static-kubeconfig.sh:

static-kubeconfig.sh

#!/bin/bash

if [[ -z "$CLUSTER_NAME" ]]; then
    echo "Ensure CLUSTER_NAME has been set"
    exit 1
fi

if [[ -z "$CA_CERTIFICATE" ]]; then
    echo "Ensure CA_CERTIFICATE has been set to the path of the CA certificate"
    exit 1
fi

if [[ -z "$ENDPOINT" ]]; then
    echo "Ensure ENDPOINT has been set"
    exit 1
fi

if [[ -z "$TOKEN" ]]; then
    echo "Ensure TOKEN has been set"
    exit 1
fi

export CLUSTER_CA_CERTIFICATE=$(cat "$CA_CERTIFICATE" | base64)

envsubst <<EOF
apiVersion: v1
kind: Config
clusters:
- name: $CLUSTER_NAME
  cluster:
    server: https://$ENDPOINT
    certificate-authority-data: $CLUSTER_CA_CERTIFICATE
users:
- name: $CLUSTER_NAME
  user:
    token: $TOKEN
contexts:
- name: $CLUSTER_NAME
  context:
    cluster: $CLUSTER_NAME
    user: $CLUSTER_NAME
current-context: $CLUSTER_NAME

EOF

For the next step, the cluster certificate (CA) is needed. How you get hold of the certificate depends on the cluster. For GKE you can view it on the GCP Console: Cluster->Details->Endpoint->”Show cluster certificate”. You will need to copy the contents of the certificate into the ca.crt file used below.

CLUSTER_NAME=demo-01 \
CA_CERTIFICATE=ca.crt \
ENDPOINT=<control-plane-ip-address> \
TOKEN=<token> ./static-kubeconfig.sh > demo-01-kubeconfig

Replace the following:

• CLUSTER_NAME: the name of your cluster i.e. demo-01
• ENDPOINT: the API server endpoint i.e. 34.218.72.31
• CA_CERTIFICATE: path to the CA certificate file of the cluster
• TOKEN: the token of the service account retrieved in the previous step

Finally create a secret for the generated kubeconfig:

kubectl create secret generic demo-01-kubeconfig \
--from-file=value=./demo-01-kubeconfig

Connect a cluster

Get started first!

Make sure you've

1. Added some common RBAC rules into the clusters/bases folder, as described in Getting started.
2. Configured the cluster bootstrap controller as described in Getting started.

Create a GitopsCluster

./clusters/management/clusters/demo-01.yaml

apiVersion: gitops.weave.works/v1alpha1
kind: GitopsCluster
metadata:
  name: demo-01
  namespace: default
  # Signals that this cluster should be bootstrapped.
  labels:
    weave.works/capi: bootstrap
spec:
  secretRef:
    name: demo-01-kubeconfig

When the GitopsCluster appears in the cluster, the Cluster Bootstrap Controller will install flux on it and by default start reconciling the ./clusters/demo-01 path in your management cluster's git repository. To inspect the Applications and Sources running on the new cluster we need to give permissions to the user accessing the UI. Common RBAC rules like this should be stored in ./clusters/bases. Here we create a kustomziation to add these common resources onto our new cluster:

./clusters/demo-01/clusters-bases-kustomization.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  creationTimestamp: null
  name: clusters-bases-kustomization
  namespace: flux-system
spec:
  interval: 10m0s
  path: clusters/bases
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

Save these 2 files into your git repository. Commit and push.

Once flux has reconciled the cluster you can inspect your flux resources via the UI!

Debugging

How to test a kubeconfig secret in a cluster

To test a kubeconfig secret has been correctly setup apply the following manifest and check the logs after the job completes:

apiVersion: batch/v1
kind: Job
metadata:
  name: kubectl
spec:
  ttlSecondsAfterFinished: 30
  template:
    spec:
      containers:
        - name: kubectl
          image: bitnami/kubectl
          args:
            [
              "get",
              "pods",
              "-n",
              "kube-system",
              "--kubeconfig",
              "/etc/kubeconfig/value",
            ]
          volumeMounts:
            - name: kubeconfig
              mountPath: "/etc/kubeconfig"
              readOnly: true
      restartPolicy: Never
      volumes:
        - name: kubeconfig
          secret:
            secretName: demo-01-kubeconfig
            optional: false

In the manifest above demo-01-kubeconfigis the name of the secret that contains the kubeconfig for the remote cluster.

---

Background

• Authentication strategies
  • X509 client certificates: can be used across different namespaces
  • Service account tokens: limited to a single namespace
• Kubernetes authentication 101 (CNCF blog post)
• Kubernetes authentication (Magalix blog post)