Managing existing clusters Enterprise

Managing non-capi clusters

Any kubernetes cluster whether capi or not can be added to Weave Gitops Enterprise. The only thing we need is a secret containing a valid kubeconfig.

Existing kubeconfig

If you already have a kubeconfig stored in a secret in your management cluster, continue below to create a GitopsCluster.

If you have a kubeconfig, you can load in into the cluster like so:

kubectl create secret generic demo-01-kubeconfig \
--from-file=value=./demo-01-kubeconfig

Create a kubeconfig for a ServiceAccount

How to create a kubeconfig secret using a service account

1. Create a new service account on the remote cluster:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
       name: demo-01
       namespace: default

2. Add RBAC permissions for the service account

   Expand to see role manifests

   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
       name: impersonate-user-groups
   subjects:
       - kind: ServiceAccount
           name: demo-01
           namespace: default
   roleRef:
       kind: ClusterRole
       name: user-groups-impersonator
       apiGroup: rbac.authorization.k8s.io
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
       name: user-groups-impersonator
   rules:
       - apiGroups: [""]
           resources: ["users", "groups"]
           verbs: ["impersonate"]
       - apiGroups: [""]
           resources: ["namespaces"]
           verbs: ["get", "list"]

   This will allow WGE to introspect the cluster for available namespaces.

   Once we know what namespaces are available we can test whether the logged in user can access them via impersonation.

3. Get the token of the service account

   First get the list of secrets of the service accounts by running the following command:

   kubectl get secrets --field-selector type=kubernetes.io/service-account-token
   NAME                      TYPE                                  DATA   AGE
   default-token-lsjz4       kubernetes.io/service-account-token   3      13d
   demo-01-token-gqz7p       kubernetes.io/service-account-token   3      99m

   demo-01-token-gqz7p is the secret that holds the token for demo-01 service account

   To get the token of the service account run the following command:

   TOKEN=$(kubectl get secret demo-01-token-gqz7p -o jsonpath={.data.token} | base64 -d)

4. Create a kubeconfig secret

   We'll use a helper script to generate the kubeconfig, save this into static-kubeconfig.sh:

   Expand to see script

   static-kubeconfig.sh

   #!/bin/bash
   
   if [[ -z "$CLUSTER_NAME" ]]; then
           echo "Ensure CLUSTER_NAME has been set"
           exit 1
   fi
   
   if [[ -z "$CA_CERTIFICATE" ]]; then
           echo "Ensure CA_CERTIFICATE has been set to the path of the CA certificate"
           exit 1
   fi
   
   if [[ -z "$ENDPOINT" ]]; then
           echo "Ensure ENDPOINT has been set"
           exit 1
   fi
   
   if [[ -z "$TOKEN" ]]; then
           echo "Ensure TOKEN has been set"
           exit 1
   fi
   
   export CLUSTER_CA_CERTIFICATE=$(cat "$CA_CERTIFICATE" | base64)
   
   envsubst <<EOF
   apiVersion: v1
   kind: Config
   clusters:
   - name: $CLUSTER_NAME
       cluster:
           server: https://$ENDPOINT
           certificate-authority-data: $CLUSTER_CA_CERTIFICATE
   users:
   - name: $CLUSTER_NAME
       user:
           token: $TOKEN
   contexts:
   - name: $CLUSTER_NAME
       context:
           cluster: $CLUSTER_NAME
           user: $CLUSTER_NAME
   current-context: $CLUSTER_NAME
   
   EOF

   For the next step, the cluster certificate (CA) is needed. How you get hold of the certificate depends on the cluster. For GKE you can view it on the GCP Console: Cluster->Details->Endpoint->”Show cluster certificate”. You will need to copy the contents of the certificate into the ca.crt file used below.

   CLUSTER_NAME=demo-01 \
   CA_CERTIFICATE=ca.crt \
   ENDPOINT=<control-plane-ip-address> \
   TOKEN=<token> ./static-kubeconfig.sh > demo-01-kubeconfig

   Replace the following:

   • CLUSTER_NAME: the name of your cluster i.e. demo-01
   • ENDPOINT: the API server endpoint i.e. 34.218.72.31
   • CA_CERTIFICATE: path to the CA certificate file of the cluster
   • TOKEN: the token of the service account retrieved in the previous step

   Finally create a secret for the generated kubeconfig:

   kubectl create secret generic demo-01-kubeconfig \
   --from-file=value=./demo-01-kubeconfig

Connect a cluster

Before you start!

Make sure you've

1. Added some common RBAC rules into the clusters/bases folder, as described in Getting started.
2. Configured the cluster bootstrap controller as described in Getting started.

Create a GitopsCluster

./clusters/management/clusters/demo-01.yaml

apiVersion: gitops.weave.works/v1alpha1
kind: GitopsCluster
metadata:
  name: demo-01
  namespace: default
  # Signals that this cluster should be bootstrapped.
  labels:
    weave.works/capi: bootstrap
spec:
  secretRef:
    name: demo-01-kubeconfig

When the GitopsCluster appears in the cluster, the Cluster Bootstrap Controller will install flux on it and by default start reconciling the ./clusters/demo-01 path in your management cluster's git repository. To inspect the Applications and Sources running on the new cluster we need to give permissions to the user accessing the UI. Common RBAC rules like this should be stored in ./clusters/bases. Here we create a kustomziation to add these common resources onto our new cluster:

./clusters/demo-01/clusters-bases-kustomization.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  creationTimestamp: null
  name: clusters-bases-kustomization
  namespace: flux-system
spec:
  interval: 10m0s
  path: clusters/bases
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

Save these 2 files into your git repository. Commit and push.

Once flux has reconciled the cluster you can inspect your flux resources via the UI!

Debugging

How to test a kubeconfig secret in a cluster

To test a kubeconfig secret has been correctly setup apply the following manifest and check the logs after the job completes:

Expand to see manifest

---
apiVersion: batch/v1
kind: Job
metadata:
  name: kubectl
spec:
  ttlSecondsAfterFinished: 30
  template:
    spec:
      containers:
        - name: kubectl
          image: bitnami/kubectl
          args:
            [
              "get",
              "pods",
              "-n",
              "kube-system",
              "--kubeconfig",
              "/etc/kubeconfig/value",
            ]
          volumeMounts:
            - name: kubeconfig
              mountPath: "/etc/kubeconfig"
              readOnly: true
      restartPolicy: Never
      volumes:
        - name: kubeconfig
          secret:
            secretName: demo-01-kubeconfig
            optional: false

In the manifest above demo-01-kubeconfig is the name of the secret that contains the kubeconfig for the remote cluster.

---

Background

• Authentication strategies
  • X509 client certificates: can be used across different namespaces
  • Service account tokens: limited to a single namespace
• Kubernetes authentication 101 (CNCF blog post)
• Kubernetes authentication (Magalix blog post)