Setup SOPS Enterprise

Weave GitOps Enterprise now supports managing secrets using SOPS, a tool that encrypts and decrypts secrets using various key management services, from the UI. To be able to use this functionality, users need to configure their private and public key-pairs using one of the guides below.

Setup SOPS on management cluster or existing leaf cluster

In this section, we will cover the prerequisites for using SOPS with Weave GitOps Enterprise, and how to configure SOPS for your existing Kubernetes cluster to work with GPG and age keys.

For a more advanced setup for SOPS with flux, please refer to this guide.

Encrypting secrets using GPG/OpenPGP

OpenPGP is a way of using SOPS to encrypt and decrypt secrets with Weave GitOps Enterprise.

Here are the steps to generate an OpenPGP key and configure your cluster to work with Weave GitOps Enterprise secrets management.

1- Generate a gpg key pairs

Expand for instructions

export KEY_NAME="gpg-key"
export KEY_COMMENT="gpg key"

gpg --batch --full-generate-key <<EOF
%no-protection
Key-Type: 1
Key-Length: 4096
Subkey-Type: 1
Subkey-Length: 4096
Expire-Date: 0
Name-Comment: ${KEY_COMMENT}
Name-Real: ${KEY_NAME}
EOF

2- Export the key pairs fingerprint in the shell

gpg --list-secret-keys "${KEY_NAME}"

sec   rsa4096 2020-09-06 [SC]
      710DC0DB6C1662F707095FC30233CB21E656A3CB

export KEY_FP="710DC0DB6C1662F707095FC30233CB21E656A3CB"

3- Export the generated private key to a kubernetes secret sops-gpg-private-key which will be used by flux's kustomize-controller to decrypt the secrets using sops.

gpg --export-secret-keys --armor "${KEY_FP}" |
kubectl create secret generic sops-gpg-private-key \
--namespace=flux-system \
--from-file=sops.asc=/dev/stdin

4- Export the generated public key to a kubernetes secret sops-gpg-public-key which will be used by Weave GitOps Enterprise to encrypt the secrets created from the UI.

gpg --export --armor "${KEY_FP}" |
kubectl create secret generic sops-gpg-public-key \
--namespace=flux-system \
--from-file=sops.asc=/dev/stdin

tip

It's recommended to remove the secret from your machine

gpg --delete-secret-keys "${KEY_FP}"

5- Create a kustomization for reconciling the secrets on the cluster and set the --decryption-secret flag to the name of the private key created in step 3.

flux create kustomization gpg-secrets \
--source=secrets \ # the git source to reconcile the secrets from
--path=./secrets/gpg \
--prune=true \
--interval=10m \
--decryption-provider=sops \
--decryption-secret=sops-gpg-private-key

6- Annotate the kustomization object created in the previous step with the name and namespace of the public key created in step 4.

kubectl annotate kustomization gpg-secrets \
sops-public-key/name=sops-gpg-public-key \
sops-public-key/namespace=flux-system \
-n flux-system

Expand to see the expected kustomization object

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: gpg-secrets
  namespace: flux-system
  annotations:
    sops-public-key/name: sops-gpg-public-key
    sops-public-key/namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: secrets
  path: ./secrets/gpg
  decryption:
    provider: sops
    secretRef:
      name: sops-gpg-private-key
  prune: true
  validation: server

note

This is an essential step in order to allow other operators and developers to utilize WeaveGitOps UI to encrypt SOPS secrets using the public key secret in the cluster.

Encrypting secrets using age

age is a simple, modern and secure file encryption tool, that can be used to encrypt secrets using Weave GitOps Enterprise.

Here are the steps to generate an age key and configure your cluster to work with Weave GitOps Enterprise secrets management.

1- Generate an age key with age-keygen

age-keygen -o age.agekey

Public key: <public key>

2- Export the generated private key to a kubernetes secret sops-age-private-key which will be used by flux's kustomize-controller to decrypt the secrets using sops.

cat age.agekey |
kubectl create secret generic sops-age-private-key \
--namespace=flux-system \
--from-file=age.agekey=/dev/stdin

4- Export the generated public key to a kubernetes secret sops-age-public-key which will be used by Weave GitOps Enterprise to encrypt the secrets created from the UI.

echo "<public key>" |
kubectl create secret generic sops-age-public-key \
--namespace=flux-system \
--from-file=age.agekey=/dev/stdin

tip

It's recommended to remove the secret from your machine

rm -f age.ageKey

5- Create a kustomization for reconciling the secrets on the cluster and set the --decryption-secret flag to the name of the private key created in step 2

flux create kustomization age-secrets \
--source=secrets \ # the git source to reconcile the secrets from
--path=./secrets/age \
--prune=true \
--interval=10m \
--decryption-provider=sops \
--decryption-secret=sops-age-private-key

6- Annotate the kustomization object created in the previous step with the name and namespace of the public key created in step 4.

kubectl annotate kustomization age-secrets \
sops-public-key/name=sops-age-public-key \
sops-public-key/namespace=flux-system \
-n flux-system

Expand to see the expected kustomization object

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: age-secrets
  namespace: flux-system
  annotations:
    sops-public-key/name: sops-age-public-key
    sops-public-key/namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: secrets
  path: ./secrets/age
  decryption:
    provider: sops
    secretRef:
      name: sops-age-private-key
  prune: true
  validation: server

note

This is an essential step in order to allow other operators and developers to utilize WeaveGitOps UI to encrypt SOPS secrets using the public key secret in the cluster.

tip

In case of using OpenPGP and age in the same cluster, you need to make the kustomizations point to different directories. This is because flux's kustomize-controller expects that all the secrets in the kustomization's path are encrypted with the same key.

Bootstrapping SOPS to leaf clusters

Bootstrapping SOPS to leaf clusters in WGE can be done by utilizing ClusterBootstrapConfig job to bootstrap Flux and SOPS. The job is a container which generates SOPS secrets key pair, creates a kubernetes secret with the private key, creates a kubernetes secret with the public key (to be used in self-serve flow) and the proper rbac for it. As well as an option to push the public key to the git repository via a PR (to be distributed).

Prerequisites

ClusterBootstrapConfig job

The following example is using GPG encryption to install SOPS and generate keys when bootstrapping leaf clusters. Create the following ClusterBootstrapConfig CR and push it to your fleet repo.

Expand to view

clusters/management/capi/boostrap/sops-bootstrap-job.yaml

apiVersion: capi.weave.works/v1alpha1
kind: ClusterBootstrapConfig
metadata:
  name: sops-installation
  namespace: default
spec:
  clusterSelector:
    matchLabels:
      weave.works/flux: "bootstrap"
  jobTemplate:
    generateName: "run-gitops-flux-{{ .ObjectMeta.Name }}"
    spec:
      containers:
        - image: ghcr.io/fluxcd/flux-cli:v0.35.0
          imagePullPolicy: Always
          name: flux-bootstrap
          resources: {}
          volumeMounts:
            - name: kubeconfig
              mountPath: "/etc/gitops"
              readOnly: true
          args:
            [
              "bootstrap",
              "github",
              "--kubeconfig=/etc/gitops/value",
              "--owner=<github-username>", # to be changed
              "--repository=<github-repository>", # to be changed
              "--path=./clusters/{{ .ObjectMeta.Namespace }}/{{ .ObjectMeta.Name }}",
            ]
          envFrom:
            - secretRef:
                name: my-pat # github token secret for flux: see https://docs.gitops.weave.works/docs/cluster-management/getting-started/
          env:
            - name: EXP_CLUSTER_RESOURCE_SET
              value: "true"
        - image: weaveworks/sops-bootstrap:0.1.0
          imagePullPolicy: Always
          name: sops-bootstrap
          resources: {}
          volumeMounts:
            - name: kubeconfig
              mountPath: "/etc/gitops"
              readOnly: true
          command: ["bash", "/root/entrypoint.sh"]
          envFrom:
            - secretRef:
                name: my-pat # github token secret for flux: see https://docs.gitops.weave.works/docs/cluster-management/getting-started/
          env:
            - name: KEY_NAME  
              value: '{{ annotation "weave.works/sops-key-name" }}'
            - name: KEY_COMMENT  
              value: '{{ annotation "weave.works/sops-key-comment" }}'
            - name: SOPS_SECRET_REF 
              value: '{{ annotation "weave.works/sops-secret-ref" }}'
            - name: SOPS_SECRET_REF_NAMESPACE
              value: '{{ annotation "weave.works/sops-secret-ref-namespace" }}'
            - name: PUSH_TO_GIT
              value: '{{ annotation "weave.works/sops-push-to-git" }}'
            - name: CLUSTER_NAME
              value: "{{ .ObjectMeta.Name }}"
            - name: CLUSTER_NAMESPACE
              value: "{{ .ObjectMeta.Namespace }}"
      restartPolicy: Never
      volumes:
        - name: kubeconfig
          secret:
            secretName: "{{ .ObjectMeta.Name }}-kubeconfig"

Cluster template updates

In order to bootstrap SOPS to leaf clusters, we need some modifications to the cluster template to allow creating a Kustomization for reconciling the secrets on the cluster using SOPS and to run the ClusterBootstrapConfig job during cluster creation.

The template metadata should have annotation, it will be used by WGE to create the Kustomization with the cluster files.

templates.weave.works/sops-enabled: "true"

The template should have the following parameters that are needed for the Kustomization

Expand to view

clusters/management/capi/templates/template.yaml

- name: SOPS_KUSTOMIZATION_NAME
  required: true
  description: This Kustomization will be used to decrypt SOPS secrets from this path `clusters/default/leaf-cluster/sops/` after reconciling on the cluster. example (`my-secrets`)
- name: SOPS_SECRET_REF
  required: true
  description: The private key secret name that will be generated by SOPS in the bootstrap job. example (`sops-gpg`)
- name: SOPS_SECRET_REF_NAMESPACE
  required: true
  description: The private key secret namespace this secret will be generated by SOPS in the bootstrap job. example (`flux-system`)
- name: SOPS_KEY_NAME
  required: true
  description: SOPS key name. This will be used to generate SOPS keys. example (`test.yourdomain.com`)
- name: SOPS_KEY_COMMENT
  required: true
  description: SOPS key comment. This will be used to generate SOPS keys. example (`sops secret comment`)
- name: SOPS_PUSH_TO_GIT
  required: true
  description: Option to push the public key to the git repository. expected values (`true`, `false`)

The template should have the following annotations under GitOpsCluster to be used in the bootstrap job

Expand to view

clusters/management/capi/templates/template.yaml

# annotation to hold the kustomization values for cluster bootstrap job
weave.works/sops-kustomization: "${SOPS_KUSTOMIZATION_NAME}"
weave.works/sops-secret-ref: "${SOPS_SECRET_REF}"
weave.works/sops-secret-ref-namespace: "${SOPS_SECRET_REF_NAMESPACE}"
weave.works/sops-push-to-git: "${SOPS_PUSH_TO_GIT}"
weave.works/sops-key-name: "${SOPS_KEY_NAME}"
weave.works/sops-key-comment: "${SOPS_KEY_COMMENT}"

Installation Steps

To bootstrap SOPS on a leaf cluster, create a new cluster using the SOPS template from the Create Cluster page and fill in the following SOPS-related values in the form:

• SOPS_KUSTOMIZATION_NAME: This Kustomization will be used to decrypt SOPS secrets from this path clusters/default/leaf-cluster/sops/ after reconciling on the cluster. example (my-secrets)
• SOPS_SECRET_REF: The private key secret name that will be generated by SOPS in the bootstrap job. example (sops-gpg)
• SOPS_SECRET_REF_NAMESPACE: The private key secret namespace this secret will be generated by SOPS in the bootstrap job. example (flux-system)
• SOPS_KEY_NAME: SOPS key name. This will be used to generate SOPS keys. example (test.yourdomain.com)
• SOPS_KEY_COMMENT: SOPS key comment. This will be used to generate SOPS keys. example (sops secret comment)
• SOPS_PUSH_TO_GIT: Option to push the public key to the git repository. expected values (true, false)

What to expect

• A leaf cluster created with Flux & SOPS bootstrapped
• A secret created on leaf cluster sops-gpg to decrypt secrets
• A secret created on leaf cluster sops-gpg-pub to encrypt secrets
• A Kustomization with decryption defined in it to SOPS location in the cluster repo location
• Added Role for the public key to be accessed through management cluster
• A PR is created to the cluster repo with the public key and SOPS creation rules (optional)
• Visit the Secrets Page and start managing your secrets via the UI

Security Recommendations

Access to sops decryption secrets should be restricted and allowed only to be read by flux's kustomize controller. This can be done using Kubernetes RBAC.

Here's an example of how you can use RBAC to restrict access to sops decryption secrets:

1. Create a new Kubernetes role that grants read access to sops decryption secrets

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: sops-secrets-role
rules:
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["sops-gpg-private-key", "sops-age-private-key"]
  verbs: ["get", "watch", "list"]

2. Bind the role to the service account of the flux's kustomize-controller

Expand to view the RoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: sops-secrets-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: sops-secrets-role
subjects:
- kind: ServiceAccount
  name: kustomize-controller

caution

You would need to ensure that no other rolebindings or clusterrolebndings would allow reading the the decryption secret at any time. This could be achieved by leveraging policy capabilities to detect existing and prevent future creation of roles that would grant read secrets permissions.