External Secrets Management

Sombra requires secure access to sensitive configuration values like JWT signing keys and TLS certificates. We recommend using the External Secrets Operator (ESO) to securely inject secrets from external secret management systems into your Kubernetes environment.

The External Secrets Operator provides several advantages for managing Sombra secrets:

Centralized Secret Management: Integrate with your existing secret management infrastructure (AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, Google Secret Manager, etc.)
Kubernetes Native: Works seamlessly with Kubernetes secrets and environment variables
Automatic Synchronization: Keeps secrets up-to-date automatically
Security Best Practices: Follows Kubernetes security patterns and RBAC
Least Privilege: Use workload identity to grant only read access to specific secret paths/names
Multi-Provider Support: Supports 30+ secret management providers

Install ESO in your Kubernetes cluster:

Shell

helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm install external-secrets external-secrets/external-secrets -n external-secrets-system --create-namespace

Create a SecretStore (namespaced) or ClusterSecretStore (cluster-wide) resource to connect to your secret management system. For multi-namespace deployments, prefer ClusterSecretStore.

Provider Documentation: External Secrets Operator: AWS Secrets Manager Provider
Recommended Authentication: Use IAM roles for service accounts (IRSA) for authentication.
Alternative: Use AWS Parameter Store.

If using IRSA, first create a ServiceAccount with IAM role binding:

YAML

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sombra
  namespace: sombra
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>

Then create the ClusterSecretStore:

YAML

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: my-secret-store
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-west-2
      auth:
        jwt:
          serviceAccountRef:
            name: sombra
            namespace: sombra

Provider Documentation: External Secrets Operator: HashiCorp Vault Provider
Recommended Authentication: Use Kubernetes authentication with a Vault role.

If using Kubernetes authentication, first create a ServiceAccount:

YAML

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sombra
  namespace: sombra

Then create the ClusterSecretStore:

YAML

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: my-secret-store
spec:
  provider:
    vault:
      server: 'https://vault.example.com'
      path: 'secret'
      version: 'v2'
      auth:
        kubernetes:
          mountPath: 'kubernetes'
          role: 'sombra-role'
          serviceAccountRef:
            name: sombra
            namespace: sombra

Provider Documentation: External Secrets Operator: Azure Key Vault Provider
Recommended Authentication: Use Workload Identity for authentication.

If using Workload Identity, first create a ServiceAccount:

YAML

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sombra
  namespace: sombra
  annotations:
    azure.workload.identity/client-id: 7d8cdf74-xxxx-xxxx-xxxx-274d963d358b
    azure.workload.identity/tenant-id: 5a02a20e-xxxx-xxxx-xxxx-0ad5b634c5d8

Then create the ClusterSecretStore:

YAML

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: my-secret-store
spec:
  provider:
    azurekv:
      vaultUrl: 'https://your-keyvault.vault.azure.net'
      authType: WorkloadIdentity
      serviceAccountRef:
        name: sombra
        namespace: sombra

Provider Documentation: External Secrets Operator: Google Secret Manager Provider
Recommended Authentication: Use Workload Identity for authentication. See the Google Cloud documentation for more information.

If using Workload Identity, first create a ServiceAccount:

YAML

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sombra
  namespace: sombra
  annotations:
    iam.gke.io/gcp-service-account: sombra@your-project-id.iam.gserviceaccount.com

Then create the ClusterSecretStore:

YAML

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: my-secret-store
spec:
  provider:
    gcpsm:
      projectID: 'your-project-id'
      auth:
        workloadIdentity:
          serviceAccountRef:
            name: sombra
            namespace: sombra

For other secret management systems, see the External Secrets Operator providers documentation.

Create ExternalSecret resources to define which secrets to fetch:

YAML

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: sombra-external-secrets
  namespace: sombra
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: my-secret-store
    kind: ClusterSecretStore
  target:
    name: sombra-external-secrets
    creationPolicy: Owner
    deletionPolicy: Retain # or Delete; choose based on your cleanup needs
  data:
    - secretKey: SOMBRA_REVERSE_TUNNEL_API_KEY
      remoteRef:
        key: sombra/reverse-tunnel-api-key
        property: value
    - secretKey: JWT_ECDSA_KEY
      remoteRef:
        key: sombra/jwt-ecdsa-key
        property: value
    - secretKey: INTERNAL_KEY_HASH
      remoteRef:
        key: sombra/internal-key-hash
        property: value

Use envFrom to load all secrets from the ESO-managed secret directly:

YAML

# values.yaml
envFrom:
  - secretRef:
      name: sombra-external-secrets

Shell

helm install sombra transcend/sombra \
  --namespace <sombra-namespace> \
  --values values.yaml

This approach loads all environment variables from the sombra-external-secrets secret that External Secrets Operator creates.

Secret Rotation: Environment variables don't update in running pods. Consider using Stakater Reloader or triggering a rollout when secrets change.

Common issues and debugging steps:

Shell

# Check ExternalSecret status
kubectl get externalsecret sombra-external-secrets -n sombra -o yaml

# Verify Kubernetes secret was created
kubectl describe secret sombra-external-secrets -n sombra

# Check External Secrets Operator logs
kubectl logs -n external-secrets-system deployment/external-secrets

Common errors:

Wrong provider path/property in remoteRef
Missing ServiceAccount/IAM role permissions
Wrong namespace for SecretStore (use ClusterSecretStore for multi-namespace)

Direct Vault integration is no longer recommended. Use External Secrets Operator instead for better security and maintainability.

If you must use the direct Vault integration, Sombra supports fetching secrets from HashiCorp Vault using a sidecar container with vault-agent. This approach is not recommended for new deployments.

To enable the direct Vault integration, set these environment variables:

Env Var Name	Description	Type	Example
`ENABLE_VAULT_FETCHER`	Set to `true` to enable Vault integration	boolean	`true`
`VAULT_AGENT_URI`	The URI of the Vault agent (typically a local address when using a sidecar container)	string	`http://127.0.0.1:8100`

The following environment variables can be configured to fetch secrets from Vault:

Secret Env Var Name	Vault Path Env Var	Vault Version Env Var	Vault Key Name Env Var	Description
`SOMBRA_JWT_ECDSA_KEY`	`VAULT_PATH_SOMBRA_JWT_ECDSA_KEY`	`VAULT_VERSION_SOMBRA_JWT_ECDSA_KEY`	`VAULT_KEY_SOMBRA_JWT_ECDSA_KEY`	The JWT signing key used for webhook signatures and JWT operations. Also serves as entropy source for local KMS key derivation when not using AWS KMS
`SOMBRA_TLS_KEY_PASSPHRASE`	`VAULT_PATH_SOMBRA_TLS_KEY_PASSPHRASE`	`VAULT_VERSION_SOMBRA_TLS_KEY_PASSPHRASE`	`VAULT_KEY_SOMBRA_TLS_KEY_PASSPHRASE`	An optional password for the TLS certificate
`SOMBRA_TLS_CERT`	`VAULT_PATH_SOMBRA_TLS_CERT`	`VAULT_VERSION_SOMBRA_TLS_CERT`	`VAULT_KEY_SOMBRA_TLS_CERT`	The Sombra TLS certificate as a base64-encoded string
`SOMBRA_TLS_KEY`	`VAULT_PATH_SOMBRA_TLS_KEY`	`VAULT_VERSION_SOMBRA_TLS_KEY`	`VAULT_KEY_SOMBRA_TLS_KEY`	The Sombra TLS key as a base64-encoded string
`TRUSTED_CLIENT_CA_CERT_ENCODED`	`VAULT_PATH_TRUSTED_CLIENT_CA_CERT_ENCODED`	`VAULT_VERSION_TRUSTED_CLIENT_CA_CERT_ENCODED`	`VAULT_KEY_TRUSTED_CLIENT_CA_CERT_ENCODED`	The public CA certificate from a client connecting to the internal Sombra service over mutual TLS. When set, Sombra will enforce that all incoming requests to non-health routes include a client certificate

External Secrets Management

Why External Secrets Operator?

Setting Up External Secrets Operator

1. Install External Secrets Operator

2. Configure Secret Store

AWS Secrets Manager

HashiCorp Vault

Azure Key Vault

Google Cloud Secret Manager

Other Secret Management Systems

3. Create ExternalSecret Resources

Using Transcend Helm Chart with External Secrets

1. Configure Helm Values

2. Deploy with Helm

Troubleshooting

Direct Vault Integration (Legacy: No longer recommended)

Vault Secret Configuration