# External Secrets Management (/docs/sombra/deploying/customizing-sombra/secrets-management)

Sombra requires secure access to sensitive configuration values like JWT signing keys and TLS certificates. We recommend using the [External Secrets Operator (ESO)](https://external-secrets.io/latest/) to securely inject secrets from external secret management systems into your Kubernetes environment.

## Why External Secrets Operator? [#why-external-secrets-operator]

The External Secrets Operator provides several advantages for managing Sombra secrets:

- **Centralized Secret Management**: Integrate with your existing secret management infrastructure (AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, Google Secret Manager, etc.)
- **Kubernetes Native**: Works seamlessly with Kubernetes secrets and environment variables
- **Automatic Synchronization**: Keeps secrets up-to-date automatically
- **Security Best Practices**: Follows Kubernetes security patterns and RBAC
- **Least Privilege**: Use workload identity to grant only read access to specific secret paths/names
- **Multi-Provider Support**: Supports 30+ secret management providers

## Setting Up External Secrets Operator [#setting-up-external-secrets-operator]

### 1. Install External Secrets Operator [#1-install-external-secrets-operator]

Install ESO in your Kubernetes cluster:

```shellscript
helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm install external-secrets external-secrets/external-secrets -n external-secrets-system --create-namespace
```

### 2. Configure Secret Store [#2-configure-secret-store]

Create a `SecretStore` (namespaced) or `ClusterSecretStore` (cluster-wide) resource to connect to your secret management system. For multi-namespace deployments, prefer `ClusterSecretStore`.

#### AWS Secrets Manager [#aws-secrets-manager]

- **Provider Documentation**: [External Secrets Operator: AWS Secrets Manager Provider](https://external-secrets.io/latest/provider/aws-secrets-manager/)
- **Recommended Authentication**: Use IAM roles for service accounts (IRSA) for authentication.
- **Alternative**: Use [AWS Parameter Store](https://external-secrets.io/latest/provider/aws-parameter-store/).

If using IRSA, first create a `ServiceAccount` with IAM role binding:

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sombra
  namespace: sombra
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>
```

Then create the `ClusterSecretStore`:

```yaml
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: my-secret-store
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-west-2
      auth:
        jwt:
          serviceAccountRef:
            name: sombra
            namespace: sombra
```

#### HashiCorp Vault [#hashicorp-vault]

- **Provider Documentation**: [External Secrets Operator: HashiCorp Vault Provider](https://external-secrets.io/latest/provider/hashicorp-vault/)
- **Recommended Authentication**: Use Kubernetes authentication with a Vault role.

If using Kubernetes authentication, first create a `ServiceAccount`:

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sombra
  namespace: sombra
```

Then create the `ClusterSecretStore`:

```yaml
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: my-secret-store
spec:
  provider:
    vault:
      server: 'https://vault.example.com'
      path: 'secret'
      version: 'v2'
      auth:
        kubernetes:
          mountPath: 'kubernetes'
          role: 'sombra-role'
          serviceAccountRef:
            name: sombra
            namespace: sombra
```

#### Azure Key Vault [#azure-key-vault]

- **Provider Documentation**: [External Secrets Operator: Azure Key Vault Provider](https://external-secrets.io/latest/provider/azure-key-vault/)
- **Recommended Authentication**: Use Workload Identity for authentication.

If using Workload Identity, first create a `ServiceAccount`:

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sombra
  namespace: sombra
  annotations:
    azure.workload.identity/client-id: 7d8cdf74-xxxx-xxxx-xxxx-274d963d358b
    azure.workload.identity/tenant-id: 5a02a20e-xxxx-xxxx-xxxx-0ad5b634c5d8
```

Then create the `ClusterSecretStore`:

```yaml
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: my-secret-store
spec:
  provider:
    azurekv:
      vaultUrl: 'https://your-keyvault.vault.azure.net'
      authType: WorkloadIdentity
      serviceAccountRef:
        name: sombra
        namespace: sombra
```

#### Google Cloud Secret Manager [#google-cloud-secret-manager]

- **Provider Documentation**: [External Secrets Operator: Google Secret Manager Provider](https://external-secrets.io/latest/provider/google-secrets-manager/)
- **Recommended Authentication**: Use Workload Identity for authentication. See the [Google Cloud documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) for more information.

If using Workload Identity, first create a `ServiceAccount`:

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sombra
  namespace: sombra
  annotations:
    iam.gke.io/gcp-service-account: sombra@your-project-id.iam.gserviceaccount.com
```

Then create the `ClusterSecretStore`:

```yaml
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: my-secret-store
spec:
  provider:
    gcpsm:
      projectID: 'your-project-id'
      auth:
        workloadIdentity:
          serviceAccountRef:
            name: sombra
            namespace: sombra
```

#### Other Secret Management Systems [#other-secret-management-systems]

For other secret management systems, see the [External Secrets Operator providers documentation](https://external-secrets.io/latest/provider/aws-secrets-manager/).

### 3. Create ExternalSecret Resources [#3-create-externalsecret-resources]

Create `ExternalSecret` resources to define which secrets to fetch:

```yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: sombra-external-secrets
  namespace: sombra
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: my-secret-store
    kind: ClusterSecretStore
  target:
    name: sombra-external-secrets
    creationPolicy: Owner
    deletionPolicy: Retain # or Delete; choose based on your cleanup needs
  data:
    - secretKey: SOMBRA_REVERSE_TUNNEL_API_KEY
      remoteRef:
        key: sombra/reverse-tunnel-api-key
        property: value
    - secretKey: JWT_ECDSA_KEY
      remoteRef:
        key: sombra/jwt-ecdsa-key
        property: value
    - secretKey: INTERNAL_KEY_HASH
      remoteRef:
        key: sombra/internal-key-hash
        property: value
```

## Using Transcend Helm Chart with External Secrets [#using-transcend-helm-chart-with-external-secrets]

Use `envFrom` to load all secrets from the ESO-managed secret directly:

### 1. Configure Helm Values [#1-configure-helm-values]

```yaml
# values.yaml
envFrom:
  - secretRef:
      name: sombra-external-secrets
```

### 2. Deploy with Helm [#2-deploy-with-helm]

```shellscript
helm install sombra transcend/sombra \
  --namespace <sombra-namespace> \
  --values values.yaml
```

This approach loads all environment variables from the `sombra-external-secrets` secret that External Secrets Operator creates.

<Callout type="info">
  **Secret Rotation**: Environment variables don't update in running pods. Consider using [Stakater Reloader](https://github.com/stakater/Reloader) or triggering a rollout when secrets change.
</Callout>

## Troubleshooting [#troubleshooting]

Common issues and debugging steps:

```shellscript
# Check ExternalSecret status
kubectl get externalsecret sombra-external-secrets -n sombra -o yaml

# Verify Kubernetes secret was created
kubectl describe secret sombra-external-secrets -n sombra

# Check External Secrets Operator logs
kubectl logs -n external-secrets-system deployment/external-secrets
```

**Common errors**:

- Wrong provider path/property in `remoteRef`
- Missing ServiceAccount/IAM role permissions
- Wrong namespace for `SecretStore` (use `ClusterSecretStore` for multi-namespace)

## Direct Vault Integration (Legacy: No longer recommended) [#direct-vault-integration-legacy-no-longer-recommended]

<Callout type="warn">
  Direct Vault integration is no longer recommended. Use External Secrets Operator instead for better security and maintainability.
</Callout>

If you must use the direct Vault integration, Sombra supports fetching secrets from HashiCorp Vault using a sidecar container with vault-agent. This approach is not recommended for new deployments.

To enable the direct Vault integration, set these environment variables:

| Env Var Name           | Description                                                                           | Type    | Example                 |
| ---------------------- | ------------------------------------------------------------------------------------- | ------- | ----------------------- |
| `ENABLE_VAULT_FETCHER` | Set to `true` to enable Vault integration                                             | boolean | `true`                  |
| `VAULT_AGENT_URI`      | The URI of the Vault agent (typically a local address when using a sidecar container) | string  | `http://127.0.0.1:8100` |

### Vault Secret Configuration [#vault-secret-configuration]

The following environment variables can be configured to fetch secrets from Vault:

| Secret Env Var Name              | Vault Path Env Var                          | Vault Version Env Var                          | Vault Key Name Env Var                     | Description                                                                                                                                                                                                   |
| -------------------------------- | ------------------------------------------- | ---------------------------------------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `SOMBRA_JWT_ECDSA_KEY`           | `VAULT_PATH_SOMBRA_JWT_ECDSA_KEY`           | `VAULT_VERSION_SOMBRA_JWT_ECDSA_KEY`           | `VAULT_KEY_SOMBRA_JWT_ECDSA_KEY`           | The JWT signing key used for webhook signatures and JWT operations. Also serves as entropy source for local KMS key derivation when not using AWS KMS                                                         |
| `SOMBRA_TLS_KEY_PASSPHRASE`      | `VAULT_PATH_SOMBRA_TLS_KEY_PASSPHRASE`      | `VAULT_VERSION_SOMBRA_TLS_KEY_PASSPHRASE`      | `VAULT_KEY_SOMBRA_TLS_KEY_PASSPHRASE`      | An optional password for the TLS certificate                                                                                                                                                                  |
| `SOMBRA_TLS_CERT`                | `VAULT_PATH_SOMBRA_TLS_CERT`                | `VAULT_VERSION_SOMBRA_TLS_CERT`                | `VAULT_KEY_SOMBRA_TLS_CERT`                | The Sombra TLS certificate as a base64-encoded string                                                                                                                                                         |
| `SOMBRA_TLS_KEY`                 | `VAULT_PATH_SOMBRA_TLS_KEY`                 | `VAULT_VERSION_SOMBRA_TLS_KEY`                 | `VAULT_KEY_SOMBRA_TLS_KEY`                 | The Sombra TLS key as a base64-encoded string                                                                                                                                                                 |
| `TRUSTED_CLIENT_CA_CERT_ENCODED` | `VAULT_PATH_TRUSTED_CLIENT_CA_CERT_ENCODED` | `VAULT_VERSION_TRUSTED_CLIENT_CA_CERT_ENCODED` | `VAULT_KEY_TRUSTED_CLIENT_CA_CERT_ENCODED` | The public CA certificate from a client connecting to the internal Sombra service over mutual TLS. When set, Sombra will enforce that all incoming requests to non-health routes include a client certificate |