Quickstart

This guide covers deploying Sombra in an AWS environment using Terraform with Kubernetes (EKS) as the deployment platform.

If you prefer to try running Sombra locally or with simpler configuration, see our Docker Run guide.

For other deployment options, refer to the Deployment Options page.

Before starting, you need access to Transcend's container registry to pull our Sombra image. If you don't have access or are unsure, contact your Transcend account representative.

Create a Transcend API Key in the Admin Dashboard under Infrastructure → API Keys. No specific scope is required for this API key.

Make sure to save this API key securely as it will only be displayed once.

  1. Go to the Sombra Gateways page

  2. Click "Create New Self Hosted Sombra"

  3. Select "Self-Hosted Sombra - Reverse Tunnel" from the dropdown

  4. Leave the "Existing Sombra ID" field blank, unless you're re-using a Sombra deployment currently registered with another organization

    Creating a Self Hosted Sombra using the Transcend Reverse Tunnel

Click the "Create" button to generate a configuration snippet that will look similar to:

SOMBRA_ID=xxxx
SOMBRA_REVERSE_TUNNEL_API_KEY=xxxx
ORGANIZATION_URI=xxxx
TRANSCEND_URL=https://api.transcend.io

Generate the JWT_ECDSA_KEY value with your by running:

openssl ecparam -genkey -name secp384r1 -noout | (base64 --wrap=0 2>/dev/null || base64 -b 0)

The Terraform snippet below contains all the code needed to:

  1. Create a new VPC in AWS
  2. Create a new EKS cluster in that VPC
  3. Deploy Sombra to the cluster
  4. Deploy the LLM Classifier to the cluster

Fill in the following values with results from the previous steps of this guide:

  • <API_KEY_FROM_STEP_1_FROM_STEP_1>
  • <ORGANIZATION_URI_FROM_STEP_2>
  • <SOMBRA_ID_FROM_STEP_2>
  • <SOMBRA_REVERSE_TUNNEL_API_KEY_FROM_STEP_2>
  • <TRANSCEND_URL_FROM_STEP_2>
  • <JWT_ECDSA_KEY_FROM_STEP_3>

If you don't need the LLM Classifier for data classification, you can save resources by changing llm-classifier = { enabled = true } to llm-classifier = { enabled = false }.

Go to the Sombra Gateways page and click the "Test" button under "Test Gateway Connection" to make a GET request to the /test endpoint of the Sombra Gateway.

Buttons to test the Sombra connection

You should see a successful message if the connection is working. Congratulations on successfully deploying Sombra and the LLM Classifier!

provider "aws" {
  region  = "eu-west-1"
}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.1.2"

  name = "sombra-vpc"
  cidr = "10.0.0.0/16"
  azs  = ["eu-west-1a", "eu-west-1b"]

  private_subnets = ["10.0.101.0/24", "10.0.102.0/24"]
  public_subnets  = ["10.0.201.0/24", "10.0.202.0/24"]

  enable_nat_gateway   = true
  enable_dns_hostnames = true
  enable_dns_support   = true
}

module "eks_cluster" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.31.6"

  cluster_name    = "sombra-eks-cluster"
  cluster_version = "1.32"

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  cluster_endpoint_public_access = true

  enable_cluster_creator_admin_permissions = true

  cluster_compute_config = {
    enabled    = true
    node_pools = ["general-purpose"]
  }
}

###################################################################################
# Everything below here depends on the above kubernetes cluster already existing, #
#   so you may want to comment out the below resources during the first apply     #
###################################################################################

provider "kubernetes" {
  host                   = module.eks_cluster.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks_cluster.cluster_certificate_authority_data)
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    args        = ["eks", "get-token", "--cluster-name", module.eks_cluster.cluster_name]
    command     = "aws"
  }
}

provider "helm" {
  kubernetes {
    host                   = module.eks_cluster.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks_cluster.cluster_certificate_authority_data)
    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      args        = ["eks", "get-token", "--cluster-name", module.eks_cluster.cluster_name]
      command     = "aws"
    }
  }
}

resource "kubernetes_manifest" "allow_gpu_node_pool" {
  manifest = {
    apiVersion = "karpenter.sh/v1"
    kind       = "NodePool"
    metadata = {
      name = "default"
    }
    spec = {
      template = {
        spec = {
          nodeClassRef = {
            group = "eks.amazonaws.com"
            kind  = "NodeClass"
            name  = "default"
          }
          requirements = [
            {
              key = "eks.amazonaws.com/instance-gpu-manufacturer"
              operator = "In"
              values = ["nvidia"]
            },
            # The A10G GPU has been tested to work well with our LLM Classifier
            {
              key = "eks.amazonaws.com/instance-gpu-name"
              operator = "In"
              values = ["a10g"]
            }
          ]
        }
      }
      limits = {
        cpu    = "16000"
        memory = "64Gi"
      }
    }
  }
}

resource "helm_release" "sombra" {
  name             = "sombra-servers"
  namespace        = "sombra"
  chart            = "sombra"
  create_namespace = true
  repository       = "https://transcend-io.github.io/helm-charts/"
  version          = "0.5.0"
  timeout          = 900 # 15 minutes for any helm operation
  values = [yamlencode({
    imageCredentials = {
      registry = "docker.transcend.io"
      username = "Transcend"
      password = "<API_KEY_FROM_STEP_1_FROM_STEP_1>"
    }
    replicaCount = 1
    envs = [
      {
        name  = "ORGANIZATION_URI"
        value = "<ORGANIZATION_URI_FROM_STEP_2>"
      },
      {
        name  = "SOMBRA_ID"
        value = "<SOMBRA_ID_FROM_STEP_2>"
      },
      {
        name  = "SOMBRA_REVERSE_TUNNEL_API_KEY"
        value = "<SOMBRA_REVERSE_TUNNEL_API_KEY_FROM_STEP_2>"
      },
      {
        name  = "TRANSCEND_URL"
        value = "<TRANSCEND_URL_FROM_STEP_2>"
      },
      {
        name  = "LLM_CLASSIFIER_URL"
        value = "http://sombra-servers-llm-classifier.transcend.svc.cluster.local:6081/"
      }
    ]
    envs_as_secret = [
      {
        name  = "JWT_ECDSA_KEY"
        value = "<JWT_ECDSA_KEY_FROM_STEP_3>"
      }
    ]
    llm-classifier = { enabled = true }
    affinity = {
      nodeAffinity = {
        requiredDuringSchedulingIgnoredDuringExecution = {
          nodeSelectorTerms = [{
            matchExpressions = [{
              key      = "eks.amazonaws.com/instance-gpu-manufacturer"
              operator = "DoesNotExist"
            }]
          }]
        }
      }
    }
  })]
}