# LLM Classifier (/docs/sombra/deploying/customizing-sombra/llm-classifier)

The LLM (Large Language Model) Classifier is an application designed to improve data classification accuracy using advanced natural language processing techniques. It is a separate, optional service that is attached to Sombra. Customers self-hosting Sombra can run the LLM Classifier in the same private network as Sombra.

## Deployment [#deployment]

Our [deployment guides](../deployment-options/index.mdx) provide instructions on how to deploy the LLM Classifier using different configuration setups. If you are using our recommended Helm Chart, deploying the LLM Classifier simply requires adding:

```yaml
llm-classifier:
  enabled: true
```

to your `values.yaml` file.

## Architecture [#architecture]

The LLM Classifier container runs a [gunicorn](https://gunicorn.org/) server that listens to requests and performs LLM classification on the inputs provided in the request body. Since the LLM performs more efficiently on Graphics Processing Units (GPUs), the container must run on a node with a supported NVIDIA GPU (see GPU Requirements below).

### Networking [#networking]

The LLM Classifier container by default listens on port `6081`. This can be changed using the `LLM_SERVER_PORT` environment variable.

To enable HTTPS connections to the LLM Classifier server, you can mount the SSL certificate and key file to the container and set the path to these files using the environment variables `LLM_CERT_PATH` and `LLM_KEY_PATH` respectively.

## Accessing the Docker Image [#accessing-the-docker-image]

You can pull our image from Transcend's private Docker registry using basic authentication.

First, please contact us and request permission to pull the `llm-classifier` image. We will then add your Transcend account to our permissions list.

Once we have added you to our allow list, you can log in to our private registry:

```shell
docker login docker.transcend.io
```

You will be prompted to enter the basic auth credentials. The username will always be "Transcend" (this is case-sensitive), and the password will be any API Key for your organization [within the Admin Dashboard](https://app.transcend.io/infrastructure/api-keys) (note: a scope is not required for the API key).

Once you've logged in, you may pull images by running:

```shell
docker pull docker.transcend.io/llm-classifier:<version_tag>
```

## GPU Requirements [#gpu-requirements]

The LLM Classifier runs on NVIDIA GPUs with at least 16 GB of VRAM, but instance choice affects both throughput and the NER tuning you should use. L4 generally offers the best balance of performance and cost where available. A10G remains a solid option. T4 is supported, but it is significantly slower and should generally be treated as a lower-throughput tier that benefits from tighter NER caps.

Recommended starting points:

- **L4 (`g6.xlarge`** **/** **`g6.2xlarge`)**: preferred tier; use the default NER settings (`80` / `20`).
- **A10G (`g5.xlarge`** **/** **`g5.2xlarge`)**: good production option; use the default NER settings (`80` / `20`).
- **T4 (`g4dn.2xlarge`)**: supported but slower; set `NER_INFERENCE_BATCH_SIZE=40` and `NER_MAX_CHUNKS_PER_INPUT=10`.

### Minimum GPU Specification [#minimum-gpu-specification]

| **Requirement**               | **Value**            |
| ----------------------------- | -------------------- |
| GPU vendor                    | NVIDIA               |
| Minimum VRAM (functional)     | 16 GB                |
| Recommended VRAM (production) | 24 GB or more        |
| CUDA support                  | Required (CUDA 12.x) |
| GPU count per node            | 1                    |

### Recommended Instance Types [#recommended-instance-types]

| **Cloud provider** | **Instance type**                 | **GPU**            | **GPU VRAM** | **Instance RAM** | **vCPU** |
| ------------------ | --------------------------------- | ------------------ | ------------ | ---------------- | -------- |
| AWS                | `g6.xlarge`                       | 1× NVIDIA L4       | 24 GB        | 16 GB            | 4        |
| AWS                | `g4dn.2xlarge` (dev/test)         | 1× NVIDIA T4       | 16 GB        | 32 GB            | 8        |
| AWS                | `g5.xlarge`                       | 1× NVIDIA A10G     | 24 GB        | 16 GB            | 4        |
| AWS                | `g5.2xlarge`                      | 1× NVIDIA A10G     | 24 GB        | 32 GB            | 8        |
| GCP                | `g2-standard-8`                   | 1× NVIDIA L4       | 24 GB        | 32 GB            | 8        |
| Azure              | `Standard_NC8as_T4_v3` or similar | 1× NVIDIA T4 or L4 | 16–24 GB     | 56 GB            | 8        |

**Notes:** The LLM Classifier has been tested on NVIDIA T4, L4, and A10G GPUs. L4 generally provides the best balance of speed and cost for this workload and is recommended for most production deployments. A10G is slightly faster than L4 at a similar VRAM size and remains a solid choice where available. T4 is fully supported but significantly slower; we recommend it primarily for development, testing, or low-volume environments. If you use another cloud provider or GPU model, ensure it has at least 16 GB of VRAM (with 24 GB or more recommended for production throughput).

### Pod Resource Requirements [#pod-resource-requirements]

Each LLM Classifier pod requires:

| Resource              | Value                   |
| --------------------- | ----------------------- |
| GPU                   | 1x `nvidia.com/gpu`     |
| Memory                | 15 GB                   |
| Replicas (production) | 2 (minimum recommended) |

The Kubernetes cluster must support `nvidia.com/gpu` as a schedulable resource. Ensure the [NVIDIA device plugin](https://github.com/NVIDIA/k8s-device-plugin) is installed in your cluster.

### Scaling [#scaling]

| Setting                             | Value                           |
| ----------------------------------- | ------------------------------- |
| Minimum replicas                    | 1 (dev/staging), 2 (production) |
| Maximum replicas                    | 2–4                             |
| Scale-out trigger (GPU utilization) | \~60% average                   |
| Scale-out trigger (response time)   | \~20 seconds average            |

The Horizontal Pod Autoscaler (HPA) monitors GPU utilization and response time. When either metric exceeds the target threshold, additional pods are scheduled. Throughput scales linearly with the number of pods — adding more replicas will proportionally increase classification capacity.

## Cost Estimates [#cost-estimates]

| Configuration        | Instance type   | Pricing model   | Estimated monthly cost |
| -------------------- | --------------- | --------------- | ---------------------- |
| Single node          | `g6.xlarge`     | On-demand       | \~$590/month           |
| Single node          | `g6.xlarge`     | 1-year reserved | \~$385/month           |
| Single node          | `g6.xlarge`     | 3-year reserved | \~$270/month           |
| Single node          | `g5.2xlarge`    | On-demand       | \~$880/month           |
| Single node          | `g5.2xlarge`    | 1-year reserved | \~$560/month           |
| Single node          | `g5.2xlarge`    | 3-year reserved | \~$380/month           |
| Production (2 nodes) | 2x `g5.2xlarge` | 1-year reserved | \~$1,120/month         |

Pricing is approximate and based on AWS US East (N. Virginia) region. Check [AWS EC2 pricing](https://aws.amazon.com/ec2/pricing/on-demand/) and [reserved instance pricing](https://aws.amazon.com/ec2/pricing/reserved-instances/pricing/) for current rates. Other cloud providers will have comparable pricing for equivalent GPU instances.

If you need more throughput (more classifications per hour), you can add more instances of the LLM Classifier to linearly scale the throughput.

## Helm Configuration [#helm-configuration]

This [values.yaml](https://github.com/transcend-io/helm-charts/blob/main/charts/sombra/values.yaml) adds an accompanying LLM Classifier to your Sombra deployment. The LLM Classifier requires an NVIDIA GPU to run, so please make sure your cluster supports `nvidia.com/gpu` as a resource.

```yaml
envs:
  # ... other env vars
  - name: LLM_CLASSIFIER_URL
    value: http://<release-name>-llm-classifier.transcend.svc:6081

llm-classifier:
  enabled: true
```

Or with TLS termination at Sombra and the LLM Classifier server:

```yaml
envs:
  # ... other env vars
  - name: LLM_CLASSIFIER_URL
    value: https://<release-name>-llm-classifier.transcend.svc:6081

envs_as_secret:
  # ... other env vars
  - name: SOMBRA_TLS_CERT
    value: <SOMBRA_TLS_CERT>
  - name: SOMBRA_TLS_KEY
    value: <SOMBRA_TLS_KEY>
  # An optional passphrase associated with your TLS private key. If you set a
  # passphrase when you created your key and certificate, you must provide it here.
  - name: SOMBRA_TLS_KEY_PASSPHRASE
    value: <SOMBRA_TLS_KEY_PASSPHRASE>

llm-classifier:
  enabled: true
  tls:
    enabled: true
    # saved as secret
    cert: |-
      -----BEGIN CERTIFICATE-----
      <base64>
      -----END CERTIFICATE-----
    # saved as secret
    key: |-
      -----BEGIN PRIVATE KEY-----
      <base64>
      -----END PRIVATE KEY-----

  # volume containing cert and key
  volumes:
    - name: llm-classifier-ssl
      secret:
        secretName: llm-classifier-secrets

  # mount the directory containing the cert and key to pod
  volumeMounts:
    - mountPath: '/etc/llm-classifier/ssl'
      name: llm-classifier-ssl
      readOnly: true

  # Set the location of cert and key in environment
  envs:
    - name: LLM_CERT_PATH
      value: '/etc/llm-classifier/ssl/llm-classifier.cert'
    - name: LLM_KEY_PATH
      value: '/etc/llm-classifier/ssl/llm-classifier.key'
```

### Environment Variables [#environment-variables]

| Variable                        | Default | Description                                                                                                                                                                                                                                                                                       |
| ------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `LLM_SERVER_PORT`               | `6081`  | Port the classifier listens on                                                                                                                                                                                                                                                                    |
| `LLM_SERVER_CONCURRENCY`        | `2`     | Number of Gunicorn workers. For most deployments, keep this aligned with your intended per-pod concurrency rather than GPU count alone.                                                                                                                                                           |
| `LLM_SERVER_WORKER_CONNECTIONS` | `1000`  | Maximum simultaneous connections per worker.                                                                                                                                                                                                                                                      |
| `LLM_SERVER_TIMEOUT`            | `120`   | Request timeout in seconds                                                                                                                                                                                                                                                                        |
| `LLM_SERVER_BACKLOG`            | `500`   | Maximum queued connections.                                                                                                                                                                                                                                                                       |
| NER_INFERENCE_BATCH_SIZE        | 80      | Maximum number of NER sequences processed in a single GLiNER forward pass. Lower this on smaller GPUs if you see CUDA OOM during unstructured classification.                                                                                                                                     |
| NER_MAX_CHUNKS_PER_INPUT        | 20      | Maximum number of text chunks processed per input for NER. This bounds work for very large documents and helps keep requests within the upstream timeout window. (https\://github.com/transcend-io/main/blob/dev/backend-services/python-llm-subdatapoint-classifier/src/classifier/constants.py) |
| `LLM_CERT_PATH`                 | —       | Path to TLS certificate (enables HTTPS)                                                                                                                                                                                                                                                           |
| `LLM_KEY_PATH`                  | —       | Path to TLS private key (enables HTTPS)                                                                                                                                                                                                                                                           |

**GPU-specific NER tuning:** The default `NER_INFERENCE_BATCH_SIZE=80` and `NER_MAX_CHUNKS_PER_INPUT=20` settings are appropriate for 24 GB GPUs such as L4 and A10G. On T4-class 16 GB GPUs, we recommend lowering these values to `40` and `10` respectively to reduce the risk of CUDA out-of-memory errors during large-document NER scans.