# Elasticsearch Integration Set Up for DSR Automation (/docs/integrations/elasticsearch/elasticsearch-integration-set-up-for-dsr-automation)

Configure your Elasticsearch integration to fulfill Data Subject Access Requests (DSRs) — both `ACCESS` (right-to-know) and `ERASURE` (right-to-be-forgotten) — against any index in your cluster.

## Overview [#overview]

With Transcend's Elasticsearch integration, you can fulfill `ACCESS` and `ERASURE` DSRs directly against the documents stored in your Elasticsearch indices.

Elasticsearch indices are schema-flexible, which means PII often lands in fields you didn't plan for — emails inside log payloads, user IDs in clickstream events, names buried in error messages. Transcend's integration is built for that reality: every index that gets discovered through <GlossaryTerm id="datapoint-schema-discovery">datapoint schema discovery</GlossaryTerm> automatically picks up DSR support. DSR's will run the `ACCESS` query and `ERASURE` redaction script, which you define on a per-index basis using the **Edit Query** UI on the datapoint.

Because Elasticsearch is most often used as a logging or analytics store, this integration **redacts** matching fields rather than deleting documents. That preserves log integrity and audit trails while still removing the personal data.

## 1. Configuring Datapoints for DSR Automation [#1-configuring-datapoints-for-dsr-automation]

The first step is making sure your Elasticsearch indices are registered as datapoints in Transcend. The recommended path is to run **Schema Discovery** against your Elasticsearch silo — this enumerates every non-system index and surfaces its fields automatically.

Once schema discovery has run, every discovered index is treated as one  <GlossaryTerm id="datapoint">datapoint</GlossaryTerm>. You'll then need to enable `ACCESS` and `ERASURE` actions for the relevant indices, as well as define a query JSON statement for the enabled actions.

Note: System indices (those whose names start with `.`, like `.kibana` and `.security-*`) are excluded from discovery by default.

## 2. Supported Query Statements [#2-supported-query-statements]

Transcend's Elasticsearch integration supports both `ACCESS` and `ERASURE`. Each datapoint accepts a JSON payload. You can either enter the statement manually or click "Generate ACCESS Query Statements" to auto-generate the statement.

| **Data Access** | **Auto-generated Statement**                                                                               | **Manually Defined Statement** |
| --------------- | ---------------------------------------------------------------------------------------------------------- | ------------------------------ |
| `ACCESS`        | Field-agnostic `query_string` search against every analyzed field, matching the request identifier         | A custom Query DSL body        |
| `ERASURE`       | Redact every top-level field of every document returned by `ACCESS`, replacing the value with `[REDACTED]` | A custom script body           |

## 3. Enabling `ACCESS` [#3-enabling-access]

For `ACCESS`, your query JSON statement is a JSON object matching [Elasticsearch's Query DSL](https://www.elastic.co/docs/explore-analyze/query-filter/languages/querydsl). What you provide is the **value of the** **`query`** **field** in a `_search` request — not the full request body.

Use the `{{identifier}}` placeholder anywhere inside your query statement where the DSR identifier (the data subject's email, user ID, etc.) should be substituted. Note: Transcend's Sombra encryption gateway fills `{{identifier}}` in just before the request leaves — the integration code itself never sees the plaintext identifier.

### Auto-generated `ACCESS` Query [#auto-generated-access-query]

If you click "Generate ACCESS Query Statement", Transcend auto-generates the following query JSON payload:

```json
{
  "query_string": {
    "query": "{{identifier}}",
    "default_operator": "AND"
  }
}
```

This matches any document where every token in the identifier appears in at least one analyzed field. The `AND` operator keeps multi-token identifiers (e.g. `"first last"`) from over-matching.

### Sample Custom `ACCESS` Query [#sample-custom-access-query]

Here is an example of a custom payload using a `bool` compound query:

```json
{
  "bool": {
    "should": [
      { "term":   { "message": "quick" }},
      { "term":   { "message": "brown" }},
      { "prefix": { "message": "f" }}
    ]
  }
}
```

In this example, Transcend will return any document whose `message` field contains the term `quick`, the term `brown`, or starts with `f`.

A more typical query statement that targets a specific field by identifier could look like this:

```json
{
  "bool": {
    "must": [
      { "term": { "user.email.keyword": "{{identifier}}" }}
    ]
  }
}
```

### Requirements [#requirements]

- The payload must be a **JSON object** -- not an array, not a raw string.
- Only the contents of the `query` parameter are supplied -- do **not** wrap your override in an outer `{ "query": { ... } }`.
- `{{identifier}}` is the only placeholder Transcend substitutes. Any other `{{...}}` token is sent to Elasticsearch as-is.

## 4. Enabling `ERASURE` [#4-enabling-erasure]

`ERASURE` runs against the same set of documents `ACCESS` finds. For every document `ACCESS` returns, Transcend calls `POST /{index}/_update/{_id}` with a Painless script that determines how the document is rewritten.

Important: `ACCESS` must be enabled for `ERASURE` to be run successfully, as `ERASURE` depends on the results of `ACCESS`.

The `ERASURE` query statement is a JSON object with **only** the following keys, matching [Elasticsearch's script object shape](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update-by-query#operation-update-by-query-body-application-json-script-source): `source`, `lang`, and `params`.

### Auto-generated `ERASURE` Query [#auto-generated-erasure-query]

If you click "Generate ERASURE Query Statements", Transcend auto-generates the following query statement:

```json
{
  "source": "List keys = new ArrayList(ctx._source.keySet()); for (def key : keys) { ctx._source[key] = params.redacted; }",
  "lang": "painless",
  "params": {
    "redacted": "[REDACTED]"
  }
}
```

This replaces **every top-level field** of `_source` with the string `[REDACTED]`. The document itself is preserved — only its field values are overwritten. This is the safest default because it guarantees no personal data remains in the document, regardless of which fields contain it.

### Sample Custom `ERASURE` Query [#sample-custom-erasure-query]

If you want to redact only specific fields (e.g. only `email` and `name`), provide a custom script:

```json
{
  "source": "if (ctx._source.containsKey('email')) { ctx._source.email = params.redacted; } if (ctx._source.containsKey('name')) { ctx._source.name = params.redacted; }",
  "lang": "painless",
  "params": {
    "redacted": "[REDACTED]"
  }
}
```

## 5. How `ACCESS` and `ERASURE` Interact [#5-how-access-and-erasure-interact]

`ACCESS` and `ERASURE` are wired together. The flow on every DSR is:

1. `ACCESS` runs the configured (or default) Query DSL search against the index. Matching documents are streamed back through Transcend's Sombra encryption gateway.
2. For each matching document, Transcend records the document `_id`.
3. `ERASURE` calls `_update/{_id}` once per document, applying the configured (or default) Painless script.

This means **scoping** **`ACCESS`** **also scopes** **`ERASURE`**. If your `ACCESS` query is broad, your `ERASURE` will run against a broad set of documents. If your `ACCESS` query is narrow, only those documents are redacted.

It also means `ERASURE` does **not** delete documents. The document remains in the index; its field values are overwritten. This is intentional: most Elasticsearch deployments are used for logs, and deleting log entries breaks audit-trail integrity. Redaction satisfies the privacy requirement while leaving the surrounding event record intact.