Using the API for DSR Automation

These endpoints enable Transcend customers to complete an end-to-end DSR (such as ACCESS or ERASURE) on behalf of their users.

These endpoints can be used in combination with, or in lieu of, Configuring the Privacy Center.

This flow can be achieved end-to-end via four endpoints:

All of the examples below demonstrate how to access Transcend with our default encryption configuration: multi-tenant Sombra.

In order to use these endpoints with a self-hosted Sombra instance, add the x-sombra-authorization header to requests and change the base URL from https://multi-tenant.sombra.transcend.io to your gateway's URL.

The first step in the process is to initiate a new data subject request using the following API:

Endpoint: Submit a DSR

POST

/v1/data-subject-request

Open in API Reference

When submitting requests via API there are a few key things to consider

When leveraging Transcend via an API-only approve, you will most likely want to set the flag isSilent: true so that Transcend skips sending of any emails and the entire communication process is handled by your code. It is possible to submit requests via API that are not in silent mode, but the user will then receive emails and be directed back to your Privacy Center to download the data.

All requests require a coreIdentifier to be specified - this is a unique identifier that can be associated back to the person that made the request. This value is used to determine if the person has already submitted a request (error message Client error: You have already made this request) or to list out other requests in the Transcend platform associated to thatperson. If you are using the DSR API and a Privacy Center - the coreIdentifier should be the same one that a customer could use in the Privacy Center to check on the status of their request(s).

When isSilent=false, the email identifier is also required in order to send the data subject updates about their DSR. If isSilent=true then the email field becomes optional.

Additional identifiers can be specified with the request via the attestedExtraIdentifiers object. Identifiers can be provided up front in the API call or enriched after the fact. Read more about identity enrichment.

When uploading requests via the API - it's useful to tag the request with custom fields that can be used to later sort requests. This could be a batch ID, the name of the application or server uploading the request, or other metadata useful for reporting purposes. Learn more about Custom Fields here.

Custom fields are provided via the attributes field:

JavaScript

const _ = {
  attributes: [
    {
      key: 'Source',
      values: ['Mobile iOS App'],
    },
  ],
};

When submitting requests to Transcend - you can do so in one of two ways:

Directly call Transcend's API just in time to submit the request.
Save the requests to a queue and upload those requests on some batched interval.

In most situations, Option 1 is the simplest and sufficient as it minimizes the amount of integration code/infrastructure needed to connect your product to Transcend. Option 2 can be benficial for situations where you want to ensure high availability of your platform and reduce the dependency of core functionality in your product being tied to Transcend'a availability. In this situation, you would save the intention of the DSR (e.g. a delete request) to some queue or table in your backend service. Then the service would periodically upload the requests to Transcend (e.g. once per hour). In addition to reducing your dependency on Transcend, this option allows for you to patch errors in your code without the user seeing an error on the client. For example, if you were to provide an invalid locale code, the Transcend API would reject the request. In such a situation, with Option 1 your user would see an error message - but in Option 2, the user would see no error, and you could push a patch to your code and have that request uploaded to Transcend after the patch is resolved.

If you go with Option 2 - it's recommended to upload requests to Transcend at most 50 requests in parallel. There is a limit of 100k requests that can be submitted per day.

A fully-functional prototype for uploading requests from a CSV file can be found in the Transcend CLI's tr-upload-request command.

The entrypoint for this command lives in the src/cli-request-upload.ts file.
The meat of the batching and parsing logic lives in the src/requests/uploadPrivacyRequestsFromCsv.ts) file.
The request is uploaded to Transcend in the src/requests/submitPrivacyRequest.ts file.

Here is some example code for submitting a new data subject request:

TypeScript

import {
  IsoCountryCode,
  IsoCountrySubdivisionCode,
  RequestAction,
  RequestStatus,
} from '@transcend-io/privacy-types';
import got from 'got';

interface TranscendPrivacyRequest {
  /** UUID of privacy request in Transcend */
  id: string;
  /** Link to the request in Transcend */
  link: string;
  /** Status of the request e.g. COMPILING | ENRICHING  */
  status: RequestStatus;
  /** Type of request e.g. ACCESS | ERASURE */
  type: RequestAction;
  /** The slug of the data subject type the request is associated for e.g. customer | employee */
  subjectType: string;
  /** Primary email address for the request */
  email: string | null;
  /** Core identifier used to determine if a person has multiple privacy requests open */
  coreIdentifier: string;
  /** Whether the request is in silent mode */
  isSilent: boolean;
  /** Whether the request is a test request */
  isTest: boolean;
  /** Country code associated with the request */
  country: IsoCountryCode | null;
  /** Country sub division code associated with the request */
  countrySubDivision: IsoCountrySubdivisionCode | null;
  /** Custom fields associated with the request */
  attributeValues: {
    /** Name of field */
    attributeKey: {
      /** Name of field */
      name: string;
    };
    /** Value attached to request */
    name: string;
  }[];
}

/**
 * Submit a new data subject request to transcend.io
 *
 * @see https://docs.transcend.io/docs/api-reference/POST/v1/data-subject-request
 * @returns The created request
 */
async function submitDataSubjectRequestToTranscend(): Promise<TranscendPrivacyRequest> {
  // Construct request instance with API keys
  const sombra = got.extend({
    // Find your API url under Infrastructure -> Sombra
    // https://app.transcend.io/infrastructure/sombra/sombras
    prefixUrl: 'https://multi-tenant.sombra.transcend.io', // EU hosting
    // prefixUrl: 'https://multi-tenant.us.sombra.transcend.io', // US hosting
    // prefixUrl: 'https://sombra.acme.com',                     // Self hosting
    headers: {
      Authorization: `Bearer ${process.env.TRANSCEND_API_KEY}`,
      ...(process.env.SOMBRA_API_KEY
        ? {
            'X-Sombra-Authorization': `Bearer ${process.env.SOMBRA_API_KEY}`,
          }
        : {}),
    },
  });

  // Make the GraphQL request
  try {
    const response = await sombra
      .post('v1/data-subject-request', {
        json: {
          type: 'ERASURE',
          subject: {
            coreIdentifier: '12345',
            email: 'test@transcend.io',
            // emailIsVerified: true,
            // attestedExtraIdentifiers: {
            //   email: [
            //     {
            //       value: 'another-email@example.com',
            //     },
            //   ],
            //   custom: [
            //     {
            //       value: 'mbrook',
            //       name: 'username',
            //     },
            //   ],
            // },
          },
          subjectType: 'customer',
          isSilent: true,
          // isTest: false,
          // locale: 'fr-FR',
          // details: 'Submitted via API',
          // attributes: [
          //   {
          //     key: 'Source',
          //     values: ['Mobile iOS App'],
          //   },
          // ],
          // region: {
          //   country: 'US',
          //   countrySubDivision: 'US-CA',
          // },
          // createdAt: '01/01/2025',
          // dataSiloIds: [...]
        },
      })
      .json<TranscendPrivacyRequest>();
    return response;
  } catch (err) {
    const msg = `${err.message} - ${JSON.stringify(
      err.response?.body,
      null,
      2,
    )}`;

    // Handle 401, unauthorized
    if (err.response?.statusCode === 401) {
      throw new Error(
        `API key credential permissions are insufficient: ${msg}`,
      );
    }

    // Handle rate limit
    if (err.response?.statusCode === 429) {
      throw new Error(`Failed to upload request due to rate limit: ${msg}`);
    }

    // Handle request submitted already
    if (msg.includes('Client error: You have already made this request')) {
      throw new Error(`Request has already been submitted: ${msg}`);
    }

    // Other errors
    throw new Error(`Received an error from server: ${msg}`);
  }
}

Once a DSR has been submitted, it will take some time to complete. The status of the DSR can be accessed via the following endpoint. This endpoint takes in the ID of the request, so you would likely want to store that ID after submitting the request in order to efficiently poll its status.

Endpoint: Poll DSR state

GET

/v1/data-subject-request/{id}

Open in API Reference

The full set of DSR statuses can be found here. The following statuses are most useful for detcting completion.

For data access requests, the request the following statuses indicate completion:

APPROVING -> the workflow has request approvals enabled and the compilation of data is completed and read for review
DOWNLOADABLE -> the request has been approved, all data compiled, and files are ready for download
VIEW_CATEGORIES -> the request has been approved, all data compiled, and files are ready for download - this status is used for category-only reports

SECONDARY_APPROVING -> the workflow is configured for approvals and the erasure has been completed
SECONDARY_COMPLETED -> the erasure has been completed

For all other request types, the completed statuses include:

APPROVING -> the workflow has been completed but is pending approval
COMPLETED -> the request has been approved and completed

Here is some example code for checking the status of a DSR, given the request ID from submitting the request.

TypeScript

import { RequestAction, RequestStatus } from '@transcend-io/privacy-types';
import got from 'got';

interface TranscendPrivacyRequestPreview {
  /** UUID of privacy request in Transcend */
  id: string;
  /** Status of the request e.g. COMPILING | ENRICHING  */
  status: RequestStatus;
  /** Type of request e.g. ACCESS | ERASURE */
  type: RequestAction;
  /** The slug of the data subject type the request is associated for e.g. customer | employee */
  subjectType: string;
  /** Primary email address for the request */
  email: string | null;
  /** Core identifier used to determine if a person has multiple privacy requests open */
  coreIdentifier: string;
}

/**
 * Check the status of a DSR
 *
 * @see https://docs.transcend.io/docs/api-reference/GET/v1/data-subject-request/(id)
 *
 * @parma requestId - UUID of request - returned from POST/v1/data-subject-request
 * @returns The updated request
 */
async function checkDataSubjectRequestStatus(
  requestId: string,
): Promise<TranscendPrivacyRequestPreview> {
  // Construct request instance with API keys
  const sombra = got.extend({
    // Find your API url under Infrastructure -> Sombra
    // https://app.transcend.io/infrastructure/sombra/sombras
    prefixUrl: 'https://multi-tenant.sombra.transcend.io', // EU hosting
    // prefixUrl: 'https://multi-tenant.us.sombra.transcend.io', // US hosting
    // prefixUrl: 'https://sombra.acme.com',                     // Self hosting
    headers: {
      Authorization: `Bearer ${process.env.TRANSCEND_API_KEY}`,
      ...(process.env.SOMBRA_API_KEY
        ? {
            'X-Sombra-Authorization': `Bearer ${process.env.SOMBRA_API_KEY}`,
          }
        : {}),
    },
  });

  // Make the GraphQL request
  try {
    const response = await sombra
      .get(`v1/data-subject-request/${requestId}`)
      .json<TranscendPrivacyRequestPreview>();
    return response;
  } catch (err) {
    const msg = `${err.message} - ${JSON.stringify(
      err.response?.body,
      null,
      2,
    )}`;

    // Handle 401, unauthorized
    if (err.response?.statusCode === 401) {
      throw new Error(
        `API key credential permissions are insufficient: ${msg}`,
      );
    }

    // Handle rate limit
    if (err.response?.statusCode === 429) {
      throw new Error(`Failed to upload request due to rate limit: ${msg}`);
    }

    // Other errors
    throw new Error(`Received an error from server: ${msg}`);
  }
}

In certain situations, rather than saving Transcend's request ID to your database, you can instead save your own request ID to Transcend and periodically poll Transcend to find all requests that are ready to be parsed back into your system.

In order to do this, you will first want to setup approvals on your workflow under DSR Automation -> Request Settings. For erasure requests you will want to enable "Approval Before Sending Erasure Report". For all other requests, you will want to enable "Approval Before Send ".

The high level process will look like:

Enable approval step on the relevant workflow(s)
Submit the requests to Transcend, passing your request ID as a custom field

JavaScript

const _ = {
  attributes: [
    {
      key: 'Internal Request ID',
      values: ['123456'],
    },
  ],
};

Poll Transcend to find all requests in status=SECONDARY_APPROVING for erasure requests, and status=APPROVING for all other request types.
For each request that is found in an approving state, grab the Internal Request ID from the custom field and perform an update in your system to mark the request as completed. For access requests, this would involve downloading the files (see below in this document) - for other request types, it will just be marking the job as fully completed.
After post-processing each request, call the Transcend API to approve the request so it is removed from the approval queue.

A fully-functional prototype of a process for downloading the files for data access requests can be found in Transcend CLI's tr-request-download-files command.

The entrypoint for this command lives in the src/cli-request-download-files.ts file.
The meat of the logic lives in src/requests/downloadPrivacyRequestFiles.ts) file.
The logic for querying in all requests in a certain state can be found in src/graphql/fetchAllRequests.ts) file.

Here is some example code for implementing this process:

TypeScript

import {
  IsoCountryCode,
  IsoCountrySubdivisionCode,
  RequestAction,
  RequestStatus,
} from '@transcend-io/privacy-types';
import { map } from 'bluebird';
import { gql, GraphQLClient } from 'graphql-request';

/**
 * GQL query to list requests in Transcend
 */
const REQUESTS = gql`
  query MyCustomImplementationRequests(
    $first: Int!
    $offset: Int!
    $filterBy: RequestFiltersInput!
  ) {
    requests(
      filterBy: $filterBy
      first: $first
      offset: $offset
      orderBy: [
        { field: createdAt, direction: ASC }
        { field: id, direction: ASC }
      ]
      useMaster: false
    ) {
      nodes {
        id
        createdAt
        email
        link
        status
        details
        isTest
        locale
        isSilent
        coreIdentifier
        type
        subjectType
        country
        countrySubDivision
        attributeValues {
          id
          name
          attributeKey {
            id
            name
          }
        }
      }
      totalCount
    }
  }
`;

/** GQL request to approve privacy requests */
const APPROVE_PRIVACY_REQUEST = gql`
  mutation MyCustomImplementationApprovePrivacyRequest(
    $input: CommunicationInput!
  ) {
    approveRequest(input: $input) {
      request {
        id
      }
    }
  }
`;

/** GQL request to approve privacy requests for type=ERASURE */
const APPROVE_PRIVACY_SECONDARY_REQUEST = gql`
  mutation MyCustomImplementationApprovePrivacyRequestSecondary(
    $input: CommunicationInput!
  ) {
    approveRequestSecondary(input: $input) {
      request {
        id
      }
    }
  }
`;

const PAGE_SIZE = 50;

type TranscendPrivacyRequestFull = {
  /** Request ID */
  id: string;
  /** Time request was made */
  createdAt: string;
  /** Email of request */
  email: string;
  /** The type of request */
  type: RequestAction;
  /** Link to request in Transcend dashboard */
  link: string;
  /** Whether request is in silent mode */
  isSilent: boolean;
  /** Whether request is a test request */
  isTest: boolean;
  /** The core identifier of the request */
  coreIdentifier: string;
  /** Request details */
  details: string;
  /** Locale of request */
  locale: LanguageKey;
  /** Status of request */
  status: RequestStatus;
  /** Type of data subject */
  subjectType: string;
  /** Country of request */
  country: IsoCountryCode | null;
  /** Subdivision of request */
  countrySubDivision: IsoCountrySubdivisionCode | null;
  /** Request attributes */
  attributeValues: Array<{
    id: string;
    attributeKey: {
      name: string;
      id: string;
    };
    name: string;
  }>;
};

/**
 * Fetch all requests matching a set of filters
 *
 * @param client - GraphQL client
 * @param options - Filter options
 * @returns List of requests
 */
async function fetchAllRequests(
  client: GraphQLClient,
  {
    actions = [],
    statuses = [],
    isTest,
    isSilent,
    isClosed,
  }: {
    /** Actions to filter on */
    actions?: RequestAction[];
    /** Statuses to filter on */
    statuses?: RequestStatus[];
    /** Return test requests */
    isTest?: boolean;
    /** Return silent mode requests */
    isSilent?: boolean;
    /** Filter by whether request is active */
    isClosed?: boolean;
  } = {},
): Promise<TranscendPrivacyRequestFull[]> {
  // read in requests
  const requests: TranscendPrivacyRequestFull[] = [];
  let offset = 0;

  // Paginate
  let shouldContinue = false;
  do {
    const {
      requests: { nodes, totalCount },
    } = await client.request<{
      /** Requests */
      requests: {
        /** List */
        nodes: TranscendPrivacyRequestFull[];
        /** Total count */
        totalCount: number;
      };
    }>(REQUESTS, {
      first: PAGE_SIZE,
      offset,
      filterBy: {
        type: actions.length > 0 ? actions : undefined,
        status: statuses.length > 0 ? statuses : undefined,
        origin: origins.length > 0 ? origins : undefined,
        isTest,
        isSilent,
        isClosed,
      },
    });

    requests.push(...nodes);
    offset += PAGE_SIZE;
    shouldContinue = nodes.length === PAGE_SIZE;
  } while (shouldContinue);

  return allRequests;
}

/**
 * A function that will pull all requests pending approval in Transcend and run
 * a post-processing operation on each request to pull back into your system.
 *
 * You may decide to run a function like this once per day or once per hour.
 */
async function checkTranscendForCompletedRequests(): Promise<void> {
  const transcendUrl = 'https://api.transcend.io'; // EU hosting
  // const transcendUrl = 'https://api.us.transcend.io'; // US hosting

  // Construct GraphQL client
  const client = new GraphQLClient(`${transcendUrl}/graphql`, {
    headers: {
      Authorization: `Bearer ${process.env.TRANSCEND_API_KEY}`,
    },
  });

  // Grab the requests pending approval
  let completedRequests = await fetchAllRequests(client, {
    status: [RequestStatus.Approving, RequestStatus.SecondaryApproving],
  });

  // We want to filter out erasure requests in status=APPROVING
  // as these requests are pending approval before erasure begins
  completedRequests = completedRequests.filter(
    (request) =>
      !(
        request.type === RequestAction.Erasure &&
        request.status === RequestStatus.Approving
      ),
  );

  // Add custom logic to post-process each request
  // See below in this document to see examples of pulling in request files
  // for access requests during this step
  await parseRequestsToMySystem(completedRequests);

  // Approve the requests after post-processing so they are not processed again
  await map(
    completedRequests,
    async (request) => {
      if (request.type === RequestAction.Erasure) {
        await client.request(client, APPROVE_PRIVACY_SECONDARY_REQUEST, {
          input: { requestId: requestToApprove.id },
        });
      } else {
        await client.request(client, APPROVE_PRIVACY_REQUEST, {
          input: { requestId: requestToApprove.id },
        });
      }
    },
    { concurrency: 20 },
  );
}

For a data access request, once the request is in an APPROVING, DOWNLOADABLE or VIEW_CATEGORIES state - it is now time to download the files that were compiled for that request.

There are two steps to this:

List out the files and their associated metadata:

Endpoint: Get the files to download

GET

/v1/data-subject-request/{id}/download-keys

Open in API Reference

Using the downloadKey for each file, stream the file contents to your destination location

Endpoint: Download individual files

GET

/v1/files

Open in API Reference

A fully-functional prototype of a process for downloading the files for data access requests can be found in Transcend CLI's tr-request-download-files command.

The entrypoint for this command lives in the src/cli-request-download-files.ts file.
The meat of the logic lives in src/requests/downloadPrivacyRequestFiles.ts) file.
The logic for querying in all requests in a certain state can be found in src/graphql/fetchAllRequests.ts) file.
The logic for grabbing the metadata associated with the files can be found in src/requests/getFileMetadataForPrivacyRequests.ts) file.
The logic for streaming the file to your location of choice can be found in src/requests/streamPrivacyRequestFiles.ts file.

Here is some example code:

TypeScript

import { existsSync, mkdirSync, writeFileSync } from 'fs';
import { dirname, join } from 'path';
import {
  RequestAction,
  RequestStatus,
  TableEncryptionType,
} from '@transcend-io/privacy-types';
import { decodeCodec, valuesOf } from '@transcend-io/type-utils';
import { map } from 'bluebird';
import colors from 'colors';
import got, { Got } from 'got';
import { GraphQLClient } from 'graphql-request';
import * as t from 'io-ts';

import { logger } from '../logger';

const FOLDER_PATH = './downloaded-privacy-requests';

const IntlMessage = t.type({
  /** The message key */
  defaultMessage: t.string,
  /** ID */
  id: t.string,
});

/** Type */
type IntlMessage = t.TypeOf<typeof IntlMessage>;

const RequestFileMetadata = t.type({
  /** The key to pass to download the file contents */
  downloadKey: t.string,
  /** Error message related to file */
  error: t.union([t.null, t.string]),
  /** Mimetype of file */
  mimetype: t.string,
  /** Size of file, stored as string as this can be a BigInt */
  size: t.string,
  /** Name of file based on datapoint names in Transcend */
  fileName: t.string,
  /** The metadata on the datapoint */
  dataPoint: t.type({
    /** ID of datapoint */
    id: t.string,
    /** The title of datapoint */
    title: t.union([IntlMessage, t.null]),
    /** Description of datapoint */
    description: t.union([IntlMessage, t.null]),
    /** Name of datapoint */
    name: t.string,
    /** Slug of datapoint */
    slug: t.string,
    /** Table level encryption information */
    encryption: t.union([valuesOf(TableEncryptionType), t.null]),
    /** The name of the data silo */
    dataSilo: t.type({
      /** ID of the data silo */
      id: t.string,
      /** The title of the data silo */
      title: t.string,
      /** The description of the data silo */
      description: t.string,
      /** The type of the data silo */
      type: t.string,
      /** The outer type of the data silo */
      outerType: t.union([t.string, t.null]),
    }),
    /** The path to the datapoint if a database (e.g. name of schema) */
    path: t.array(t.string),
  }),
});

/** Type override */
type RequestFileMetadata = t.TypeOf<typeof RequestFileMetadata>;

const RequestFileMetadataResponse = t.type({
  /** The list of file metadata */
  nodes: t.array(RequestFileMetadata),
  /** The total number of file metadata */
  totalCount: t.number,
  /** Links to next pages */
  _links: t.partial({
    /** The link to the next page of file metadata */
    next: t.union([t.string, t.null]),
    /** The link to the previous page of file metadata */
    previous: t.union([t.string, t.null]),
  }),
});

/** Type override */
type RequestFileMetadataResponse = t.TypeOf<typeof RequestFileMetadataResponse>;

/**
 * Given a list of privacy requests, download the file metadata
 * for these requests - this is useful to prepare the files in a
 * data access request for download.
 *
 * @param requests - The list of privacy requests to download files for
 * @param options - Options
 * @returns The number of requests canceled
 */
async function getFileMetadataForPrivacyRequests(
  requests: Pick<TranscendPrivacyRequest, 'id' | 'status'>[],
  {
    sombra,
    limit = 100,
  }: {
    /** Sombra instance */
    sombra: Got;
    /** Number of files to pull at once */
    limit?: number;
  },
): Promise<
  [Pick<TranscendPrivacyRequest, 'id' | 'status'>, RequestFileMetadata[]][]
> {
  // Loop over the requests
  const results = await map(
    requests,
    async (
      requestToDownload,
    ): Promise<
      [Pick<TranscendPrivacyRequest, 'id' | 'status'>, RequestFileMetadata[]]
    > => {
      const localResults: RequestFileMetadata[] = [];

      // Paginate over the file metadata for this request
      let shouldContinue = true;
      let offset = 0;
      while (shouldContinue) {
        let response: RequestFileMetadataResponse;
        try {
          // Grab the file metadata for this request
          // eslint-disable-next-line no-await-in-loop
          const rawResponse = await sombra
            .get(
              `v1/data-subject-request/${requestToDownload.id}/download-keys`,
              {
                searchParams: {
                  limit,
                  offset,
                },
              },
            )
            .json();
          response = decodeCodec(RequestFileMetadataResponse, rawResponse);
          localResults.push(...response.nodes);

          // Increase offset and break if no more pages
          offset += limit;
          shouldContinue =
            // eslint-disable-next-line no-underscore-dangle
            !!response._links.next && response.nodes.length === limit;
        } catch (err) {
          throw new Error(
            `Received an error from server: ${
              err?.response?.body || err?.message
            }`,
          );
        }
      }
      return [requestToDownload, localResults];
    },
    { concurrency: 5 },
  );

  return results;
}

/**
 * This function will take in a set of file metadata for privacy requests
 * call the Transcend API to stream the file metadata for these requests
 * and pass that through a callback function
 *
 * @param fileMetadata - Metadata to download
 * @param options - Options for the request
 */
async function streamPrivacyRequestFiles(
  fileMetadata: RequestFileMetadata[],
  {
    requestId,
    sombra,
    onFileDownloaded,
    concurrency = 20,
  }: {
    /** Request ID for logging */
    requestId: string;
    /** Sombra got instance */
    sombra: Got;
    /** Handler on each file */
    onFileDownloaded: (metadata: RequestFileMetadata, stream: Buffer) => void;
    /** Concurrent downloads at once */
    concurrency?: number;
  },
): Promise<void> {
  // Loop over each file
  await map(
    fileMetadata,
    async (metadata) => {
      try {
        // Construct the stream
        await sombra
          .get('v1/files', {
            searchParams: {
              downloadKey: metadata.downloadKey,
            },
          })
          .buffer()
          .then((fileResponse) => onFileDownloaded(metadata, fileResponse));
      } catch (err) {
        if (err?.response?.body?.includes('fileMetadata#verify')) {
          logger.error(
            colors.red(
              `Failed to pull file for: ${metadata.fileName} (request:${requestId}) - JWT expired. ` +
                'This likely means that the file is no longer available. ' +
                'Try restarting the request from scratch in Transcend Admin Dashboard. ' +
                'Skipping the download of this file.',
            ),
          );
          return;
        }
        throw new Error(
          `Received an error from server: ${
            err?.response?.body || err?.message
          }`,
        );
      }
    },
    {
      concurrency,
    },
  );
}

/**
 * Download a set of data subject ACCESS requests to disk
 */
export async function downloadPrivacyRequestFiles(): Promise<void> {
  // Construct clients
  const transcendUrl = 'https://api.transcend.io'; // EU hosting
  // const transcendUrl = 'https://api.us.transcend.io'; // US hosting
  const client = new GraphQLClient(`${transcendUrl}/graphql`, {
    headers: {
      Authorization: `Bearer ${process.env.TRANSCEND_API_KEY}`,
    },
  });
  const sombra = got.extend({
    // Find your API url under Infrastructure -> Sombra
    // https://app.transcend.io/infrastructure/sombra/sombras
    prefixUrl: 'https://multi-tenant.sombra.transcend.io', // EU hosting
    // prefixUrl: 'https://multi-tenant.us.sombra.transcend.io', // US hosting
    // prefixUrl: 'https://sombra.acme.com',                     // Self hosting
    headers: {
      Authorization: `Bearer ${process.env.TRANSCEND_API_KEY}`,
      ...(process.env.SOMBRA_API_KEY
        ? {
            'X-Sombra-Authorization': `Bearer ${process.env.SOMBRA_API_KEY}`,
          }
        : {}),
    },
  });

  // Create the folder to save files to
  if (!existsSync(FOLDER_PATH)) {
    mkdirSync(FOLDER_PATH);
  }

  // Pull in the access requests requests in a DOWNLOADABLE or APPROVING status
  const allRequests = await fetchAllRequests(client, {
    actions: [RequestAction.Access],
    statuses: [RequestStatus.Downloadable, RequestStatus.Approving],
  });

  // Download the file metadata for each request
  const requestFileMetadata = await getFileMetadataForPrivacyRequests(
    allRequests,
    {
      sombra,
    },
  );

  // Download the files for each request
  await map(
    requestFileMetadata,
    async ([request, metadata]) => {
      // Create a new folder to store request files
      const requestFolder = join(FOLDER_PATH, request.id);
      if (!existsSync(requestFolder)) {
        mkdirSync(requestFolder);
      }

      // Stream each file to disk
      await streamPrivacyRequestFiles(metadata, {
        sombra,
        requestId: request.id,
        onFileDownloaded: (fil, stream) => {
          // Ensure a folder exists for the file
          // filename looks like Health/heartbeat.csv
          const filePath = join(requestFolder, fil.fileName);
          const folder = dirname(filePath);
          if (!existsSync(folder)) {
            mkdirSync(folder, { recursive: true });
          }

          // Write to disk
          writeFileSync(filePath, stream);
        },
      });
    },
    { concurrency: 20 },
  );
}

Using the API for DSR Automation

Self-hosted Sombra configuration

Submit a DSR

Endpoint: Submit a DSR

Silent Mode

Multiple identifiers

Custom Fields

Batching Uploads

Example API Call

Poll DSR state

Endpoint: Poll DSR state

Detecting Completion of Request

type=ACCESS

type=ERASURE

Other types

Example Code

Using request status=APPROVING as a queue

Example Code

Download the files for an ACCESS request

Endpoint: Get the files to download

Endpoint: Download individual files

Example Code