API to Read from Data Inventory
This guide runs through the API calls and developer tooling that can be used to pull data from your Data Inventory.
Some useful resources to review before this guide:
- Check out this article to understand some potential motivations for using this API.
- Check out this article to understand the tabs and concepts in the Data Inventory like "Data Silos", "Vendors", "Data Categories", "Processing Purposes, "Identifiers", "Data Subjects" and more.
- Check out this tab to learn more about Developer Tools.
In general, there are 3 strategies for reading data from the Data Inventory:
- Transcend CLI - This command line interface allows for you to generate an API key and export data into a YAML file. This avoids the need for you to implement pagination or understand the Transcend API shapes to pull data. This is generally best for doing full exports.
- Transcend GraphQL API - Everything that you can do in the Transcend Admin Dashboard can also be done using this API. This allows for building highly specific queries and deeper integration into internal tools and dashboards.
- Fivetran ETL - FiveTran has an ETL integration to export data from Transcend into your Data Warehouse using an API key.
Using the Transcend CLI is a great way to export your data from Transcend Data Inventory without needing to write code.
To start:
- Install the CLI using npm or yarn.
- Generate an API key in the Transcend Dashboard. You will need to assing the API key scopes. These scopes will vary depending on which resource types you intend to pull.
- Run the CLI command in your terminal to export your Data Inventory to a yml file.
You can see all examples of usage here as well as some example transcend.yml files here.
Some common example:
Shell
tr-pull --auth=$TRANSCEND_API_KEY
Shell
tr-pull --auth=$TRANSCEND_API_KEY --file=./custom/location.yml
Shell
tr-pull --auth=$TRANSCEND_API_KEY --resources=apiKeys,templates,dataSilos,enrichers
See the full list of resource types here.
Shell
tr-pull --auth=$TRANSCEND_API_KEY --integrationNames=salesforce,snowflake
You can run the following query in the GraphQL Playground to get the valid integrationNames.
GraphQL
query { catalogs(first: 100, filterBy: {}) { nodes { integrationName title website } } }
Shell
tr-pull --auth=$TRANSCEND_API_KEY ---dataSiloIds=710fec3c-7bcc-4c9e-baff-bf39f9bec43e
You can run the following query in the GraphQL Playground to get the valid dataSiloIds.
GraphQL
query { dataSilos(first: 100) { nodes { id title } } }
The Transcend GraphQL API is the most flexible way to pull data from Transcend. This is the same API that Transcend's Admin Dashboard uses, so any query or filter you can make in the Transcend Dashboard is also available in this API.
To start:
- Generate an API key in the Transcend Dashboard. You will need to assing the API key scopes. These scopes will vary depending on which resource types you intend to pull.
- Run the CLI command in your terminal to export your Data Inventory to a yml file.
To start, go into the Transcend Admin Dashboard to create an API key.
The scopes that you will grant to the API key will depend on the routes you plan to use. You can visualize the relationship between Scopes and GraphQL Routes on the Scopes tab of the Admin Dashboard.
GraphQL APIs have a lot of great open source development tooling built around them. You can call the GraphQL API with any standard request tooling, however you find it nice to use a GraphQL client package within the language of your choice. For example see graphql-request for JavaScript.
You would instantiate your GraphQL client providing the API key as a bearer token.
JavaScript
import { GraphQLClient } from 'graphql-request'; const transcendUrl = 'https://api.transcend.io'; // for EU hosting // const transcendUrl = 'https://api.us.transcend.io'; // for US hosting const client = new GraphQLClient(`${transcendUrl}/graphql`, { headers: { Authorization: `Bearer ${process.env.TRANSCEND_API_KEY}`, }, });
Once you've created your GraphQL client, the next step is to create the GraphQL that pulls the data that you care to export. The best way to do this is using the GraphQL playground to construct and test your payload. You will first what to make sure that you are logged in to the Transcend Admin Dashboard and if you have multiple Trancsend accounts, ensure that you have switched into the correct account.
You can then navigate to either the Default EU Hosted GraphQL Playground or the US Hosted GraphQL Playground.
On the right hand side of the screen, you can open up the "Docs" tab to view all of the GraphQL routes avilable, and all of the input/output document types.
You should use the GraphQL playground to get the most up to date set of parameters for each Data Inventory routes. GraphQL gives you the ability to specify exactly which fields you want to pull. By specifying the minimal set of parameters in the API, you will achieve faster request times.
Some good starting points for GraphQL queries include:
GraphQL
query DataInventoryGraphQLExampleDataSilos { dataSilos( first: 5 filterBy: { hasPersonalData: YES } orderBy: [{ direction: ASC, field: title }] ) { nodes { id title type outerType url isLive processingPurposeSubCategories { name purpose } description dataSubCategories { name category } attributeValues { name attributeKey { name } } } } }
This may produce an output like
JSON
{
"data": {
"dataSilos": {
"nodes": [
{
"id": "26b1af13-c88c-4bec-9d99-e5490fb8fb06",
"title": "Ada",
"type": "promptAPerson",
"outerType": "ada",
"url": null,
"isLive": false,
"processingPurposeSubCategories": [
{
"name": "",
"purpose": "ANALYTICS"
},
{
"name": "",
"purpose": "ADVERTISING"
},
{
"name": "",
"purpose": "MARKETING"
}
],
"description": "An AI-powered customer service platform that uses an AI agent to answer customers on voice and email. ",
"dataSubCategories": [
{
"name": "Average BPM",
"category": "HEALTH"
}
],
"attributeValues": [
{
"name": "Text box",
"attributeKey": {
"name": "New Column 123"
}
},
{
"name": "GDPR",
"attributeKey": {
"name": "Related LAws"
}
},
{
"name": "CPRA",
"attributeKey": {
"name": "Related LAws"
}
},
{
"name": "5 Year Deletion",
"attributeKey": {
"name": "Retention Period"
}
},
{
"name": "Dept A",
"attributeKey": {
"name": "Dept"
}
},
{
"name": "Application A",
"attributeKey": {
"name": "Mobile App"
}
},
{
"name": "NEEDED",
"attributeKey": {
"name": "DPIA Status"
}
},
{
"name": "C",
"attributeKey": {
"name": "Test Attribute "
}
},
{
"name": "Yes ",
"attributeKey": {
"name": "B2B "
}
},
{
"name": "NEEDED",
"attributeKey": {
"name": "Vendor Risk Assessment Status"
}
},
{
"name": "Contractual Necessity",
"attributeKey": {
"name": "Lawful Basis of Processing"
}
},
{
"name": "Site 1",
"attributeKey": {
"name": "Site"
}
},
{
"name": "2 years",
"attributeKey": {
"name": "Data Retention Period"
}
},
{
"name": "Low",
"attributeKey": {
"name": "Risk Level"
}
},
{
"name": "NOT_NEEDED",
"attributeKey": {
"name": "DPIA Status Tracker"
}
},
{
"name": "Medium",
"attributeKey": {
"name": "Risk Level 2"
}
},
{
"name": "Social Media Integrations",
"attributeKey": {
"name": "Source of Data"
}
},
{
"name": "Yes",
"attributeKey": {
"name": "Fully Confirmed"
}
},
{
"name": "eShopIt AI Risk Assessment",
"attributeKey": {
"name": "AI Impact Assessment Status"
}
}
]
},
{
"id": "2b9396b0-7061-4aed-9c70-44fd8cb50d2f",
"title": "Amazon.com",
"type": "promptAPerson",
"outerType": "amazon",
"url": null,
"isLive": true,
"processingPurposeSubCategories": [],
"description": "Amazon Business combines the selection, convenience, and value of Amazon with features that help improve business purchasing.",
"dataSubCategories": [],
"attributeValues": [
{
"name": "Indefinite",
"attributeKey": {
"name": "Retention Period"
}
},
{
"name": "Dept B",
"attributeKey": {
"name": "Dept"
}
},
{
"name": "Medium",
"attributeKey": {
"name": "Risk Level"
}
},
{
"name": "5 years",
"attributeKey": {
"name": "Data Retention Period"
}
},
{
"name": "Consent",
"attributeKey": {
"name": "Lawful Basis of Processing"
}
},
{
"name": "D",
"attributeKey": {
"name": "Test Attribute "
}
},
{
"name": "NOT_NEEDED",
"attributeKey": {
"name": "AI Impact Assessment Status"
}
}
]
},
{
"id": "ce4adb09-e746-41f7-a17f-b9f7ce7eac10",
"title": "Amplitude",
"type": "amplitude",
"outerType": null,
"url": null,
"isLive": false,
"processingPurposeSubCategories": [],
"description": "Amplitude is a behavioral analytics platform for web & mobile.",
"dataSubCategories": [],
"attributeValues": []
},
{
"id": "5ddf8f29-d2a4-4310-89a7-5699b115ee95",
"title": "App Backend",
"type": "server",
"outerType": null,
"url": "https://transcend-example.onrender.com/transcend/new-dsr",
"isLive": false,
"processingPurposeSubCategories": [],
"description": "Send a webhook to a server and POST back through our API.",
"dataSubCategories": [],
"attributeValues": [
{
"name": "Indefinitely",
"attributeKey": {
"name": "Data Retention Period"
}
},
{
"name": "Contractual Necessity",
"attributeKey": {
"name": "Lawful Basis of Processing"
}
}
]
},
{
"id": "6720ae03-7d67-41b7-b829-3bfb63f8e9fc",
"title": "Apptimize",
"type": "promptAPerson",
"outerType": "apptimize",
"url": null,
"isLive": false,
"processingPurposeSubCategories": [],
"description": "Apptimize, an innovation engine providing industry-leading experimentation and feature release management, supports the creation of exceptional user experiences across all digital platforms: mobile, hybrid, OTT, web, and server",
"dataSubCategories": [],
"attributeValues": []
}
]
}
}
}
GraphQL
query DataInventoryGraphQLExampleVendors { vendors(first: 5, orderBy: [{ direction: ASC, field: title }]) { nodes { id title websiteUrl headquarterCountry processingPurposeSubCategories { name purpose } description attributeValues { name attributeKey { name } } } } }
This may produce an output like
JSON
{
"data": {
"vendors": {
"nodes": [
{
"id": "fbe81d3d-e2f5-4d28-bf4b-11884328c724",
"title": "15Five, Inc.",
"websiteUrl": null,
"headquarterCountry": null,
"processingPurposeSubCategories": [],
"description": "15Five is the performance management platform that drives business results.",
"attributeValues": []
},
{
"id": "2873a5d6-cd7b-4c67-9c5e-3c413fa8af8e",
"title": "Accel-KKR",
"websiteUrl": null,
"headquarterCountry": null,
"processingPurposeSubCategories": [
{
"name": "",
"purpose": "ANALYTICS"
},
{
"name": "",
"purpose": "MARKETING"
},
{
"name": "",
"purpose": "ADVERTISING"
},
{
"name": "",
"purpose": "ESSENTIAL"
},
{
"name": "",
"purpose": "ADDITIONAL_FUNCTIONALITY"
}
],
"description": "Accel-KKR is a technology-focused private equity firm specialized in growth and middle market investments.",
"attributeValues": [
{
"name": "High",
"attributeKey": {
"name": "Risk Score"
}
}
]
},
{
"id": "5d54e491-8bbf-481b-9a4e-a9463252131d",
"title": "Accountalent",
"websiteUrl": null,
"headquarterCountry": null,
"processingPurposeSubCategories": [],
"description": "A description",
"attributeValues": []
},
{
"id": "6a260e94-3793-40fc-ad20-aae754739394",
"title": "Ada Support, Inc.",
"websiteUrl": null,
"headquarterCountry": "EU",
"processingPurposeSubCategories": [],
"description": "Ada system",
"attributeValues": [
{
"name": "Medium",
"attributeKey": {
"name": "Risk Score"
}
}
]
},
{
"id": "1f6f06fb-0df9-4728-bd71-0af45229f6a9",
"title": "Adobe Inc.",
"websiteUrl": null,
"headquarterCountry": null,
"processingPurposeSubCategories": [
{
"name": "",
"purpose": "MARKETING"
}
],
"description": "A Vendor",
"attributeValues": [
{
"name": "Medium",
"attributeKey": {
"name": "Risk Score"
}
}
]
}
]
}
}
}
GraphQL
query DataInventoryGraphQLExampleDataCategories { dataSubCategories(first: 5) { nodes { name category description dataSilos { id title } attributeValues { name attributeKey { name } } } } }
This may produce an output like:
JSON
{
"data": {
"dataSubCategories": {
"nodes": [
{
"name": "",
"category": "FINANCIAL",
"description": "Generic financial information.",
"dataSilos": [
{
"id": "965b0eb2-b87d-464c-a849-571e31384001",
"title": "AWS Redshift"
},
{
"id": "1c246182-314c-459a-896c-4c61e4ecf4bd",
"title": "Amazon S3 - 207539467107"
},
{
"id": "ce4adb09-e746-41f7-a17f-b9f7ce7eac10",
"title": "Amplitude"
},
{
"id": "11d6e82d-99c2-466b-9b32-5febec81b433",
"title": "Asana"
},
{
"id": "e455008a-d73d-4a23-9ee4-c536bf176e37",
"title": "Braintree"
},
{
"id": "02ecfa88-8842-467e-9679-1191c847c604",
"title": "Braze"
},
{
"id": "73863c19-7102-4b95-8f69-6d20ce3fcfcd",
"title": "Braze - created at 2024-08-22 21:06:52"
},
{
"id": "4fa3dce5-a32b-4d9f-974d-a4984f3445d8",
"title": "Chargebee"
},
{
"id": "2842fcb9-393f-4a89-b42c-830f419b248a",
"title": "Concur"
},
{
"id": "6831ee85-4749-4c6a-ba22-6c12adb8ffa2",
"title": "Expensify"
},
{
"id": "0bd04292-cdb9-424c-bb5f-c4bafb5c30ab",
"title": "Forter"
},
{
"id": "e823f6b5-12cc-4d83-a5e4-e1ad30a18a72",
"title": "Google BigQuery"
},
{
"id": "fe3a9e00-1337-4f75-b123-1697276811f2",
"title": "Google BigQuery - Ladan"
},
{
"id": "4b5b482e-b253-4bd2-ae8c-1b4ddb57de4a",
"title": "MongoDB"
},
{
"id": "fb7a8161-6c4f-4352-a35e-bd554569fef9",
"title": "New Amplitude"
},
{
"id": "a17c7402-bcbe-4fed-a8ac-d02373a1aa70",
"title": "PostgreSQL "
},
{
"id": "9a39e7dc-a596-4854-86c4-60e5cbddc3ee",
"title": "ProfitWell"
},
{
"id": "747ff22c-483a-4ae7-a9c8-fa090443af06",
"title": "Quickbooks"
},
{
"id": "a7a2c53b-936f-40cb-b740-2da0754c1f0b",
"title": "Recurly"
},
{
"id": "13abbe03-8c7c-4db5-97d4-85e6d2ce1c7c",
"title": "Shopify"
},
{
"id": "a9b8b8c8-c7ff-4104-8953-98b1381c771e",
"title": "Stripe"
}
],
"attributeValues": [
{
"name": "Delete",
"attributeKey": {
"name": "Masking Policy"
}
},
{
"name": "High",
"attributeKey": {
"name": "Risk Level"
}
},
{
"name": "Yes",
"attributeKey": {
"name": "Requires DPIA"
}
},
{
"name": "Payment Card Data (PCI Data)",
"attributeKey": {
"name": "Types of Personal Identifiers"
}
},
{
"name": "Sensitive Data (CCPA)",
"attributeKey": {
"name": "Super Category"
}
}
]
},
{
"name": "Account Number",
"category": "FINANCIAL",
"description": "An account number relating to an individual's finances",
"dataSilos": [
{
"id": "11d6e82d-99c2-466b-9b32-5febec81b433",
"title": "Asana"
},
{
"id": "4b5b482e-b253-4bd2-ae8c-1b4ddb57de4a",
"title": "MongoDB"
}
],
"attributeValues": [
{
"name": "Somewhat Sensitive",
"attributeKey": {
"name": "Sensitivity Level"
}
},
{
"name": "Sensitive Data (CCPA)",
"attributeKey": {
"name": "Super Category"
}
},
{
"name": "Payment Card Data (PCI Data)",
"attributeKey": {
"name": "Types of Personal Identifiers"
}
}
]
},
{
"name": "Credit Card Number",
"category": "FINANCIAL",
"description": "Credit card number",
"dataSilos": [
{
"id": "1c246182-314c-459a-896c-4c61e4ecf4bd",
"title": "Amazon S3 - 207539467107"
},
{
"id": "e823f6b5-12cc-4d83-a5e4-e1ad30a18a72",
"title": "Google BigQuery"
},
{
"id": "4b5b482e-b253-4bd2-ae8c-1b4ddb57de4a",
"title": "MongoDB"
},
{
"id": "a17c7402-bcbe-4fed-a8ac-d02373a1aa70",
"title": "PostgreSQL "
},
{
"id": "a9b8b8c8-c7ff-4104-8953-98b1381c771e",
"title": "Stripe"
}
],
"attributeValues": [
{
"name": "Redact",
"attributeKey": {
"name": "Masking Policy"
}
},
{
"name": "Highly Sensitive",
"attributeKey": {
"name": "Sensitivity Level"
}
},
{
"name": "Sensitive Data (CCPA)",
"attributeKey": {
"name": "Super Category"
}
},
{
"name": "Payment Card Data (PCI Data)",
"attributeKey": {
"name": "Types of Personal Identifiers"
}
}
]
},
{
"name": "Income",
"category": "FINANCIAL",
"description": "Information about an individual's income",
"dataSilos": [
{
"id": "965b0eb2-b87d-464c-a849-571e31384001",
"title": "AWS Redshift"
},
{
"id": "1c246182-314c-459a-896c-4c61e4ecf4bd",
"title": "Amazon S3 - 207539467107"
},
{
"id": "4b5b482e-b253-4bd2-ae8c-1b4ddb57de4a",
"title": "MongoDB"
},
{
"id": "a17c7402-bcbe-4fed-a8ac-d02373a1aa70",
"title": "PostgreSQL "
}
],
"attributeValues": [
{
"name": "Highly Sensitive",
"attributeKey": {
"name": "Sensitivity Level"
}
},
{
"name": "Payment Card Data (PCI Data)",
"attributeKey": {
"name": "Types of Personal Identifiers"
}
},
{
"name": "Sensitive Data (CCPA)",
"attributeKey": {
"name": "Super Category"
}
}
]
},
{
"name": "Routing Number",
"category": "FINANCIAL",
"description": "A bank's routing number",
"dataSilos": [
{
"id": "965b0eb2-b87d-464c-a849-571e31384001",
"title": "AWS Redshift"
}
],
"attributeValues": [
{
"name": "Highly Sensitive",
"attributeKey": {
"name": "Sensitivity Level"
}
},
{
"name": "Sensitive Data (CCPA)",
"attributeKey": {
"name": "Super Category"
}
},
{
"name": "Payment Card Data (PCI Data)",
"attributeKey": {
"name": "Types of Personal Identifiers"
}
}
]
}
]
}
}
}
GraphQL
query DataInventoryGraphQLExampleProcessingPurposes { processingPurposeSubCategories(first: 5) { nodes { name purpose description dataSilos { id title } attributeValues { name attributeKey { name } } } } }
This may produce an output like:
JSON
{
"data": {
"processingPurposeSubCategories": {
"nodes": [
{
"name": "",
"purpose": "ESSENTIAL",
"description": "Provide a service that the user explicitly requests and that is part of the product's basic service or functionality",
"dataSilos": [
{
"id": "26b1af13-c88c-4bec-9d99-e5490fb8fb06",
"title": "Ada"
},
{
"id": "054a695d-6b69-4f1b-b3fb-2749095ded6f",
"title": "Aircall"
},
{
"id": "1c246182-314c-459a-896c-4c61e4ecf4bd",
"title": "Amazon S3 - 207539467107"
},
{
"id": "2b9396b0-7061-4aed-9c70-44fd8cb50d2f",
"title": "Amazon.com"
},
{
"id": "5ddf8f29-d2a4-4310-89a7-5699b115ee95",
"title": "App Backend"
},
{
"id": "2098b1c3-7588-4d26-b75b-7c292c5b26d4",
"title": "Atlassian"
},
{
"id": "e455008a-d73d-4a23-9ee4-c536bf176e37",
"title": "Braintree"
},
{
"id": "4bc1e9de-a113-4e0e-b583-2601b1a33009",
"title": "Branch.io"
},
{
"id": "262bb590-c831-4cb2-b5f0-0a4cb16f3983",
"title": "Branch.io - created at 2023-10-19 18:45:03"
},
{
"id": "4fa3dce5-a32b-4d9f-974d-a4984f3445d8",
"title": "Chargebee"
},
{
"id": "2842fcb9-393f-4a89-b42c-830f419b248a",
"title": "Concur"
},
{
"id": "7f65ef88-57ba-4be1-978b-a3856fef56ec",
"title": "Confluence"
},
{
"id": "4882df07-b2f6-47a3-9adb-36831029d493",
"title": "Database - example"
},
{
"id": "0e50696e-dce1-483c-8964-6288afca80a3",
"title": "DocuSign 2"
},
{
"id": "9e8f49e1-3114-4ae1-ad09-802bc962dde9",
"title": "DocuSign Apps Launcher"
},
{
"id": "f879feee-7317-4ae0-bca0-87cefe70752d",
"title": "Dropbox Business"
},
{
"id": "6831ee85-4749-4c6a-ba22-6c12adb8ffa2",
"title": "Expensify"
},
{
"id": "0bd04292-cdb9-424c-bb5f-c4bafb5c30ab",
"title": "Forter"
},
{
"id": "2e6ad582-38a6-40c7-b8e6-57a84291ba8f",
"title": "Gong"
},
{
"id": "e823f6b5-12cc-4d83-a5e4-e1ad30a18a72",
"title": "Google BigQuery"
},
{
"id": "15ee9c58-7909-444a-9311-f92c922a007d",
"title": "Mailchimp"
},
{
"id": "9fd51671-df3e-4945-a0e9-010c0f8d804f",
"title": "Microsoft Office 365"
},
{
"id": "f72c28d2-ac2a-40cc-8001-e406c84701cd",
"title": "Microsoft Office 365 - Ashley"
},
{
"id": "4200ffa7-8ff3-4b96-bd61-eb0044db1cc0",
"title": "Microsoft Office 365 - Mike F - 6/13"
},
{
"id": "31e724f9-654f-4c72-89f2-bf7a689217b6",
"title": "Microsoft Office 365 - Transcend Demo"
},
{
"id": "c7ceff69-6119-46d7-b99d-dcd0fe734816",
"title": "Microsoft Office 365 - created at 2023-08-22 18:27:42"
},
{
"id": "6aaa4408-5937-425a-ae51-7622349b519f",
"title": "Microsoft Office 365 - created at 2024-10-25 13:16:24"
},
{
"id": "1c6123ac-1df0-49e9-a5ea-92da306fc828",
"title": "Microsoft Office 365 - created at 2024-10-28 13:48:54"
},
{
"id": "b6f22295-05c0-4bf1-b801-9c4b34bf99b6",
"title": "Microsoft SharePoint - Unstructured Discovery"
},
{
"id": "d9e54c27-17eb-41a3-91ed-c400d5c09c7a",
"title": "Microsoft SharePoint - created at 2023-08-22 18:24:56"
},
{
"id": "4683a9a0-da4e-4e4f-8316-35b17513d4be",
"title": "Microsoft SharePoint - mike test 8/23"
},
{
"id": "baa562ff-ff96-43fb-85eb-90cce67b8d67",
"title": "Microsoft Sharepoint"
},
{
"id": "26c8e140-dacc-48d6-8659-51ba77860be2",
"title": "Movable Ink"
},
{
"id": "5b3e4af5-0940-41f0-b225-c4eb1ea71750",
"title": "Netsuite"
},
{
"id": "99816114-fc98-442f-9cd0-2723bcc61c8a",
"title": "Okta (eBrands)"
},
{
"id": "1b36cbac-2e59-44c2-8f4f-c387926e371a",
"title": "Okta (eShopIt)"
},
{
"id": "9df95fd7-4d91-44c8-b762-f00f196565a8",
"title": "Okta - Test"
},
{
"id": "db6aacc6-4284-48f3-91be-c6448fa2d04d",
"title": "Okta - Test 10/10"
},
{
"id": "18551ac5-81e8-4e57-a191-ea7bf6274dae",
"title": "Okta - created at 2023-07-27 17:38:09"
},
{
"id": "5e76f564-ba3e-4f2f-9f51-e9bfb28eec86",
"title": "Okta - created at 2023-07-27 17:38:10"
},
{
"id": "81ac64bf-a912-4867-ab55-2e127eabb409",
"title": "Okta - created at 2023-11-01 15:16:00"
},
{
"id": "85b1c327-d77b-4809-b588-3e2dc752bae5",
"title": "Okta - created at 2023-11-16 17:27:30"
},
{
"id": "ffa28cf7-7fc3-475d-9f8f-662e6a219a3a",
"title": "Okta - created at 2024-02-21 16:10:18"
},
{
"id": "456e30ae-8078-4437-b5c3-2a4521fe0237",
"title": "Okta - created at 2024-03-18 18:49:35"
},
{
"id": "8e4848bc-9176-4104-a3ad-a1949384faed",
"title": "Okta - created at 2024-06-12 20:47:24"
},
{
"id": "6d2a123b-b8d6-4257-980f-8f21092cada5",
"title": "Okta - created at 2024-07-30 20:31:19"
},
{
"id": "1fc54a8f-4ba5-433f-85ff-a19757b18db4",
"title": "Okta - created at 2024-07-31 12:10:04"
},
{
"id": "bc7e2e40-d4ce-4500-a8e0-e8b7320fcbac",
"title": "Okta OIN Submission Tester"
},
{
"id": "89ef42a2-2778-44a6-a003-79600e5ec45e",
"title": "Okta Test"
},
{
"id": "9e96bc61-ac18-4100-85b1-6ca916d9308f",
"title": "Onfido"
},
{
"id": "4e8d8e4d-b11a-4de9-9838-4a16ae5936fd",
"title": "OwnBackup"
},
{
"id": "c6bd59e2-7ae0-4667-bed2-9163dfa1a3f8",
"title": "Pendo"
},
{
"id": "a17c7402-bcbe-4fed-a8ac-d02373a1aa70",
"title": "PostgreSQL "
},
{
"id": "747ff22c-483a-4ae7-a9c8-fa090443af06",
"title": "Quickbooks"
},
{
"id": "a7a2c53b-936f-40cb-b740-2da0754c1f0b",
"title": "Recurly"
},
{
"id": "b5960c48-a9ec-44bc-8e46-bf0b4069d957",
"title": "ServiceNow"
},
{
"id": "d789688d-95b0-4a04-b130-be3170692d44",
"title": "ServiceNow - Test"
},
{
"id": "bd032ce8-27d2-4f2b-bc31-1902dcbdcfb6",
"title": "ServiceNow - created at 2024-01-05 22:46:36"
},
{
"id": "08ea8c5b-de13-49a0-ab31-2d2466289879",
"title": "ServiceNow - created at 2024-02-09 18:52:21"
},
{
"id": "5ff42794-7d97-480e-b9d7-9c4cdb0ae196",
"title": "ServiceNow - created at 2024-02-29 17:37:03"
},
{
"id": "13abbe03-8c7c-4db5-97d4-85e6d2ce1c7c",
"title": "Shopify"
},
{
"id": "a9b8b8c8-c7ff-4104-8953-98b1381c771e",
"title": "Stripe"
},
{
"id": "54fe5122-ddaf-41c4-b393-baae6c9c07ec",
"title": "Twilio"
},
{
"id": "4a0d40c6-6c32-4896-a012-41c982182f6e",
"title": "Twilio - created at 2023-10-06 15:50:32"
},
{
"id": "0944867c-b41d-43b0-b613-25a90a80328f",
"title": "Twilio - created at 2023-10-06 15:50:50"
},
{
"id": "3216342c-5130-4620-8b2f-87aa3dda19dc",
"title": "Zendesk"
},
{
"id": "974b42f8-3d52-44d5-b050-ffae269e8714",
"title": "Zendesk - created at 2024-08-15 18:15:37"
},
{
"id": "7352f1cb-1847-4797-8870-632b4a4a6c6f",
"title": "Zendesk UK"
},
{
"id": "60e0f38b-00c1-40fc-9ba0-74450e414e5e",
"title": "mParticle"
}
],
"attributeValues": [
{
"name": "Indefinite",
"attributeKey": {
"name": "Retention Period"
}
},
{
"name": "Contractual Necessity",
"attributeKey": {
"name": "Lawful Basis of Processing"
}
},
{
"name": "NEEDED",
"attributeKey": {
"name": "Processing Purpose Assessment"
}
},
{
"name": "Payroll",
"attributeKey": {
"name": "Product Name"
}
}
]
},
{
"name": "Allowing you to log into your account",
"purpose": "ESSENTIAL",
"description": "",
"dataSilos": [],
"attributeValues": [
{
"name": "New PIA 1/24/24",
"attributeKey": {
"name": "Processing Purpose Assessment"
}
}
]
},
{
"name": "Basic data processing",
"purpose": "ESSENTIAL",
"description": "",
"dataSilos": [],
"attributeValues": []
},
{
"name": "Payment",
"purpose": "ESSENTIAL",
"description": "Process for payment",
"dataSilos": [
{
"id": "a9b8b8c8-c7ff-4104-8953-98b1381c771e",
"title": "Stripe"
}
],
"attributeValues": []
},
{
"name": "",
"purpose": "ADDITIONAL_FUNCTIONALITY",
"description": "Provide a service that the user explicitly requests but that is not a necessary part of the product's basic service",
"dataSilos": [
{
"id": "5ddf8f29-d2a4-4310-89a7-5699b115ee95",
"title": "App Backend"
},
{
"id": "11d6e82d-99c2-466b-9b32-5febec81b433",
"title": "Asana"
},
{
"id": "02ecfa88-8842-467e-9679-1191c847c604",
"title": "Braze"
},
{
"id": "73863c19-7102-4b95-8f69-6d20ce3fcfcd",
"title": "Braze - created at 2024-08-22 21:06:52"
},
{
"id": "0e50696e-dce1-483c-8964-6288afca80a3",
"title": "DocuSign 2"
},
{
"id": "9e8f49e1-3114-4ae1-ad09-802bc962dde9",
"title": "DocuSign Apps Launcher"
},
{
"id": "f879feee-7317-4ae0-bca0-87cefe70752d",
"title": "Dropbox Business"
},
{
"id": "d22e5a3f-8858-41f4-8cc8-a383a2ceda18",
"title": "Gmail"
},
{
"id": "96b824b5-6180-44d8-bff4-adf118cc4097",
"title": "HubSpot"
},
{
"id": "8c0d5a62-3b03-4cfe-83dc-7c5c3194a76a",
"title": "HubSpot - created at 2024-01-12 17:00:53"
},
{
"id": "b8bc4491-1394-43e5-ad38-fa52bafea5a8",
"title": "Intercom"
},
{
"id": "f89264f7-d352-46cb-97d0-3676675a26eb",
"title": "Intercom - created at 2023-12-06 23:11:27"
},
{
"id": "a7a2c53b-936f-40cb-b740-2da0754c1f0b",
"title": "Recurly"
},
{
"id": "4cddeca6-6f19-47f5-a66a-51976d57db0c",
"title": "Simon Data"
},
{
"id": "d8eef949-a7dc-4bb9-a3a9-561f89f7bb40",
"title": "SurveyMonkey"
},
{
"id": "3216342c-5130-4620-8b2f-87aa3dda19dc",
"title": "Zendesk"
},
{
"id": "974b42f8-3d52-44d5-b050-ffae269e8714",
"title": "Zendesk - created at 2024-08-15 18:15:37"
},
{
"id": "7352f1cb-1847-4797-8870-632b4a4a6c6f",
"title": "Zendesk UK"
},
{
"id": "784fc77c-781a-495a-9b40-566f159b846b",
"title": "Zoom"
},
{
"id": "59cbe1e5-810d-496a-b514-74e94b9fa8de",
"title": "Zoom - Sandbox"
}
],
"attributeValues": [
{
"name": "Consent",
"attributeKey": {
"name": "Lawful Basis of Processing"
}
}
]
}
]
}
}
}
GraphQL
query DataInventoryGraphQLExampleBusinessEntities { businessEntities(first: 5) { nodes { title description dataSilos { id title } attributeValues { name attributeKey { name } } } } }
This may produce an output like:
JSON
{
"data": {
"businessEntities": {
"nodes": [
{
"title": "eShopIt",
"description": "Primary eCommerce brand",
"dataSilos": [
{
"id": "965b0eb2-b87d-464c-a849-571e31384001",
"title": "AWS Redshift"
},
{
"id": "0f0ca58e-9f00-4e7c-be5b-570c767875d1",
"title": "Amazon Web Services (AWS)"
},
{
"id": "ce4adb09-e746-41f7-a17f-b9f7ce7eac10",
"title": "Amplitude"
},
{
"id": "5ddf8f29-d2a4-4310-89a7-5699b115ee95",
"title": "App Backend"
},
{
"id": "2098b1c3-7588-4d26-b75b-7c292c5b26d4",
"title": "Atlassian"
},
{
"id": "a3589f44-3f36-4abb-8df5-39cb406f5d30",
"title": "Azure Storage Account - created at 2024-01-11 19:21:42"
}
],
"attributeValues": [
{
"name": "Controller",
"attributeKey": {
"name": "Processor/Controller Status"
}
}
]
},
{
"title": "Payroll ",
"description": "Parent company with a lot to acquire",
"dataSilos": [
{
"id": "26b1af13-c88c-4bec-9d99-e5490fb8fb06",
"title": "Ada"
},
{
"id": "1c246182-314c-459a-896c-4c61e4ecf4bd",
"title": "Amazon S3 - 207539467107"
},
{
"id": "7003cb39-5b68-42ee-a3d8-a1df97827c7f",
"title": "Amazon S3 - Unstructured Discovery"
},
{
"id": "0f0ca58e-9f00-4e7c-be5b-570c767875d1",
"title": "Amazon Web Services (AWS)"
},
{
"id": "2b9396b0-7061-4aed-9c70-44fd8cb50d2f",
"title": "Amazon.com"
},
{
"id": "ce4adb09-e746-41f7-a17f-b9f7ce7eac10",
"title": "Amplitude"
},
{
"id": "eee74db1-b7a9-4f8e-8db8-d6e3b5d55d45",
"title": "Attentive"
},
{
"id": "da12a96a-33fb-481e-9515-7491b931b4e1",
"title": "Avalara"
},
{
"id": "365e86bb-cb88-4079-a1ff-17179f63d481",
"title": "Azure Storage Account - cloud2 - d4029453-639b-46d3-a3bd-24d5f0068279"
}
],
"attributeValues": [
{
"name": "Processor",
"attributeKey": {
"name": "Processor/Controller Status"
}
}
]
},
{
"title": "Sneaker Supply - Germany",
"description": "Recently acquired company",
"dataSilos": [
{
"id": "054a695d-6b69-4f1b-b3fb-2749095ded6f",
"title": "Aircall"
},
{
"id": "0f0ca58e-9f00-4e7c-be5b-570c767875d1",
"title": "Amazon Web Services (AWS)"
},
{
"id": "5ddf8f29-d2a4-4310-89a7-5699b115ee95",
"title": "App Backend"
},
{
"id": "11d6e82d-99c2-466b-9b32-5febec81b433",
"title": "Asana"
},
{
"id": "5baf2e88-8945-45ea-8be7-845a1d7808fb",
"title": "Auth0"
}
],
"attributeValues": [
{
"name": "Controller",
"attributeKey": {
"name": "Processor/Controller Status"
}
}
]
}
]
}
}
}
GraphQL
query DataInventoryGraphQLExampleIdentifiers { identifiers(first: 5) { nodes { type name attributeValues { name attributeKey { name } } } } }
This may return a result like:
JSON
{
"data": {
"identifiers": {
"nodes": [
{
"type": "custom",
"name": "user_id",
"attributeValues": []
},
{
"type": "custom",
"name": "transaction_status",
"attributeValues": []
},
{
"type": "phone",
"name": "phone",
"attributeValues": []
},
{
"type": "onfidoApplicantId",
"name": "onfidoApplicantId",
"attributeValues": []
},
{
"type": "custom",
"name": "name",
"attributeValues": []
}
]
}
}
}
GraphQL
query DataInventoryGraphQLExampleDataSubjects { internalSubjects { title { defaultMessage } subjectClass attributeValues { name attributeKey { name } } } }
This my return a result like:
JSON
{
"data": {
"internalSubjects": [
{
"title": {
"defaultMessage": " Customer"
},
"subjectClass": "END_USER",
"attributeValues": []
},
{
"title": {
"defaultMessage": "Employee"
},
"subjectClass": "EMPLOYEE",
"attributeValues": []
},
{
"title": {
"defaultMessage": "Contractor"
},
"subjectClass": "EMPLOYEE",
"attributeValues": []
},
{
"title": {
"defaultMessage": "Enterprise User"
},
"subjectClass": "END_USER",
"attributeValues": []
},
{
"title": {
"defaultMessage": "Authorized Agent"
},
"subjectClass": "END_USER",
"attributeValues": []
},
{
"title": {
"defaultMessage": "Custom Data Subject"
},
"subjectClass": "OTHER",
"attributeValues": []
},
{
"title": {
"defaultMessage": "Marketo Data Subject"
},
"subjectClass": "END_USER",
"attributeValues": []
},
{
"title": {
"defaultMessage": "Example Data Subject"
},
"subjectClass": "OTHER",
"attributeValues": []
}
]
}
}
Once you have your GraphQL query constructed, you will want to define it as a constant in your code. For example:
JavaScript
import { gql } from 'graphql-request'; const GET_DATA_SILOS = gql` query GetDataSilosForMetrics(first: Int!, $offset: Int!) { dataSilos(first: $first, offset: $offset, filterBy: { isLive: true }) { nodes { id title type connectedActions outerType teams { name } owners { name email } } } } `;
You then will want to use your GraphQL client from Step 2) to make this request:
JavaScript
const paged = await client.request(GET_DATA_SILOS, { first: 5, offset: 0, });
In most situations, the routes will be paginated with a maximum page size of 100. Depending on your use case, you may want to implement pagination:
JavaScript
const data = []; const MAX_PAGE = 100; let offset = 0; let pageSize = MAX_PAGE; while (pageSize === MAX_PAGE) { const paged = await client.request(GET_DATA_SILOS, { first: MAX_PAGE, offset, }); data.push(...paged.dataSilos.nodes); pageSize = paged.dataSilos.nodes.length; offset += MAX_PAGE; }
Check out the Fivetran connector documentation for more information.