Structured Discovery
Structured Discovery continuously detects and classifies all objects and properties inside a given database.Now that you have used Silo Discovery to discover data systems with personal data, you can use Structured Discovery to discover datapoints within those systems.
To classify this personal data, first you need to connect your data stores; there are three ways to do this:
Navigate to Structured Discovery on the left side menu. To add a data silo for datapoints scanning, click âAdd Data Silo".
![Adding a data silo for Structured Discovery](/_next/image?url=%2Fdocs%2Fscreenshot%2F2022-06-15-add-data-silo.png&w=3840&q=75)
Youâll now see a filtered list of data silos that are compatible with Structured Discovery. Add as many as you need by hovering over the data silos and selecting âQuick Addâ. This will take you to the Integrations view under the platformâs Infrastructure section. Find and select the data silo you just added.
![Quick add data silos for Structured Discovery](/_next/image?url=%2Fdocs%2Fscreenshot%2F2022-06-15-classification-catalog.png&w=3840&q=75)
From the âConnectionâ tab in the data silo, click âConnectâ and follow the connection instructions (see database integration documentation), such as entering your server, database, and login information.
![Instructions for connecting internal databases](/_next/image?url=%2Fdocs%2Fscreenshot%2F2022-06-15-database-connection.png&w=3840&q=75)
Once connected, click on the Structured Discovery tab and turn on the "Datapoint schema discovery", and âDatapoint classificationâ plugins.
From here, click back to Structured Discovery to see the results of this data silo scan.
P.S. Alternatively, you can add and configure data silos one by one. Select your desired data silo, scroll down and click the âAddâ button. Then, click âView Databaseâ to open up the view for this specific integration. From here, follow the connection instructions and turn on the "Datapoint schema discovery" and âDatapoint classification" plugins, as before.
![Adding an individual data silo for Structured Discovery](/_next/image?url=%2Fdocs%2Fscreenshot%2F2022-06-15-add-data-silo-modal.png&w=3840&q=75)
Follow a similar process to connect SaaS tools, like Salesforce. From Structured Discovery, click âAdd Databaseâ and select the desired SaaS vendor.
As with your database connection, navigate to the specific vendor from Infrastructure > Integrations and follow connection instructions. You may be prompted to connect with OAuth or another authorization protocol.
![Connecting Salesforce via OAuth](/_next/image?url=%2Fdocs%2Fscreenshot%2F2022-06-15-oauth-connection.png&w=3840&q=75)
Once connected, click on the Structured Discovery tab and turn on the "Datapoint schema discovery" and âDatapoint classification" plugins.
After turning Datapoint Schema Discovery on for a specific data store, you can adjust how often Transcend runs this plugin to scan for datapoints. Navigate to the specific data store from Infrastructure > Integrations, and then change the frequency inputs and start time under Structured Discovery.
Note: volumes scanned here are counted towards usage credits. If you are looking for your current scan volume, remaining allocation or want to adjust your plan, check with your Transcend account manager.
From here, clicking on âView Datapointsâ will take you back into Structured Discovery to a filtered view of the specific datapoints discovered from this data silo.
We will continue to scan your data based on the frequency set in Infrastructure > Integrations. You can see the status of a current scan, scheduled scan, or future scan date in the Structured Discovery view.
![Viewing the progress of a scan for a specific data silo](/_next/image?url=%2Fdocs%2Fscreenshot%2F2024-03-29-cc-scan-progress.png&w=3840&q=75)
The view you see above is the count of all objects of different types (differs per integration) found as part of the most recent scan run by Transcend on the data silo. This view is intended to provide you a progress indicator of how the scan Transcend is running is going.
The counts you see here may differ from the actual number of objects visible to you in the "Browse" view for the silo. This can be due to any number of reasons, some of which are: there is a change in permissions granted to Transcend for the data silo, or changes were made in the data silo's schema on your end.
You can click into the integration to see more details on the progress of the scan, as well as operational metrics around datapoints found, confirmed data categories, and progress on tables needing confirmation.
![Dashboard views for progress on confirming classifications for a specific integration](/_next/image?url=%2Fdocs%2Fscreenshot%2F2023-09-13-content-classification-integration-dashboard.png&w=3840&q=75)
We allow users to define custom regexes to help with classification. This can be done by navigating to the "Inventory" tab in the "Data Inventory" view. Here you can add, edit, and delete custom regexes to help with classification for each data category.
![Adding custom regexes to help with classification](/_next/image?url=%2Fdocs%2Fscreenshot%2F2024-04-11-custom-regexes.png&w=3840&q=75)
The custom regexes you define here will be used to help with classification in Structured Discovery. If a column in your data silo matches a custom regex, it will be classified as the data category you have defined in the "Data Inventory" view.
The results will appear with the labeled method of classification as "Regex Matching" in the "Datapoints" view in Structured Discovery.
![Classification Methods for each datapoint](/_next/image?url=%2Fdocs%2Fscreenshot%2F2024-04-11-classification-method.png&w=3840&q=75)
Transcend leverages machine learning techniques to quickly determine exactly where each and every personal datapoint lives within your individual data silo. With Structured Discovery, Transcend eliminates the need to derive queries for an internal database and maintain them through inevitable database schema changes.
We do this by prompting you to answer a series of simple questions related to the database's content. This trains our classifier and allows us to quickly learn your database schema and reliably detect the tables and rows that contain personal data.
Note: If you would like to try the newest classifier leveraging a Large Language Model (LLM), please check with your Transcend account manager.
From Structured Discovery, navigate to a specific data silo, then click on the âTrainâ tab. Answer our prompted question, Is the NAME datapoint a CATEGORY|SUBCATEGORY?â by either:
- Confirming by clicking on the button "Confirm Category" or pressing c on the keyboard.
- Selecting a different category from the dropdown, and then confirming it
- Skipping the category by clicking on the button "Skip Category" or pressing s on the keyboard.
![Train Transcend on classifying your personal data through a series of questions](/_next/image?url=%2Fdocs%2Fscreenshot%2F2024-04-16-train.png&w=3840&q=75)
As you answer these questions, Transcend will improve our classifier for the various data categories in your data silos.
You can also confirm classifications in bulk for data points with the same name as the one presented. You can see all instances of datapoints with the same name by clicking on the âView all instances of 'name'â button, which will open a new tab with all instances of datapoints with the same name. You can then confirm the category for all instances of the datapoint by clicking on the âBulk Confirmâ button.
The breadcrumbs you saw earlier can be traced through the âBrowseâ tab for each individual data silo. Here you can select the main folder and subfolders all the way down to a specific table.
![Browse data silo table](/_next/image?url=%2Fdocs%2Fscreenshot%2F2022-06-15-browse-salesforce.png&w=3840&q=75)
The âDatapointsâ tab lists out all datapoints in this data silo alongside their respective Data Category. This includes completed categories, those still in progress under the âTrainâ tab, and Unspecified categories. You can add notes and more information by clicking in the Description field and editing directly in line.
![View data, categories and purposes of processing](/_next/image?url=%2Fdocs%2Fscreenshot%2F2022-06-15-datapoints-salesforce.png&w=3840&q=75)
If you are still in the process of training Transcend on this data silo, you will see potential classification categories for each column alongside our confidence score for each category. We also include sample data below each column for you to reference.
Hover over each column to directly delete, add and edit categories. This will bypass the need for you to train Transcend on that specific table column.
Select âFilterâ in the top right to filter datapoints by Data Category, Purpose of Processing or classification status.
Connecting Data Stores
Utilize Structured / Unstructured Discovery to discover datapoints within your Data Silos
Browse, Edit, and Filter Structured Discovery Results
Find your target Data Silos and/or Datapoints with ease through filtering
Confirming / Training Classifications
Leverage machine learning to quickly determine exactly where each and every personal datapoint lives within your individual data silo
Developing Classifications
Understand how classification works for the different Data Discovery products
Managing Connected Data Stores from Discovery
Manage and view your connected data stores with Structured / Unstructured Discovery
Structured and Unstructured Discovery Overviews
Get to know the high-level details of Transcend's data mapping products, Structured and Unstructured Discovery