dbt core
Getting started with dbt Core
dbt is a secondary integration that adds additional metadata on to your data warehouse or relational database tables. Before connecting dbt make sure to connect a data warehouse or relational database first. These include Snowflake, BigQuery, Postgres, Redshift, etc.
There are several options to connect dbt core with Seda:
(Recommended) Connect an AWS, GCP, or Azure storage bucket/container
Upload a
manifest.json
andrun_results.json
through the UIUpload a
manifest.json
andrun_results.json
through the API
Option 1 – Storage bucket (container)
This option is recommended to ensure that Seda always has the latest manifest.json
and run_results.json
files from dbt Core. Seda will only sync these files from the bucket.
1a. Connect an AWS S3 bucket
You can connect to the AWS S3 bucket using an AWS IAM user, or AWS Roles.
1b. Connect a GCS S3-compatible bucket
Login to GCP cloud console.
Create a service account.
Grant access to the service account from the Bucket page as “Storage Object Viewer”.
Turn on interoperability on the bucket. Generate HMAC keys for a service account with read access to the bucket. Both located here:
Setup CORS. GCP requires this be done over CLI. Like the following:
Copy
gsutil cors set cors.json gs://bucket-name
cors.json
Copy
[
{
"origin": ["*"],
"method": ["GET", "POST", "PUT", "DELETE", "OPTIONS"],
"responseHeader": ["Content-Type"],
"maxAgeSeconds": 3600
}
]
Save the HMAC keys to be used in the connection form.
Access Key Id
Secret
Region bucket region for GCP
S3 Endpoint must be added and set to
https://storage.googleapis.com
Connect your S3 bucket to Seda
Navigate to https://app.seda.co/integrations/new and click dbt Core
Choose the Access Key tab and add the HMAC keys saved above to the relevant fields.
Test the Connection - if successful you'll be prompted to run your initial sync
1c. Connect a Azure Blob Storage container
Go to portal.azure.com and then click Storage accounts.
Copy the name of the desired storage account. Enter that in the integration form.
Click on your storage account and under Security + networking select Access keys.
Copy the Connection string and add to your integration form.
Test the connection.
Option 2 – Upload a single manifest.json
The dbt manifest file contains complete information about how tables are transformed and how they are connected in terms of data lineage. It details the model to table relationships, providing a complete and accurate lineage view.
This is a one time sync with your manifest.json file. You can upload the file following these steps:
Navigate to https://app.seda.co/integrations/new and click dbt Core
Choose the File Upload tab and select your manifest.json and run_results.json files using the file select
Test the Connection - if successful you'll be prompted to run your initial sync
4. Now that a sync has been triggered, you can now monitor your sync in the UI!
Last updated