Create and configure dbt project

Make sure that the following prerequisites are met:

• You are working with DataSpell version 2023.3 or later. If you still do not have DataSpell, download it from this page. To install DataSpell, follow the instructions, depending on your platform.

• You have access to a data platform.

note

When dbt_project.yml file is found in the project directory, DataSpell activates dbt support for the project. Once dbt support is activated, an icon that resembles the dbt logo will appear on top of the project's structure.

The newly created project contains dbt-specific files and directories.

The structure of the project is visible in the Project tool window (Alt01):

• analyses directory is used for storing ad-hoc SQL queries or analyses that aren't part of the main data transformation logic. These queries are often used for exploratory analysis or one-time investigations.

• macros directory is where you can store SQL files that define reusable snippets of SQL code called macros. Macros can be used to encapsulate commonly used SQL patterns, making your code more modular and easier to maintain.

• models directory is one of the most important directories in a dbt project. It's where you define your dbt models, which are SQL files containing the logic for transforming and shaping your data. Models are the core building blocks of a dbt project.

• seeds directory, is where you can store seed data in a dbt project. Seeds are static datasets that you manually create and manage. Unlike source tables, which dbt typically reads directly from a data warehouse, seeds are user-defined tables that you provide as input to your dbt models.

• snapshots directory is used for creating incremental models or snapshots of the data. Snapshots are useful when you want to capture changes in the data over time.

• tests directory is where you define tests for your dbt models. Tests help ensure the quality of data transformations by checking for expected outcomes, such as verifying that certain columns are not null or that a column is unique.

• dbt_project.yml is the main configuration file for your dbt project. It contains settings such as your project name, source configurations, and target configurations.

• README.md file provides an introductory welcome and a list of useful resources.

These directories and files collectively provide a structured environment for developing, testing, and documenting your data transformations using dbt.

When you run a dbt command, dbt reads the dbt_project.yml file to identify the project's name, and then looks for a profile with the same name within the profiles.yml file.

Create a profiles.yml file in your home directory (~/.dbt), and configure it with the necessary information to connect to your data warehouse:

# example profiles.yml file
dbtProject: #your_project's_name
  target: dev
  outputs:
    dev:
      type: postgres
      host: localhost
      user: jetbrains
      password: <password>
      port: 5432
      database: sakila
      schema: dbt_jetbrains
      threads: 4

Depending on a database vendor, you need to configure a corresponding data source to use it to connect to your data platform.

To check your warehouse connection, run dbt debug command.