Snowflake, dbt, and the Modern Data Stack in an AI World

The modern data stack that emerged between 2018 and 2022 is adapting to the AI era. The tools that managed analytical data are becoming the infrastructure for AI training data, feature stores, and model monitoring.

The modern data stack defined

The modern data stack refers to a set of cloud-native tools that handle analytical data workflows: a cloud data warehouse (Snowflake, BigQuery, or Redshift) for storage and query, dbt for data transformation and lineage, Fivetran or Airbyte for data ingestion, and Looker or Metabase for visualisation. This stack replaced complex on-premises ETL pipelines with composable cloud services connected by SQL.

dbt as the transformation layer

dbt (data build tool) standardised SQL-based data transformation in the data warehouse. Rather than writing ETL scripts that extract, transform, and load data into new tables, dbt treats your transformation logic as code in the repository, with testing, documentation, and lineage tracking built in. The adoption of dbt is now essentially universal among modern data teams. Its 2023 momentum is from expansion: dbt Semantic Layer for consistent metric definitions across tools, and dbt Cloud for managed execution.

Snowflake and the AI integration

Snowflake acquired Streamlit in 2022 and integrated it into Snowflake to let data scientists build ML applications that query Snowflake directly. Cortex, Snowflake's ML functions layer, adds LLM capabilities (summarisation, classification, translation) directly in SQL queries. The vision is data and AI in the same platform, removing the ETL step between your data warehouse and your ML infrastructure.

The data quality problem for AI

AI models are only as good as their training data. The modern data stack's investment in data quality, testing in dbt, schema validation in Great Expectations, data observability in Monte Carlo, is becoming directly relevant to AI development. Teams that have invested in data quality for analytics have a head start on AI data infrastructure. Teams that have not now have two reasons to invest.