Azure data platforms separate storage, ingestion, transformation, serving, governance, and consumption. Data Factory orchestrates and moves data. Data Lake Storage provides scalable object storage with hierarchical namespace. Azure Databricks offers a managed Spark/lakehouse platform. Microsoft Fabric integrates data engineering, factory, warehouse, real-time intelligence, science, and Power BI under a SaaS capacity model.
These products overlap. Choose an operating model, not the largest service list. A small team rarely benefits from two competing orchestration systems without a clear boundary.
Names such as bronze/silver/gold are less important than explicit contracts: ownership, schema, quality, retention, access, lineage, and recovery.
Ways to build
Choose the Azure tool you want to use. The underlying resource stays the same.
Open Data factories → Create, then launch the authoring studio. Create managed-identity linked services/connections, source and sink datasets, and a parameterized pipeline with Copy activity. Use Debug with non-sensitive sample data, publish through Git integration, trigger a run, and verify its input, output, row counts, and duration in Monitor.
Use managed identity for supported data sources. Use a self-hosted integration runtime only when network placement requires it, and patch/scale that runtime like production infrastructure.
Bicep/Terraform can also create the factory, identity, private endpoints, linked-service scaffolding, Git configuration, and diagnostics. Pipeline JSON should remain versioned. Do not embed database passwords in pipeline definitions.
Fabric organizes experiences around workspaces and capacities with OneLake integration. Capacity sizing, item permissions, domains, deployment pipelines, and tenant governance are central.
Databricks organizes workspaces, clusters/serverless compute, Unity Catalog, notebooks/jobs, Delta tables, and ML capabilities. Network injection, private link, cluster policies, catalogs, secretless cloud access, and job compute policies are central.
Run a proof of value with representative pipelines, concurrency, security, BI/ML needs, and total operating cost. Avoid comparing only list price per compute unit.
Curate approved, lineage-tracked document and structured datasets before indexing them for RAG. Separate raw confidential data from AI-ready views. Apply row/column security upstream. Record document version, access labels, source URI, retention, and deletion events so the search index can honor governance.
Infrastructure code should create workspaces/factories, storage, identities, networking, role assignments, policies, diagnostic destinations, and capacity where supported. Data product code should version notebooks, SQL, transformations, tests, schemas, and pipelines. Some SaaS configuration may require provider/API support beyond ARM; document manual gates rather than hiding them.