Databricks releases, every cloud
- Unit testing for pipelines (Beta)
Validates Python and SQL transformations against mock data. Requires Lakeflow Pipelines Editor.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - C++ SDK for the Zerobus Ingest connector is now available in Beta
Supports JSON and Protocol Buffers for ingesting into Unity Catalog Delta tables.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Override the REPLACE WHERE predicate for a single refresh
Adds a WHERE clause to the REFRESH STREAMING TABLE command. Applies to standalone streaming tables with FLOW REPLACE WHERE clause.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Reddit Ads connector is now available in Beta
Ingests advertising campaign data and performance metrics from Reddit Ads into Databricks.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Anthropic managed ingestion connector (Beta)
Ingests Compliance API data, including audit activity and Claude chats, into Databricks.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Query-based connectors in Lakeflow Connect will soon be available by default for workspaces with the compliance security profile enabled Coming soon
Replaces CDC configuration, supports Oracle, Teradata, and others.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Run the change, do not just read it. Hands-on labs in your own Databricks workspace, graded when you submit. Browse labs
- Change to default pipeline editor for workspaces with the compliance security profile enabled Coming soon
Replaces legacy editor, removing it in August.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Strac connector (Beta)
Ingests Strac data-security audit events using a Strac API key.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Behavior changes for continuous jobs and pipelines Coming soon
Configures continuous schedule from pipeline page, wrapping pipeline in a continuous job. Replaces pipeline's built-in continuous setting for new pipelines.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - REPLACE WHERE flows are now generally available
Replaces full table reprocessing with row-level updates based on a predicate.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Standard performance mode is now supported for materialized view or streaming table refreshes in jobs
Consumes fewer DBUs when Performance optimized setting is off.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Lakeflow Connect row filtering is now generally available
Works like a SQL WHERE clause, applies to initial and incremental updates.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Veeva Vault connector (Beta)
Ingests standard and custom objects from Veeva using OAuth 2.0 M2M auth.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Run a subset of job tasks with their dependencies
Add + modifier to task key to include dependencies, usable via UI, API, or CLI.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - SQL alert task in Lakeflow Jobs is now generally available
Evaluates a Databricks SQL alert as part of a Lakeflow Job, returning its state for downstream tasks.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Microsoft Dynamics 365 connector (GA)
Ingests CRM and ERP data into Databricks using Azure Synapse Link.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Change the owner of standalone pipeline streaming tables and materialized views
Uses ALTER STREAMING TABLE and ALTER MATERIALIZED VIEW with SET OWNER TO. Applies to standalone pipeline datasets in Databricks SQL.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Standalone pipeline materialized views support expectations
Validates records in materialized views using CONSTRAINT clause. Replaces manual validation in standalone pipelines.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Get this table by email. One Monday mail covering the week, filtered to the clouds and products you run. Weekly digest
- Square managed ingestion connector (Beta)
Ingests point-of-sale data like payments and orders from Square into Databricks.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Wiz Audit Logs managed ingestion connector (Beta)
Ingests Wiz audit log entries, issues, and vulnerability findings into Databricks.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Netskope Logs managed ingestion connector (Beta)
Ingests Netskope event and alert logs into Databricks.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Incremental ingestion for Workday Reports (Beta)
Ingests only changed rows since last pipeline run if a cursor column is configured.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Salesforce Marketing Cloud managed ingestion connector (Beta)
Ingests campaigns, journeys, and email events from Salesforce Marketing Cloud into Databricks.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Zip connector (Beta)
Ingests data from Zip, a new option in Lakeflow Connect.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Reorganized Spark Declarative Pipelines on Lakeflow documentation
Conceptual topics are now under Concepts.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - The update_flow API for Spark Declarative Pipelines on Lakeflow is generally available
Writes to a sink in update output mode, emitting only changed rows. Supports stateful aggregations without a watermark.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Aha! connector (Beta)
Ingests data from Aha!, using Lakeflow Connect.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Monday.com managed ingestion connector (Beta)
Ingests boards, users, teams, and activity logs from Monday.com into Databricks.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Zoho Books managed ingestion connector (Beta)
Ingests invoices, bills, contacts, and bank transactions from Zoho Books into Databricks.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - External data access for pipeline streaming tables and materialized views (Public Preview)
Enables external Delta and Iceberg clients to read data without copying. Uses Unity Catalog and Iceberg catalog REST APIs.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Managed SharePoint connector adds structured file ingestion and file metadata support (Beta)
Supports CSV, JSON, XML, Excel, Parquet, Avro, ORC ingestion, replacing unstructured-only approach.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Table update triggers on Delta Sharing and system tables (Beta)
Monitors data shared through Delta Sharing and system tables, triggering jobs on updates, supporting Databricks-to-Databricks sharing only.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Managed RabbitMQ connector (Beta)
Streams messages from RabbitMQ classic queues into Unity Catalog streaming tables.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Pendo connector (Beta)
Ingests data from Pendo into Lakeflow. Requires Lakeflow Connect.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Query-based connectors for Lakeflow Connect are now generally available
Ingests data from databases like Oracle and PostgreSQL without CDC configuration.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Google Drive managed ingestion connector (Beta)
Ingests files from Google Drive into Databricks, supporting unstructured and structured data.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Real-time mode in Spark Declarative Pipelines on Lakeflow, and the update_flow API are now available (Public Preview)
Achieves end-to-end latency as low as 5ms, uses update output mode.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Refresh policies and EXPLAIN CREATE MATERIALIZED VIEW for materialized views are now generally available
Requires Databricks Runtime 17.3 and above. Available in Databricks SQL.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Spark Declarative Pipelines on Lakeflow parameters are now in Beta
Define key-value pairs at pipeline level, referenced in SQL using named parameter syntax.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud) - Unified runs list is generally available
Replaces separate job and pipeline run lists, adds filtering and error code visualization.
AWS, read it on their docsAzure, read it on their docsGCP, read it on their docsSAP(not on this cloud)
Headlines, dates and product areas are Databricks' own, from the release notes published for each cloud, and every item links to the note it came from. The one-line summaries are ours. Not a Databricks product and not affiliated with Databricks, Inc.