Databricks

Databricks Data Intelligence Platform for Retail comes to NRF 2024

Learn more about Databricks’ participation in the NRF Big Show in January 2024, including its Data Intelligence Platform demos and how the platform better empowers retailers to meet consumer demands and solve operational challenges.

Predict & Analyze marketing campaign effectiveness with SAP Datasphere and Databricks Data Intelligence Platform

Predict & Analyze marketing campaign effectiveness with SAP Datasphere and Databricks Lakehouse.

Databricks @ AFROTECH

Members of our DEI Team and Black Employee Network attended #AFROTECH23 in Austin, TX earlier this month! Plus for the first time ever, Databricks was represented on the Product & Engineering Stage.

Even Santa Claus has AI fever

Christmas Eve is an operational nightmare. This blog walks through using Databricks' Foundation Model APIs and Llama2 with MosaicML Inference to create an AI model for analyzing children’s letters to Santa and identifying the right present! Find out how they did it by leveraging techniques like synthetic data generation and name entity recognition with the Databricks Data Intelligence Platform.

Introducing Databricks Assistant for Help

Building on the momentum of Databricks Assistant, the context-aware AI assistant integrated within Databricks Notebooks, SQL editor, and file editor, and now powering AI-generated documentation in Unity Catalog, we are thrilled to announce the public preview release of Databricks Assistant for Help.

Managing AI Security Risks: Introducing a new workshop for CISOs

An overview of the security risks associated with AI and ML systems for CISOs.

Offline LLM Evaluation: Step-by-Step GenAI Application Assessment on Databricks

Background

Bringing the Lakehouse to R developers: Databricks Connect now available in sparklyr

We’re excited to ann

Lakehouse Monitoring: A Unified Solution for Quality of Data and AI

Databricks Lakehouse Monitoring allows you to monitor all your data pipelines – from data to features to ML models – without additional tools and complexity. Built into Unity Catalog, you can track quality alongside governance and get deep insight into the performance of your data and AI assets. Lakehouse Monitoring is fully serverless so you never have to worry about infrastructure or tuning compute configuration. 

Delivering cost-effective data in real time with dbt and Databricks

Learn how to leverage Streaming Tables and Materialized Views directly on dbt-databricks, generating near-real time insights with the simplicity of SQL!

Build GenAI Apps Faster with New Foundation Model Capabilities

Databricks Model Serving now offers a unified interface, making it easier to experiment, customize, and productionize foundation models across all clouds and providers. This means you can create high-quality GenAI apps using the best model for your use case while securely leveraging your organization's unique data. The new unified interface lets you manage all models in one place and query them with a single API, no matter if they're on Databricks or hosted externally. Additionally, we are releasing Foundation Model APIs that provide you instant access to popular LLMs, such as Llama2 and MPT models, directly from within Databricks. These APIs come with on-demand pricing options, such as pay-per-token or provisioned throughput, reducing cost and increasing flexibility.

Improve your RAG application response quality with real-time structured data

We are excited to announce the public preview of Databricks Feature & Function Serving, a low latency real-time service designed to serve structured data from the Databricks Data Intelligence Platform. Feature Serving real-time endpoints enable you to instantly access your pre-computed ML features. Function Serving is an all-new cutting edge technology that enables you to serve any Python function in Unity Catalog as an endpoint for real-time data transformations. Together they streamline your AI development, unlocking a plethora of real-time applications such as rule engines and on-demand feature calculation and serving in classical ML and Gen AI applications.

Accelerate Business Value from Data Sharing with Databricks Unity Catalog and Tredence UnityGO!

Tredence has launched a new Brickbuilder Accelerator, UnityGO!, which speeds up enterprise data implementations to Unity Catalog on the Databricks Data Intelligence Platform.

Introducing Databricks Vector Search Public Preview

Following the announcement we made yesterday around Retrieval Augmented Generation (RAG), today, we’re excited to announce the public preview of Databricks Vector Search. We announced the private preview to a limited set of customers at the Data + AI Summit in June, which is now available to all our customers. Databricks Vector Search enables developers to improve the accuracy of their Retrieval Augmented Generation (RAG) and generative AI applications through similarity search over unstructured documents such as PDFs, Office Documents, Wikis, and more. Vector Search is part of the Databricks Data Intelligence Platform, making it easy for your RAG and Generative AI applications to use the proprietary data stored in your Lakehouse in a fast and secure manner and deliver accurate responses.

Creating High Quality RAG Applications with Databricks

Retrieval-Augmented-Generation (RAG) has quickly emerged as the canonical way to incorporate proprietary, real-time data into Large Language Model (LLM) applications. Today we are excited to announce a suite of RAG tools to help Databricks users build high-quality, production LLM apps using their enterprise data.

Automotive Giant Turns Data Into Business Value With Databricks

Leading the charge is one of the largest American automakers with 165,000 employees focused on pushing the envelope of automotive engineering, design, and technology.

Announcing Databricks Middle East Expansion and Launch of Azure Qatar

Databricks launches Azure Qatar, making it easier than ever for organizations across the Middle East to leverage Databricks Data Intelligence Platform to unify data engineering, data science, machine learning, and analytics on Azure’s global, scalable, and elastic network.

Managing Recalls with Barcode Traceability on the Delta Lake

In this article, we argue why a central Delta Lake on top of multiple production plants dramatically helps reduce affected damage by shortening problem-solving cycle times.

Databricks Insurance: Live in the Guidewire Marketplace

To mark the announcement of Databricks listing in Guidewire Marketplace, our GTM Director for Insurance, sat down to discuss why Databricks inclusion in the Guidewire Marketplace matters most for Insurance companies.

Crafting a Data-Driven Talent Intelligence Strategy in Healthcare

To understand the healthcare talent landscape, organizations need a comprehensive 360-degree view of the market. That's where People Data Labs (PDL) and LinkUp help - providing data on both employee data and job market data.

Presenting New Partner Integrations in Partner Connect

Introducing five new integrations in Databricks Partner Connect—a one-stop portal to use partner solutions with your Databricks Data Intelligence Platform data.

Automating Governance of PHI Data in Healthcare

With Databricks Unity Catalog, healthcare organizations can securely manage and govern sensitive information including PHI. Find out how in this blog.

Enhancing your team’s performance by building a data culture

Building a data culture can be challenging for many reasons. There are many studies that show us the value of being more data-driven. Here are five steps that every organization can take to be more data forward.

Databricks Wins AWS ISV Partner of the Year Award in NAMER

Databricks has won the AWS ISV Partner of the Year award for North America. Achieving this recognition underpins the strength of our partnership and ability to provide a simple, unified data intelligence platform that’s seamlessly integrated with AWS.

Your data, your model: How custom LLMs can turbocharge operations while protecting valuable IP

Given the influence of generative AI on the future of many enterprises, bringing model building and customization in-house becomes a critical capability. Naveen Rao (CEO and Co-founder of MosaicML, acquired by Databricks) shares more in this blog from Databricks.

Guest Post: Real-Time Fraud Detection in the Lakehouse

In this blog, we’ll introduce some key ways in which you can leverage Tecton on Databricks to build your real-time fraud detection system. Read through for some actual examples at the end!

Unwrapping the secrets of a data + AI career

Upskill today across data engineering, data analysis, and data science. Join the thousands who have elevated their career with Databricks training & certification.

Creating a bespoke LLM for AI-generated documentation

Learn how Databricks built a bespoke LLM for AI-generated documentation in Databricks Unity Catalog. This feature was built by two engineers in just one month.

Announcing Enhanced Control Flow in Databricks Workflows

Databricks Workflows now offers enhanced control flow with the introduction of conditional execution and job parameters, now generally available.

Cybersecurity Lakehouses Best Practices Part 4: Data Normalization Strategies

Tips, tricks and best practices for normalizing machine-generated cybersecurity data into a common information model.

Announcing the General Availability of Azure Databricks support for Azure confidential computing (ACC)

Today we are excited to announce the general availability of Azure Databricks support for Azure confidential computing (ACC), allowing you to run your Azure Databricks workloads on Azure confidential virtual machines (VMs).

Data Intelligence Platforms

Our viewpoint on how AI will fundamentally change data platforms, and how data will change enterprise AI

Python Dependency Management in Spark Connect

The blog post introduces session-based dependency management in Spark Connect for Apache Spark 3.5.0. It allows updating Python dependencies during runtime.

Named Arguments for SQL Functions

Introducing named arguments for SQL functions - simplifying function invocation and boosting user productivity.

Cybersecurity Lakehouses Part 3: Data Parsing Strategies

Tips, tricks and best practices for parsing cybersecurity log sources into the Cybersecurity Lakehouse.

It’s Time for Streaming Architectures for Every Use Case

If you are processing a lot of data, you probably need to implement a streaming architecture. From once-a-day to once-a-second and below, Databricks makes it easy and saves you money.

How the Lakehouse can optimize provider networks and improve member care

You can now find Ribbon Health's Provider Directory & Location Directory on the Databricks Marketplace so health plans and care providers/navigators can start using this data today.

Fastest way to get SAP HANA data into Databricks using SAP FedML

This blog is about the quickest way of accessing SAP HANA’s data in Databricks for doing ML/AI on SAP data.

Simplifying Production MLOps with Lakehouse AI

We’re excited to announce that the second edition of the Big Book of MLOps is now available for download. The Book walks through how to architect both an MLOps and LLMOps setup using the Lakehouse AI tools.

Announcing Public Preview of AI Generated Documentation In Databricks Unity Catalog

AI-generated documentation for tables and columns in Unity Catalog

Introducing Redkite’s Pricing Analytics Solution Powered by Databricks

Learn about Redkite’s Pricing Analytics Solution built on the Databricks Lakehouse Platform to help customers make crucial pricing insights and navigate the outcomes to make informed decisions in support of their short-term and long-term pricing goals.

LLM Training on Unity Catalog data with MosaicML Streaming Dataset

We are introducing dataset preprocessing to MosaicML Data Shards (MDS) format using Spark and streaming the same dataset from Unity Catalog (UC) Volume into a training cluster for Generative AI training workflows.

Building a complete and composable CDP on the Lakehouse

Databricks Ventures invested in Hightouch to power the Data Lakehouse-native Customer Data Platform. Hightouch provides all the features that Databricks users need to collect, store, model, and activate customer data directly from the Lakehouse.

Announcing the General Availability of the Databricks SQL Statement Execution API

We are excited to announce the general availability of the Databricks SQL Statement Execution API on AWS and Azure. You can use the API to connect to your Databricks SQL warehouse over a REST API to access and manipulate data managed by the Databricks Lakehouse Platform.

Scalable, In-House Quality Measurement with a NCQA-Certified Engine on the Lakehouse

Building quality measurements for NCQA measures can be accelerated on the Databricks Lakehouse. Learn how ApolloMed can help.

LLM Inference Performance Engineering: Best Practices

In this blog post, the MosaicML engineering team shares best practices for how to capitalize on popular open source large language models (LLMs) for production usage.

How Blitz and Databricks are Powering a New Era of Competitive Gaming

Blitz simplifies the process of mastering difficult games by helping gamers improve at every step of their journey using data and AI.

Announcing public preview of Databricks Asset Bundles: Apply software development best practices with ease

Databricks Asset Bundles facilitate the adoption of software engineering best practices, including source control, code review, testing and continuous integration and delivery (CI/CD).

Llama 2 Foundation Models Available in Databricks Lakehouse AI

We’re excited to ann

How Crisp and Databricks Enable Retail-Supplier Collaboration through Shared Sales and Inventory Data

Retailers have long

Databricks Obtains ISO 27701 Certification

We’re excited to announce that Databricks has obtained the International Standards Organization (ISO) 27701 certification as a data processor.

Databricks and Shell collaborate to simplify industrial time series data analytics on the Lakehouse

Databricks and Shell worked together to develop an open-source, cloud-native framework which extends the lake house to accommodate their global footprint of industrial time series data.

A Pattern for the Lightweight Deployment of Distributed XGBoost and LightGBM Models

This blog describes a simple approach for converting distributed models into single-threaded variants for lightweight model deployment. This is helpful in scenarios where a XGBoost, LightGBM or similar model is trained against a large data set using a Spark cluster but needs to be deployed within a container or Databricks model serving.

Accelerate Your AI Journey with Pre-built Industry Solutions on Databricks Marketplace

Innovate faster using new industry-specific Solution Accelerators with pre-built notebooks and time-saving tools – now available on Databricks Marketplace.

How Ribbon Health and Databricks Unlock Better Patient Care

By leveraging Ribbon’s capabilities through Databricks, customers can save significant time and cost with streamlined data flows, automate and scale data intake processes, and improve “Find Care” experiences to drive member satisfaction and retention.

Announcing Inference Tables: Simplified Monitoring and Diagnostics for AI models

Inference Tables enable you to continuously capture input and predictions from Model Serving endpoints and log them into a Unity Catalog Delta Table.

Databricks Expands Brickbuilder Program to Include Lakehouse Accelerators

We’re excited to announce Brickbuilder Accelerators, an expansion to the Brickbuilder Program that pairs the expertise of system integrator and consulting partners with their proven frameworks and pre-built code.

Bringing Software Engineering Best Practices to Life Sciences R&D at Exai Bio

Learn how Exai Bio is bringing software engineering best practices to life sciences research with the Lakehouse.

Cracking the Code: How Databricks is Reshaping Major League Baseball with Biomechanics Data

Databricks, with its Lakehouse architecture, is the game-changing solution that empowers MLB teams to harness the full potential of biomechanics data.

Ballard Power Systems RDU (Remote Diagnostics Unit) Visualization Platform for Interactive At-Scale Industrial IoT Streaming Analytics

To lead in the zero-emission energy space, Ballard has combined the power of Databricks and Plotly Dash to build an RDU (Remote Diagnostics Unit) Visualization Platform capable of delivering actionable information globally to business leaders.

Announcing the Public Preview of Lakeview Dashboards!

We are excited to announce the public preview of the next generation of Databricks SQL dashboards, dubbed Lakeview dashboards. Available today, this new dashboarding experience is optimized for ease of use, broad distribution, governance and security.

Deploy Private LLMs using Databricks Model Serving

Deploy Generative AI Models with full control of your Data and Model.

easyJet bets on Databricks Lakehouse and Generative AI to be an Innovation Leader in Aviation

This blog is authored by Ben Dias, Director of Data Science and Analytics and Ioannis Mesionis, Lead Data Scientist at easyJet 

Governing cybersecurity data across multiple clouds and regions using Unity Catalog & Delta Sharing

TCO ensures customers keep costs in line, particularly in enhancing security measures. Query federation is vital for real-time threat analysis, all while avoiding the security risks associated with copying data across geographic boundaries.

Making Spark Accessible: My Databricks Summer Internship

My summer internship on the PySpark team was a whirlwind of exciting events. The PySpark team develops the Python APIs of the open-source Apache Spark library and Databricks Runtime.

New Support for Conflict Resolution in Repos: Merge, Rebase and Pull

At Databricks, users can now run Git merge, Git Rebase and resolve merge conflicts directly from the Repos UI.

The Data + AI Trifecta: People, Process, and Platform

It’s easy to underestimate the scope of changes to put data and AI to work. Our Databricks CTOs share practical advice on how to bring people, processes, and platforms along on your transformation journey.

Using Images and Metadata for Product Fuzzy Matching with Zingg

Product matching is an essential function in many retail and consumer goods organizations. Learn how Zingg can be used on Databricks to perform product matching with product metadata and images.

Congratulations to Summer Hackathon Winners!

Data teams from around the globe came together to build a unique use case and or technique showcasing LLMs. Check out the awesome projects that came out ahead.

Solution Accelerator: LLMs for Manufacturing

Since the publicatio

Apache Spark ❤️ Apache DataSketches: New Sketch-Based Approximate Distinct Counting

Say goodbye to traditional counting methods and welcome powerful SQL functions into your toolkit! Unlock the full potential of your data analysis workflows and make informed decisions based on accurate approximations of uniqueness.

How Edmunds builds a blueprint for generative AI

Long envisioned as a key milestone in computing, we've finally arrived at the moment where machines can seemingly understand us and respond in our own natural language. While no one should be confused that large language models (LLMs) only give the appearance of intelligence, their ability to engage us on a wide range of topics in an informed, authoritative, and at times creative manner is poised to drive a revolution in the way we work.

Orchestrating Data Analytics with Databricks Workflows

In the evolving landscape of data analysis tools, Databricks Workflows bridges the gap between data analysts and the complexities of data orchestration.

Unexpected Tools in the Databricks Marketplace to Supercharge Manufacturing Supply Chains

Optimizing supply chain operations is a strategic imperative for manufacturing, automotive, and energy firms to secure a competitive edge. A data solution that brings together all of their data and flexes to meet their needs is critical.

Introducing the Support of Lateral Column Alias

Lateral Column Alias (LCA simplifies complex SQL queries by allowing users to reuse an expression specified earlier in the same SELECT list, eliminating the need to use nested subqueries and Common Table Expressions (CTEs) in many cases.

A Costa Rica journey with a Twist of Pura Vida

Ariel Vargas, Director of International Operations, shares an in-depth look at the expansion journey of Databricks Costa Rica. See how Bricksters represent Pura Vida and its importance to building community within the tech sector.

Introducing Apache Spark™ 3.5

Learn more about the latest release of Apache Spark, version 3.5, including Spark Connect, and how you begin using it through Databricks Runtime 14.0.

The Power of a Trusted Data Lakehouse: Go Bust or Boom

In this blog, we'll look at a typical desired data governance framework, what it looks like with Unity Catalog at the core, and how organizations can integrate the solution with other custom or third-party platforms for data governance.

Introducing MLflow 2.7 with new LLMOps capabilities

As part of MLflow 2’

Bringing Scalable AI to the Edge with Databricks and Azure DevOps

From better alignment of production with consumer demand to improved process control, ML/AI is poised to transform how manufacturers run their businesses, making these technologies a principal area of focus in Industry 4.0 initiatives.

Crossing Bridges: Reporting on NYC taxi data with RStudio and Databricks

One objective for the Posit and Databricks partnership is to improve support for Spark Connect in R through sparklyr, simplifying the process of connecting to Databricks clusters via Databricks Connect.

Best Practices for LLM Evaluation of RAG Applications

Chatbots are the mos

Retail Personalization with RFM Segmentation and the Composable CDP

Databricks partners with Data Activation platform Census to support a best-of-breed approach to personalized, data-driven marketing with the Composable CDP.

Introducing Databricks Bengaluru Development Center

Databricks recently announced a new development center in Bengaluru, India. Learn how the company is building out its R&D teams and other updates!

Shared Clusters in Unity Catalog for the win: Introducing Cluster Libraries, Python UDFs, Scala, Machine Learning and more

We are thrilled to announce great enhancements to onboard more workloads to Unity Catalog clusters in shared access mode, Databricks' highly efficient, secure multi-user clusters.

Upskill with instructor-led training and save 20% off today

At Databricks, we want to ensure that our users and community are always staying ahead and set up for success. For years, we’ve been creating, and improving technical trainings for learners at all levels and across multiple disciplines.

Databricks introduces the Delivery Solutions Architect

At Databricks, this year, we launched a role that revolutionizes the way businesses leverage data and drive digital transformation: the Delivery Solution Architect.

Efficient Fine-Tuning with LoRA: A Guide to Optimal Parameter Selection for Large Language Models

With the rapid advan

Activating Data from the Lakehouse: Databricks Ventures Invests in Hightouch

Databricks Ventures invests in Hightouch, the leading Data Activation platform.

Getting started with generative AI in healthcare and life sciences

ChatGPT has influenced every industry to reexamine their AI strategies. LLMs have elevated AI to a strategic initiative healthcare leaders can no longer ignore.

Automated Analysis of Product Reviews Using Large Language Models (LLMs)

Learn about this Solution Accelerator to streamline the summarization of customer feedback, allowing your organization to process a high volume of reviews at a lower cost, and task an LLM to extract different sets of information from each high-level category of reviews

Optimizing Promotional Offers using Causal Machine Learning

Learn how to use causal machine learning to optimize promotional offers for customers to increase sales

Simplify and Accelerate IoT Data-Driven Innovation

The lakehouse platform allows teams to simplify the data architecture, efficiently scale unit cost of data processing, and power all use cases, truly allowing customers to accelerate data-driven innovation and capture the immense value potential in IoT data.

The Simplification of AI Data

This summer, Databricks has taken another monumental stride by integrating feature engineering and management directly into Unity Catalog. This convergence not only strengthens AI data and feature management but aligns it with top-tier data catalogs, refining the process of feature engineering and model training.

Introducing the latest integrations available in Partner Connect

We are excited to announce two new integrations in Databricks Partner Connect: Alation and Hunters. These partnerships extend the power of the lakehouse, with Alation enabling seamless data discovery, collaboration, and governance, and Hunters empowering proactive threat detection and response.

Processing Uncommon File Formats at Scale with MapInPandas and Delta Live Tables

Ingest and process files of any format into your Lakehouse using Spark mapInPandas() functions in Delta Live Tables pipelines.

Using MLflow AI Gateway and Llama 2 to Build Generative AI Apps

Virtual Event Advantage Lakehouse: Fueling Innovation in Data and AI Lakehouse architecture is built for modern data and AI initiatives. Discover how it unifies data to speed up everything from ETL to SQL to AI.

How ActionIQ Integrates with Databricks Lakehouse Part Two: Step-by-Step Workflow to Activate Propensity Modeling

ActionIQ, powered by its HybridCompute technology, offers a unique integration pattern that allows organizations to deploy their composable CDP directly within the Databricks Lakehouse environment.

Introducing "Ask Databricks": Your Direct Line to Our Product Experts!

Introducing "Ask Databricks," an engaging and fun new live streaming series hosted in collaboration with Advancing Analytics, providing Databricks users and data practitioners a direct line to product experts. The series offers a forum to interact with product managers and engineers, gaining valuable insights and best practices for optimizing Databricks products and the Lakehouse Platform.

How ActionIQ Integrates with the Databricks Lakehouse Part One: Enable Personalization Without Data Replication

This blog discusses how synergies between the Lakehouse and ActionIQ’s Composable CDP enable better customer experiences.

Modular Orchestration with Databricks Workflows

Thousands of Databricks customers use Databricks Workflows every day to orchestrate business critical workloads on the Databricks Lakehouse Platform.

Delta UniForm: a universal format for lakehouse interoperability

In this blog, we walk through the benefits that Delta UniForm provides and show examples of how to create and use UniForm tables.

Delta Live Tables Now Generally Available on Google Cloud

Today we are announcing the general availability of Delta Live Tables (DLT) on Google Cloud. DLT is now available on all three major cloud providers.

What’s new with Databricks SQL?

Databricks SQL delivers industry-leading performance for your EDA and BI workloads, while improving cost savings - with no manual tuning.

How Verana Health Uses the Databricks Lakehouse to Democratize Data and Deploy AI for Medical Innovation

By combining cutting-edge technology with deep expertise, Verana Health and Databricks are driving innovation and propelling the healthcare industry forward.

A New Partnership with Redox and How We Unlock Healthcare Data to Drive Advanced Analytics

Redox and Databricks are unlocking healthcare data, enabling BI to AI use cases across healthcare providers and payers. Learn more about the partnership in this blog.

How to execute your operating model for Data and AI

Having a platform that allows different teams with different roles to work together using the same tools, that facilitates the integration of work products is essential to achieve positive business results.

Multiple Stateful Operators in Structured Streaming

Structured Streaming in Apache SparkTM is the leading open source stream processing engine, optimized for large data volumes and low latency, and it is the core technology that makes the Databricks Lakehouse the best platform for streaming.

Protecting Your Compute Resources From Bitcoin Miners With a Data Lakehouse

MLflow can automatically retrain machine learning models when new data becomes available. This keeps the model up-to-date with the latest data and patterns of abuse.

Announcing new security controls and compliance certifications for Azure Databricks and AWS Databricks SQL Serverless

Databricks now provides HIPAA, PCI-DSS, and FedRAMP Moderate security and compliance controls on AWS Databricks SQL Serverless and Azure Databricks.

Announcing Databricks Belgrade Development Center

Databricks is expanding with a new development center in Belgrade, Serbia. World-class engineers can join our Belgrade founding team and play a pivotal role in the future trajectory of Databricks' product.

Databricks and Technology Partners: Personalized Medicine with a Tailored Approach

Databricks + Technology Partners unlocks all healthcare data, enabling business insights and long-term medical research.

Introducing Databricks Assistant, a context-aware AI assistant

We are excited to announce the public preview of Databricks Assistant, a context-aware AI assistant, that lets you query data through a conversational interface.

Best Practices and Guidance for Cloud Engineers to Deploy Databricks on AWS: Part 3

Learn the backbone of cloud automation in deploying Databricks on AWS. Discover 3 API endpoints & examples in CloudFormation and Terraform.

Managing Complex Propensity Scoring Scenarios with Databricks

Consumers increasingly expect to be engaged in a personalized manner but understanding customer preferences is challenging. Check out our Solution Accelerators for Propensity Scoring to build propensity scoring models on Databricks.

Patient Disease Risk Prediction with Lakehouse

Databricks helps healthcare deliver precision health through its Lakehouse Platform, integrating all data with real-time frequency and tooling for the full ML lifecycle.

Managing and Analyzing Game Data at Scale

Learn more about how Azure PlayFab and Databricks can help gaming companies effectively collect and analyze vast amounts of game data in real-time, enabling data-driven decisions that optimize player engagement and monetization.

Announcing the MLflow AI Gateway

MLflow AI Gateway exposes a unified LLM interface to clients and routes to LLM providers. The MLflow AI Gateway is an API gateway that centralizes credential management and other features to allow users to easily and securely experiment with different vendor SaaS LLMs in their projects.

Now Generally Available: All users can now establish a connection to Fivetran via Partner Connect

All users, with sufficient privileges, can now establish a connection to Fivetran via Partner Connect. Bring data into Databricks from hundreds of different sources.

The Improved Databricks Navigation is Enabled for Everyone

Databricks improved the navigation experience for everyone. We simplified the home page, added features to the sidebar, and expanded global search.

Go from Months to Hours with Databricks Marketplace for Retailers

Databricks Marketplace simplifies the challenge of rapidly extending the value of data for Retail and Consumer Goods, reducing time to value to hours and days.

The Executive’s Guide to Data, Analytics and AI Transformation, Part 7: Move to production and scale adoption

Strategies for Data Leaders to scale the new modern data platforms across their enterprise and establish metrics to measure adoption.

Estimating Customer Lifetime Value on the Lakehouse

It is crucial that retail and consumer goods organizations identify high-value customers, cultivate long-term relationships with them, and attract more customers of this caliber, while limiting their investments in customers from whom they are not likely to see a return.

Databricks and Posit announce new integrations, simplifying Lakehouse access for developers

Databricks and Posit are announcing a strategic partnership to provide R and Python developers with an unmatched experience when working with Databricks.

Databricks + MosaicML

Virtual Event Advantage Lakehouse: Fueling Innovation in Data and AI Lakehouse architecture is built for modern data and AI initiatives. Discover how it unifies data to speed up everything from ETL to SQL to AI.

Building your Generative AI apps with Meta's Llama 2 and Databricks

Today, Meta released

Never Miss a Beat: Announcing New Monitoring and Alerting capabilities in Databricks Workflows

Databricks Workflows includes a new real-time insights dashboard to see all your production job runs in one place, advanced and detailed task tracking for every workflow, and new alerting capabilities to help you catch issues before problems arise.

Simplified Analytics Engineering with Databricks and dbt Labs

No matter where your data teams like to work, dbt on the Databricks Lakehouse Platform is a foundationally great place to start. Together, dbt and Databricks help your data teams collaborate effectively, run simpler and cheaper data pipelines, and unify data governance.

LLMs in the lakehouse: a quantum leap forward for the public sector

A detailed look at opportunities and challenges around deploying LLMs within Public Sector organizations.

Announcing Public Preview of Volumes in Databricks Unity Catalog

Discover, govern and query any non-tabular data including text, image, audio, video, PDF, or XML files with Unity Catalog

Complete Personalization, Complete Control: The Composable CDP

To deliver personalized customer experiences, organizations are investing in the development and deployment of specialized Customer Data Platforms (CDPs). The key areas that comprise the modern CDP are…

The Executive’s Guide to Data, Analytics and AI Transformation, Part 6: Allocate, monitor and optimize costs

Strategies for Data Leaders to help monitor and optimize costs and increase efficiency of their data platform.

How Databricks Unity Catalog Helped Amgen Enable Data Governance at Enterprise Scale

Learn how Amgen benefits from Databricks Unity Catalog's governance capabilities in a highly regulated industry

How to Build a Credit Data Platform on the Databricks Lakehouse

The banking and payment industries are witnessing a surge in customer demands for a Databricks Lakehouse design. This design offers a credit data platform that provides a holistic and efficient solution to the credit decisioning process.

Introducing English as the New Programming Language for Apache Spark

Introducing the English SDK for Apache Spark - an innovative tool using English language input to streamline your development process, enhance code efficiency, and accelerate data insights.

Helping Enterprises Responsibly Deploy AI

Virtual Event Advantage Lakehouse: Fueling Innovation in Data and AI Lakehouse architecture is built for modern data and AI initiatives. Discover how it unifies data to speed up everything from ETL to SQL to AI.

Announcing Delta Lake 3.0 with New Universal Format and Liquid Clustering

We are excited to an

What is a MMM and why does it matter for marketers?

This blog dives into how the Databricks Lakehouse can help companies leverage media mix modeling (MMM) to make smarter investments in their marketing strategies.

Asian Employee Network: Celebrating the Expansive Asian Culture

The Asian Employee Network ERG celebrates its third year. AEN members share some of their favorite experiences.

Built on Databricks: Fueling Data and AI innovation in modern software products

Databricks invests in programs to help Built on Databricks customers and partners succeed. Learn why modern applications are building on the Lakehouse with examples built on customer innovation.

Powering Customer-Led Growth: Databricks Ventures Invests in Catalyst

Databricks Ventures has invested in Catalyst, an advanced platform that helps software companies accelerate growth by optimizing relationships with their existing customers.

Model Risk Management, a true accelerator to corporate AI

Explore EY and Databricks’ joint solution accelerator for Model Risk Management, bringing a more transparent approach to AI through automated documentation.

Driving a Large Language Model Revolution in Customer Service and Support

Want to build your

Announcing the Public Preview of Azure Databricks support for Azure confidential computing

We are excited to announce Azure Databricks support for Azure confidential computing (ACC), now in public preview!

The 2023 State of Data + AI: How Businesses Are Preparing for the New Age of AI

The historic surge o

How Databricks improved query performance by up to 2.2x by automatically optimizing file sizes

With Databricks, you don’t need to choose between ease-of-use and performance. The Lakehouse Platform simplifies your data architecture while improving performance.

Announcing the General Availability of Databricks SQL Serverless !

Databricks SQL Serverless is now Generally Available! It provides best performance with instant and elastic compute, lowers costs, and enables you to focus on delivering the most value to your business rather than managing infrastructure.

Track health and fitness goals with Apple Healthkit and Databricks

With the explosion of data volumes at our fingertips – it is easy to be overwhelmed. The Lakehouse paradigm makes this simple and easy by making all of the necessary capabilities available under one platform.

New debugging features for Databricks Notebooks with Variable Explorer

View your notebook's variable state, inspect and explore data frames in one click, and interactively debug Python code with pdb and the new Variable Explorer.

Databricks on GCP - A practitioners guide on data exfiltration protection.

Learn details of how you could set up a secure Databricks on GCP architecture to protect data exfiltration.

Accelerating Grid-Edge Analytics using COMTRADE Files with Apache Spark

Using a large collection of COMTRADE formatted files, we demonstrate how data may be accessed and processed at scale using Apache Spark running in the Databricks platform.

Latency goes subsecond in Apache Spark Structured Streaming

In this blog, we will focus on the improvements we have made around offset management to lower the inherent processing latency of Structured Streaming. These improvements primarily target operational use cases such as real time monitoring and alerting that are simple and stateless.

How Habu Integrates With Databricks to Protect Sensitive Data

Habu’s data clean room technology works seamlessly with Databricks infrastructure to enable privacy-preserving access to data across teams and partners.

How Lakehouse powers LLM for Customer Service Analytics in Insurance

Our goal with this solution accelerator is to demonstrate how easy it can be to achieve digitization capabilities by utilizing customer unstructured data and leveraging NLP, Transformers, and a Lakehouse platform.

Fleet optimization with CARTO & Databricks

Good route optimization can provide the company with a competitive advantage. By optimizing their distribution network, they can offer faster, more cost-effective and more reliable delivery than their competitors, which can help them increase market share.

Cluster Policy Onboarding Primer

Data democratization is the self-serve process of getting data into the hands of people that can add value to it without undue process bottlenecks and without expensive and embarrassing faux pas moments.

Introducing Databricks Fleet Clusters for AWS

Databricks Fleet clusters unlock the potential of Spot pricing without the hassle of manual instance selection by allowing Databricks to intelligently optimize and automate the process for you.

Actioning Customer Reviews at Scale with Databricks SQL AI Functions

We show you how you can apply Azure OpenAI's GPT models to unstructured data that is residing in your Databricks Lakehouse and end up with well-structured queryable data.

Using Structured Streaming with Delta Sharing in Unity Catalog

Sharing streaming data using structured streaming with Delta Sharing in Unity Catalog

Unifying Your Data Ecosystem with Delta Lake Integration

We will illustrate how to connect the Delta Lakehouse to other platforms in an enterprise environment, demonstrating the various ways that’re currently available and some that will be available in the future.

Unify Data and Business Teams with Sigma on Databricks

This blog explores how Sigma Computing and Databricks integrate through Databricks Partner Connect to create a seamless data analytics workflow.

The Hitchhiker's Guide to data privilege model and access control in Unity Catalog

Distill concepts within the Unity Catalog privilege model in a simple, digestible way, to support different access needs and patterns.

Strengthening the Lakehouse Governance Ecosystem: Databricks Ventures Invests in Immuta

Databricks announces an investment in Immuta, which is based on our shared success in delivering on some of the largest and most complex cloud data projects across almost every industry.

Empowered to Raise the Bar

Özge shares her experience as a Brickster and how she’s empowered to raise the bar!

Announcing Terraform Databricks modules

30+ reusable Terraform modules to provision your Databricks Lakehouse platform - look into the current list of available modules and examples and try them to deploy your Databricks infrastructure!

Understanding Caching in Databricks SQL: UI, Result, and Disk Caches

Databricks SQL has 3 types of caching - UI Cache, Result Cache (Local and Remote), and Disk Cache. Each caching layer improves query performance, minimizes cluster usage, and optimizes resource utilization for a seamless data warehouse experience.

Welcome Okera: Adopting an AI-centric approach to governance

Databricks has entered into a definitive agreement to acquire Okera, the world's first AI-centric data governance platform. Okera solves data privacy and governance challenges across the spectrum of data and AI. It simplifies data visibility and transparency, helping organizations understand their data, which is essential in the age of LLMs and to address concerns about their biases.

The Executive’s Guide to Data, Analytics and AI Transformation, Part 4: Democratize access to quality data with governance

Databricks Lakehouse platform provides a unified governance solution for all your data and AI assets, built-in data quality to streamline data curation and a rich collaborative environment for data teams to discover new insights.

Five Reasons to Build your Modern Data Stack on the Lakehouse with Databricks, dbt Labs and Fivetran

Databricks, dbt Cloud and Fivetran simplify your modern data stack with a unified approach that eliminates the data silos that separate and complicate data engineering, analytics, BI, data science and machine learning.

Unlocking the Power of Retail Media Networks: How Data-Driven Advertising is Changing the Retail Promotions Landscape

Building a Retail Media Network requires a significant investment in time, resources and expertise but can have massive benefits. The Databricks Lakehouse Platform can make it easier by enabling retailers to streamline and centralize the collection and management of all shopper data.

A data architecture pattern to maximize the value of the Lakehouse

Following the approach outlined in this blog, data scientists using Machine Learning and AI will have easy access to reliable data derived from the organization's business information model.

Databricks ❤️ Hugging Face

Generative AI has be

Announcing the General Availability of Predictive I/O for Reads

Predictive I/O gives you all the benefits of indexes and optimization services, but without the complexity and cost of maintaining them.

Announcing the Public Preview of Predictive I/O for Updates

Are your ETL pipelines or CDC ingestion jobs taking a long time to execute? Do you have updates spread across your data? Predictive I/O can now significantly speed up those queries!

Improving Public Sector Decision Making With Simple, Automated Record Linking

Announcing the release of Databricks Arc, a new solution accelerator for unsupervised data linking (aka entity resolution).

Processing data simultaneously from multiple streaming platforms using Delta Live Tables

With Databricks introducing new features into DLT regularly, it’s finding wide adoption among clients for ETL workloads. Try Delta Live Tables today.

Anomaly Detection to Prevent Energy Loss

Detecting energy loss in the utilities field is a difficult problem to scale both with the workforce as well as technically. This blog covers how to start with an unsupervised approach to build a training dataset and then moves into taking this labeled data to build a generic supervised model at scale.

YipitData leverages Databricks Unity Catalog to Scale Data Services

Learn about YipitData's migration from Hive metastore to Unity Catalog and benefits of implementing Unity Catalog as the centerpiece of Lakehouse architecture.

Three simple lessons that will catapult financial services into its golden age

Hear how some of the most innovative financial institutions are making huge strides in data, analytics and AI.

PyTorch on Databricks - Introducing the Spark PyTorch Distributor

In this article, we will go through the new library and how to use it. Databricks is also proud to contribute this back to the open source community.

Building Data Applications on the Lakehouse With the Databricks SQL Driver for Node.js

We are excited to announce the general availability of the Databricks SQL Driver for NodeJS. Node.js developers can now easily build data applications on the lakehouse in pure Javascript or TypeScript.

Building Data Applications on the Lakehouse With the Databricks SQL Driver for GO

We are excited to announce the general availability of the Databricks SQL Driver for GO. GO developers can now easily build data applications on the lakehouse in GO.

Launching a New Files Experience for the Databricks Workspace

You can now use and reference any file type in the Workspace without any additional setup or the introduction of source control. Files support allows Databricks users to store Python source code, reference data sets, or any other type of file content directly alongside their notebooks.

The Executive’s Guide to Data, Analytics and AI Transformation, Part 3: Build Successful Operating Models for Data Teams

How data leaders can organize their people and teams to maximize cohesion and effectiveness for data transformation.

Scale Vision Transformers (ViT) on the Databricks Lakehouse Platform with Spark NLP

A demonstration of how to scale out Vision Transformer (ViT) models from Hugging Face and deploy them in production-ready environments for accelerated and high-performance inference, showing how to scale a ViT model by 21x times by using Databricks, Nvidia, and Spark NLP.

Developing an Open Data Ecosystem with SAP

SAP Datasphere enables a business data fabric architecture that provides SAP data with the business context and data model views intact, serving as a semantic layer for SAP data definitions, relationships, and hierarchies.

Spark Connect Available in Apache Spark 3.4

As part of the recently released Apache SparkTM 3.4, Spark Connect is now generally available. This blog post walks through what Spark Connect is, how it works, and how to use it.

Use Databricks from anywhere with Databricks Connect “v2”

There was no easy way to remotely connect to Databricks from languages other than SQL. We’ve now made it really easy: Users simply embed the Databricks Connect library into their applications and connect to their Databricks Lakehouse!

Introducing MLflow 2.3: Enhanced with Native LLM Support and New Features

With over 13 million

Introducing AI Functions: Integrating Large Language Models with Databricks SQL

With all the incredi

Introducing Apache Spark™ 3.4 for Databricks Runtime 13.0

To further unify Spark, bring Spark to applications anywhere, increase productivity, simplify usage, and add new capabilities, Spark 3.4 introduces a range of new features, including…

How We Performed ETL on One Billion Records For Under $1 With Delta Live Tables

Today, Databricks se

How Collective Health uses Delta Live Tables and Structured Streaming for Data Integration

We cover a use case at Collective Health where our partners send us files at a given cadence. By leveraging the Databricks Lakehouse Platform and Delta Live Tables, we are able to ingest files incrementally and also visualize and validate the quality of incoming records.

Retail in the Age of Generative AI

What do we predict will be popular uses for generative AI? The list is as endless as the ChatGPT knowledge base, but here are some key categories we’re hearing or expect retailers to invest in generative AI for…

Free Dolly: Introducing the World's First Truly Open Instruction-Tuned LLM

Databricks announces Dolly 2.0, the first open source, instruction-following LLM fine-tuned on a human-generated instruction corpus licensed for commercial use. Learn how you can get started!

Synthetic Data for Better Machine Learning

You've likely tried

Introducing Gamehouse: A Lovelytics Game Analytics Brickbuilder Solution

Quickly stand up a 360-degree view of your game using Lovelytics’ game analytics Brickbuilder Solution.

Enabling the Customer Data Platform with Databricks ETL Support

Tackle large and complex data processing challenges with the Databricks platform to surface actionable insights to your Marketing team.

The Executive’s Guide to Data, Analytics and AI Transformation, Part 2: Identify and prioritize use cases

To accelerate the delivery of high-impact use cases, Databricks provides a library of industry-focused Solution Accelerators.

Announcing General Availability of Cluster Policies

We are excited to announce that cluster policies are now generally available. In this blog post we will describe why cluster policies are useful.

Databricks for GxP

Databricks supports GxP and has customers using Databricks with GxP-validated workloads. In this blog, we guide customers on how they can install, configure, and validate Databricks in GxP-compliant environments.

Visual Data modeling using erwin Data Modeler on the Databricks Lakehouse Platform

Learn more about a Dimensional Model in data warehouses and how to implement it within the Bronze/Silver/Gold layers of Databricks to get the best performance with the Databricks Lakehouse Platform.

Preview the New Workspace Browser

To simplify navigating in Databricks, we are releasing a new workspace browsing experience. The new Workspace Browser makes it easier for you to organize and browse content.

Exciting new updates coming to Workflows in April

Whether you are an experienced data engineer or just starting out with SQL query automation, these new features are intended to simplify your workflow, boost productivity, and help you achieve your goals more efficiently.

The Lakehouse for Manufacturing

The Lakehouse offers what data warehouses and clouds cannot, the ability to leverage all of your data to perform AI at scale with reduced TCO.

Saving Mothers with ML: How CareSource uses MLOps to Improve Healthcare in High-Risk Obstetrics

CareSource used a Databricks private preview product, MLOps Stacks, to accelerate their ML deployment process.

Claims Automation on Databricks Lakehouse

The Databricks Lakehouse paradigm streamlines the current architecture, paving the way for future organizational expansion while addressing FWA (Fraud, Waste and Abuse).

Pandas-Profiling Now Supports Apache Spark

With the addition of Spark DataFrames support, ydata-profiling opens the door for both data profiling at scale as a standalone package, and for seamless integration with platforms already leveraging Spark, such as Databricks.

Lakehouse Playhouse: Introducing Apache Spark for Kids

Databricks introduces the Scratch API for Apache Spark, providing kids everywhere with a linearly scalable solution for their most data-intensive classroom application.

Security best practices for the Databricks Lakehouse Platform

Your data security

Run SQL Queries on Databricks From Visual Studio Code

Learn more about the new Databricks driver for the SQLTools extension and how it allows you to run SQL queries from within the popular Visual Studio Code (VS Code) editor on the Databricks platform.

The Executive’s Guide to Data, Analytics and AI Transformation, Part 1: A blueprint for modernization

A tactical guide for CxOs to execute a comprehensive, modern data architecture transformation and strategy.

Building the Trusted Research Environment with Azure Databricks

Data-driven innovation in health and life sciences is providing more opportunities for improving health services than ever before. Solution accelerators like Azure TRE means that more organisations and researchers can take advantage of these offerings.

Security Operations on the Data Lakehouse: Hunters SOC Platform is now available for Databricks customers

Hunters is announcing the availability of its SOC Platform for Databricks customers. Customers will be able to attain an end-to-end, security operations platform on their own Databricks Lakehouse deployments.

Cost Effective and Secure Data Sharing: The Advantages of Leveraging Data Partitions for Sharing Large Datasets

Learn how Unity Catalog enables secure and cost effective data sharing of large data sets across clouds, regions and data platforms with data partitions.

Hello Dolly: Democratizing the magic of ChatGPT with open models

Introducing Dolly, a breakthrough in LLM from Databricks. Learn how Databricks open sourced the model and all its training code, enabling organizations to re-create Dolly at a minimal cost.

Announcing the General Availability of Private Link and CMK for Databricks on AWS

We are excited to an

Announcing the General Availability of Private Link and Customer Managed Keys for Azure Databricks

We are excited to announce that Private Link and using customer-managed keys (CMK) for encryption are now Generally Available (GA) for Azure Databricks!

Announcing General Availability of Databricks Unity Catalog on Google Cloud Platform

Learn how Unity Catalog provides a unified governance for data, analytics and AI on Google Cloud Platform.

Barracuda Networks uses ML on Databricks Lakehouse to prevent email phishing attacks at scale

Barracuda Networks uses machine learning to assess and identify malicious messages and protect their customers. Using ML on the Databricks Lakehouse Platform, Barracuda is able to block tens of thousands of malicious emails daily.

Using Real-Time Propensity Estimation to Drive Online Sales

We leverage publicly available datasets and make available detailed notebooks and coded assets that walk Data Scientists and Data Engineers through all the ends and outs of getting a solution off the ground.

Fine-Tuning Large Language Models with Hugging Face and DeepSpeed

Large language model

Production-Ready and Resilient Disaster Recovery for DLT Pipelines

Using the Terraform code, pipelines can be made to start up for failover and failback, and consumers may be redirected. With the support of our Disaster Recovery solution, we intend to increase platform availability for users' workloads.

Transparency, visibility, data: Optimizing the Manufacturing Supply Chain with a Semantic Lakehouse

By leveraging data, manufacturers can monitor, predict and respond to internal and external factors – including warehouse constraints and geopolitical disruption – which reduces risk and promotes agility.

Get started with new role-based onboarding trainings for Databricks Lakehouse Platform

To address the growing talent demand in the market to onboard to Databricks Lakehouse, we are now offering role-based onboarding training multiple times a day across key geographic regions.

Building the Lakehouse for Healthcare and Life Sciences - Processing DICOM images at scale with ease

The databricks.pixels solution accelerator is an easy way to kickstart DICOM image ingestion in the Lakehouse.

Enhancing the Amperity CDP with Personalized Product Recommendations

Leveraging Amperity’s Databricks Delta table destination feature, data can easily be pushed and pulled between the Amperity CDP and the Databricks platform.

Unlock the Power of Your Game Data and AI with Our Ultimate Guide

Learn more about how game data, analytics, and AI can help game studios better acquire, engage, and retain players.

Real-Time Insights: The Top Three Reasons Why Customers Love Data Streaming with Databricks

Real-time decisions made simple with the Databricks Lakehouse Platform.

Unsupervised Outlier Detection on Databricks

Learn how we are integrating the popular ML library - PyOD - with the best practices of the MLflow platform and taking advantage of the scaling that hyperopt provides.

Level up with Databricks at Game Developers Conference

Whether you’re an independent developer or a global AAA studio, discover how the Databricks Lakehouse can help you create more engaging gamer experiences driven by real-time data.

Distributed Data Governance and Isolated Environments with Unity Catalog

Learn the best practices to establish organizational boundaries and isolated environments for distributed data governance with Unity Catalog.

Finding a Data Platform that Can Do More, With Less

Azure Databricks SQL Serverless is now available in public preview and comes with Intelligent Workload Management, which uses AI to learn from the history of your workloads.

Data Vault Best practice & Implementation on the Lakehouse

Learn more about a Data Vault Model and how to implement it within the Bronze/Silver/Gold layer and how to get the best performance of a Data Vault Model with Databricks Lakehouse Platform.

The 3Ds of Migrating Teradata Workloads to the Databricks Lakehouse Platform

Learn more about Teradata, how it differs from the Databricks Lakehouse Platform, and the exact steps involved with migrating your data from Teradata to Databricks.

Best practices for cross-government data sharing

Government can share data for various reasons, such as to improve government operations’ efficiency, provide better services to the public, or support research and policy-making.

Empowering Business Growth in Asia Pacific With Data Governance: Learnings From Economist Impact Webinar

Data management and governance must be perceived as key components of business growth, and open-source platforms exist as essential tools that can enable the right balance of governance versus accessibility and use.

Accelerate your model development with the new MLflow Experiments UI

With the new and improved MLflow Experiments UI, it’s never been easier to develop high-quality models at scale and effortlessly identify the optimal models for production.

Best Practices for Realtime Feature Computation on Databricks

Examine the most effective architectures for providing real-time models with fresh and accurate data using Databricks Feature Store and MLflow.

Announcing New Partner Integrations in Databricks Partner Connect

We’re happy to share that we’ve added our first partners for both the Data Governance and Data Security categories in Partner Connect!

Databricks ❤️ IDEs

We are thrilled to announce an entirely new and powerful development experience for the Databricks Lakehouse Platform: the Visual Studio Code Extension for Databricks.

Announcing General Availability of orchestrating dbt Projects with Databricks Workflows

Learn more about the support for dbt in Databricks Workflows (now in GA) and how data teams can leverage the integration to collaboratively transform, test, and document data in Databricks SQL warehouses.

How To Migrate Your Oracle PL/SQL Code to Databricks Lakehouse Platform

Learn more about a Dimensional Mode and how to implement it within the Bronze/Silver/Gold layer and how to get the best performance of Dimensional Mode with Databricks Lakehouse Platform.

Databricks Expands Brickbuilder Solutions for Migrations in EMEA

Learn more about Databricks’ expanded Brickbuilder Solutions for migrations, developed in collaboration with key EMEA partners to help companies across the globe migrate to the Databricks Lakehouse Platform.

Getting started with NLP using Hugging Face transformers pipelines

Advances in Natural

Announcing Multi-Cloud support for Security Analysis Tool (SAT)

Learn how Security Analysis Tool (SAT) can help monitor the security health of your Databricks Lakehouse account workspaces over time.

Creating Health Plan Price Transparency in Coverage With the Lakehouse

Learn more about the Databricks Lakehouse for HLS and the new price transparency solution accelerator, including starter notebooks, making it easier to provide customers with visibility into their health care while complying with new regulatory reporting requirements.

Five Challenges CIOs Need to Overcome in the New Year

Learn the top five data challenges CIOs and CDOs should focus on tackling in 2023.

Design Patterns for Batch Processing in Financial Services

Learn how to design a batch processing architecture for financial services to support data transformation and plethora of data-driven use cases.

Using Abandoned Cropland for Carbon Sequestration: a Data-driven Approach

Learn more about the collaboration between Databricks and the Terrer Lab at MIT and the technical details of how their partnership helps to predict the estimated absorption capabilities of different locations across North America.

Announcing Support for Enhanced Fan-Out for Kinesis on Databricks

Learn more about stream processing on Databricks and how the new Kinesis connectors, which supports EFO, adds new functionality and makes stream processing faster, more consistent and easier to use.

Introducing Upgrades to Databricks Notebooks - New Editor, Python Formatting, and More

Learn more about the latest improvements to the Databricks Notebook and how they help data teams work faster and more efficiently.

A Gousto use case: how Databricks helps create personalized recipe recommendations for customers at scale

Gousto is the UK's best value recipe box, serving up more recipe choices and variety than anyone else in the market. The recipes are designed by professional chefs and include a diverse range of flavors and cuisines, allowing you to try new recipes without having to source exotic ingredients or spend time measuring portions. Each box contains pre-measured, fresh ingredients and easy-to-follow recipes so you can quickly prepare delicious, nourishing meals at home without any food waste.

Best Practices and Guidance for Cloud Engineers to Deploy Databricks on AWS: Part 2

In part two of the series, we explore the technical best practices for hardening a Databricks deployment on AWS by way of an imagined conversation between a Databricks solution architect and a cloud engineer.

Work With Large Monorepos With Sparse Checkout Support in Databricks Repos

Learn more about Sparse Checkout and how you can now use it with Databricks Repos to clone and work on a specific subset of your repositories’ directories.

Building a Life Sciences Knowledge Graph with a Data Lake

Learn more about how to build a life sciences knowledge graph leveraging the new Apache Spark-based library, Graphster, and the solution accelerator with included sample data, prebuilt code, and step-by-step instructions.

Loading a Data Warehouse Slowly Changing Dimension Type 2 Using Matillion on Databricks Lakehouse Platform

Learn more about modern, low-code approaches to ETL and how the combination of Databricks and the Matillion visual ELT platform make it easy to integrate data from any source into a Databricks SQL warehouse.

Bringing Models and Data Closer Together

Learn more about the new AutoML capability and how to quickly and easily use Databricks Feature Store data to improve model outcomes, allowing users to find more ways to grow their revenues or reduce their costs using ML.

Enabling Operational Analytics on the Databricks Lakehouse Platform With Census Reverse ETL

Learn more about Reverse ETL and how to use the combination of Census Reverse ETL and the Databricks Lakehouse to operationalize your data for greater insights, accessibility, and visibility across sales, marketing and ops.

New Built-in Functions for Databricks SQL

Learn more about SQL built-in functions, review a subset of recent examples designed and published by Databricks and see how they help to improve the data processing experience across a variety of common use cases.

What’s New With SQL User-Defined Functions

Learn more about SQL user-defined functions and how Databricks’ recent enhancements, such as support for Unity Catalog and dependency tracking, make them even more user-friendly and powerful.

Easy Ingestion to Lakehouse With COPY INTO

Learn more about ingesting files into the Databricks Lakehouse and how the simple, powerful SQL command COPY INTO allows you to perform batch file ingestion into Delta Lake from cloud object stores.

Better Data for Better Decisions in the Public Sector Through Entity Resolution - Part 1

Learn more about requirements of ‘good’ data quality, its importance to public sector decision-making and how the use of tools like Splink can help public data teams scale entity resolution for improved efficiency and outcomes.

Supercharging H3 for Geospatial Analytics

Learn more about the new Databricks Runtime (DBR) and SQL (DB SQL) releases supporting H3, including expressions, performance benchmarks, and multiple approaches for point-in-polygon spatial joins.

Databricks Power BI Connector Now Supports Native Query

Learn more about the Power BI connector from Databricks and how you can now use it to access data in Databricks SQL and build data models with increased flexibility and productivity.

The Impact of Data and AI on a Modern Business

Data explosion is happening across industries. Learn how data-driven organizations are leveraging lakehouse architecture to tackle the biggest data challenges and drive business initiatives.

Streaming in Production: Collected Best Practices, Part 2

A collection of best practices for using Apache Spark’s Structured Streaming in Production from the Databricks field streaming SME team.

Building Geospatial Data Products

Learn more about how geospatial data fits within the product paradigm, its potential as a valuable data assets, and how the Databricks Lakehouse Platform can be used to unlock that potential to solve some of the world’s most difficult problems.

Products We Think You Might Like: Generating Personalized Recommendations Using Matrix Factorization

Learn more about product recommendation systems, why matrix factorization can be the best approach to a broad range of use cases, and how to get started -- includes examples and link to starter notebook.

How Corning Built End-to-end ML on Databricks Lakehouse Platform

Corning has been one of the world’s leading innovators in materials science for nearly 200 years. These innovations include the first glass bulbs for Thomas Edison’s electric light, the first low-loss optical fiber, the cellular substrates that enable catalytic converters, and the first damage-resistant cover glass for mobile devices. At Corning, as we continue to push boundaries, we use disruptive technologies like machine learning to deliver better products and drive efficiencies.

Achieve Your Goals With Databricks Certifications

Learn more about Databricks Certifications and how its educational programs and role-based learning pathways can help individuals and organizations acquire the data + AI skills and knowledge needed to meet the challenges ahead.

Better Alerts: Announcing New Workflows Notifications

Learn more about Databricks Workflow and the new improvements and capabilities that allow greater team collaboration directly in the tools of their choice.

Available On-Demand: Databricks Media & Entertainment Symposium

Learn from industry experts and explore the exciting possibilities of data and AI in the media and entertainment industry.

Reuse Existing Workflows Through Terraform

Part 2 of the blog series on deploying Workflows through Terraform. In this post, we focus on how to convert inherited Workflows into Terraform IaC

Bending Retails’ Curve – Moving Beyond Possible With Tredence

Learn more about Tredence’s Atom.ai, a fully-baked AI solution for retail that powers the development and deployment of retail-centric AI applications 50% faster and is built on the Databricks Lakehouse Platform.

Why Startups Build on Databricks

In this day and age, startups must innovate with data for their products to stand out. Build your startup on Databricks Lakehouse Platform to stay ahead of the competition.

Databricks Named a Leader in 2022 Gartner® Magic Quadrant™ for Cloud Database Management Systems

Learn more about Databricks selection as a Leader in 2022 Gartner® Magic Quadrant for Cloud Database Management Systems.

How Databricks Powers Stantec's Flood Predictor Engine

Learn more about Stantec, its rapid flood estimation product Flood Predictor, and how the Databricks Lakehouse Platform enables its small team to deploy production workloads.

Build Reliable and Cost Effective Streaming Data Pipelines With Delta Live Tables’ Enhanced Autoscaling

Learn more about Enhanced Autoscaling for Delta Live Tables (DLT) and how it enables analysts and data engineers to quickly create production-ready streaming or batch data pipelines.

An Automated Guide to Distributed and Decentralized Management of Unity Catalog

Learn more about the support for Unity Catalog objects in the Databricks Terraform provider and how to manage a distributed governance pattern on the lakehouse effectively.

Why We Migrated From Apache Airflow to Databricks Workflows at YipitData

Learn more about YipitData and why it moved from Apache Airflow to Databricks Workflows to analyze petabytes of data that many of the world’s largest investment funds and corporations depend on.

ArcGIS GeoAnalytics Engine in Databricks

Learn more about the release of the first ArcGIS GeoAnalytics Engine, the new plugin for Apache Spark™ that extends data frames with very fast spatial processing and analytics, ready to run in Databricks.

Intermittent Demand Forecasting With Nixtla on Databricks

Learn about the intermittent demand forecasting solution from Nixtla and Databricks and how its advanced techniques demonstrate a new path forward for supply chain forecasting and modernization.

How Databricks IT Transforms Business with Product-Driven Approach

Hear from three IT product specialists about Databricks' product-driven approach to IT, implementing the same customer obsession in pursuit of optimal business processes, technology solutions and outcomes.

MLOps at Walgreens Boots Alliance With Databricks Lakehouse Platform

Learn more about the importance of MLOps and the MLOps accelerator co-developed by Walgreens Boots Alliance (WBA) and Databricks, how WBA automates and standarizes MLOps, and how you can replicate their success.

Databricks Workflows Through Terraform

In part 1 of this blog series on deploying Workflows through Terraform, learn how to create complex jobs / workflows from scratch in Databricks using Terraform Infrastructure-as-Code.

Memory Profiling in PySpark

Learn more about the new Memory Profiling feature in Databricks 12.0 and how it provides data teams with a simple way to profile and optimize PySpark UDF performance.

Near-real-time IoT Robust Anomaly Detection Framework

Learn more about anomaly detection and how to build a real-time robust outlier detection framework on Databricks for streaming big data, e.g., IoT time series monitoring at scale.

Accelerate Your Career With Data Engineer Learning Pathway Improvements

Learn about Databricks Academy and how it helps Databricks users master the latest and most relevant data, analytics, and AI skills with Databricks Academy to accelerate in their current role or expand to new opportunities.

Solution Accelerator: Multi-factory Overall Equipment Effectiveness (OEE) and KPI Monitoring

Learn more about the OEE and KPI monitoring solution from Databricks and how it enables organizations to accelerate their analytics workloads by automating the complexity of building and maintaining analytics pipelines through open architecture and formats.

Serving Up a Primer for Unity Catalog Onboarding

Just in time for the holidays, the data chefs at Databricks provide the perfect recipe for a successful Unity Catalog and Identify Federation Onboarding, complete with instructions and examples.

Introducing Ingestion Time Clustering with DBR 11.2

Learn more about the Ingestion Time Clustering feature from Databricks and how it enables natural clustering based on the time that data is ingested, providing great data skipping out of the box.

Introducing a Native, High-performance Integration With dbt Cloud

Learn more about dbt Cloud, its partnership with Databricks and the new added support for dbt-databricks, a purpose-built dbt adapter for Databricks.

Announcing Availability of MLflow 2.0

Learn more about MLflow 2.0 and how it incorporates extensive user feedback to simplify data science workflows and deliver innovative, first-class tools for MLOps.

The Values That Define Databricks’ Culture

Learn more about Databricks’cultural principles and how Bricksters apply them to their day-to-day lives at Databricks.

How Lakehouse Powers Data-driven Insurance While Reducing TCO

Learn about the emerging data and analytics trends in insurance and how the lakehouse paradigm helps organizations adapt to them, as well as the benefits, including decreased Total Cost of Ownership (TCO) derived.

Upskill on the Lakehouse and Build Community Through Databricks

Learn more about the professional resources available through the Databricks Academy and how they make it easy to learn, get accredited, and stay at the forefront of data & AI innovations like the Lakehouse driving the world forward.

Leveraging Data to Better Educate Our Youth

Learn more about Learn To Be, its partnership with Databricks, and how the non-profit organization leverages the Lakehouse to help it better serve its constituents with educational programs and opportunities.

Orchestrate the Lakehouse: Announcing Support for Refreshing SQL Queries, Dashboards and Alerts in Databricks Workflows

Learn more about the new SQL task type and how it lets you easily author, schedule, inspect, and operate workflows that refresh Databricks SQL queries, dashboards and alerts, to unleash the full power of the Lakehouse.

Building a Semantic Lakehouse With AtScale and Databricks

Learn more about AtScale’s Universal Semantic Layer and how it helps organizations get consistent access to all of their enterprise data while leveraging the full power of their Databricks Lakehouse Platform.

Scrappy Halloween Insights: How We Used Databricks and Chewy Data to Identify Trendy Pet Costumes

Learn why Databricks was named a Leader and how the lakehouse platform delivers on both your data warehousing and machine learning goals.

Build a Customer 360 Solution with Fivetran and Delta Live Tables

Learn more about Fivetran and Delta Live Tables and how you can use them to build a Customer 360 solution on the lakehouse, delivering data and insights that would typically take months of effort on legacy platforms.

FINOS’ Legend to integrate with Databricks Lakehouse, improving open data exchange across the financial services industry

Learn more about the integration between Goldman Sach’s Legend and the Databricks Lakehouse and how it better supports real-time analytics, BI, and powerful AI capabilities on all data types while mitigating regulatory risk.

There’s No AI in Pumpkin Pie, But There Should Be: Delivering Novel Application Experiences Using ML & AI

To show the simplicity of AI with Databricks, we decided to solve a top-of-mind challenge: How can we make the best pumpkin pie? This blog post walks through how we used the Databricks Serverless Real-Time Interface with powerful open source technologies to build an AI-driven search app.

Optimizing AWS S3 Access for Databricks

Learn more about the most common S3 networking access architectures and how to optimize them to cut your AWS cloud costs and simplify access to your data.

Databricks Ventures Invests in Matillion to Boost Data Productivity on the Lakehouse

Databricks Ventures announces a new investment in Matillion, a leader in cloud data productivity specializing in low-code/no-code ETL. Learn more!

Loading a Data Warehouse Data Model in Real Time with the Databricks Lakehouse

Learn more about a Dimensional Model and how to implement it within the Bronze/Silver/Gold layer and how to get the best performance of a Dimensional Model with Databricks Lakehouse Platform.

How to Seamlessly Upgrade Your Hive Metastore Objects to the Unity Catalog Metastore Using SYNC

Learn more about Databricks Unity Catalog and makes it easy to upgrade your Hive metastore tables, views to Unity Catalog with a simple command.

Monitoring Notebook Command Logs With Static Analysis Tools

Learn more about static analysis tools and how the introduction of Databricks verbose notebook audit logs allows us to monitor commands run by users and apply the detections we want in a scalable, automated fashion.

Announcing the Security Analysis Tool (SAT)

Learn how Security Analysis Tool (SAT) can help monitor the security health of your Databricks Lakehouse account workspaces over time.

Simplifying Data Governance Across the Lakehouse: Why We're Investing in Alation

Learn why Databricks was named a Leader and how the lakehouse platform delivers on both your data warehousing and machine learning goals.

Sunstone: Data-Powered Customer Success

Learn how the Databricks Lakehouse supports Sunstone use of data-powered tools, including self-service diagnostics, to help its customer teams better understand and serve its customers.

Welcome to the Haunted Lakehouse

At Databricks, Halloween is also a big season for our customers. In this blog post, learn how one of our Bricksters built the “Haunted Lakehouse” game, completely powered by open source standards and Databricks, so you can see the amazing possibilities that exist within a Lakehouse!

Simplify Access Policy Management With Privilege Inheritance in Unity Catalog

Learn more about privilege inheritance in Unity Catalog, where data administrators can easily manage access policies at scale and reduce operational overhead.

Databricks Financial Services Lakehouse for Quantitative Research

The Databricks Lakehouse takes the research to production-ready quantitative models lifecycle from months to minutes.

Solution Accelerator: Building Digital Twins for Operational Efficiency With Databricks

Learn more about digital twins and how this new Solution Accelerator (with notebooks included) allows any data scientist to quickly use Databricks AutoML to train a model for fault detection, then integrate with Azure Digital Twins while fulfilling MLOps best practices for data versioning, experiment tracking, and model management.

Databricks Expands Brickbuilder Solutions for Manufacturing

Get product updates, Apache Spark best-practices, use cases, and more from the Databricks team.

Databricks Workspace Administration – Best Practices for Account, Workspace and Metastore Admins

In this latest edition of the Admin Essentials series, we explore the admin considerations of managing a workspace, including how to set up policies and guardrails to future-proof onboarding, govern usage, and more.

Announcing General Availability of Delta Sharing

Today we are excited