Skip to main content

How Microsoft and Databricks crafted a unique partnership for AI data processing

Microsoft Azure
The Microsoft Azure logo.
Image Credit: Microsoft

Join us in Atlanta on April 10th and explore the landscape of security workforce. We will explore the vision, benefits, and use cases of AI for security teams. Request an invite here.


Microsoft is bringing its Azure Databricks cloud service out of beta today to help its customers better process massive amounts of data, powered by a partnership unlike anything the tech titan has done before.

The company worked with Databricks to produce the service, which is an analytics system based on the popular Apache Spark open source project. Customers use it to ingest and process a large amount of data using machine learning techniques. The Azure Databricks service is supposed to give them an easier experience of running big data jobs than rolling their own Spark deployment, and offers a deeper level of compatibility with Microsoft’s first-party offerings than other services offered through the Azure Marketplace.

This release is a move to capitalize on enterprises’ interest in using more data to power artificial intelligence systems. The companies started working together two years ago, in response to their shared customers’ interest in a version of Databricks that runs on Azure. It entered beta last November as part of a suite of product updates.

“The discussions initially started like any other third-party partnership, but then we realized that if we really, really want to make these companies successful with big data and AI, this really has to be integrated with all the other services,” Databricks cofounder and CEO Ali Ghodsi told VentureBeat. “So, a product like Databricks, if it’s not natively really, really integrated with something like Stream Analytics, or something like Azure SQL Data Warehouse, or something like Azure Cosmos DB, or if it’s not really integrated all the way with Azure Active Directory, those enterprises won’t get the value that they need, and it won’t actually simplify the things that they want to build.”

VB Event

The AI Impact Tour – Atlanta

Continuing our tour, we’re headed to Atlanta for the AI Impact Tour stop on April 10th. This exclusive, invite-only event, in partnership with Microsoft, will feature discussions on how generative AI is transforming the security workforce. Space is limited, so request an invite today.
Request an invite

Using Azure Databricks, customers can take in data ingested through other services, prepare it, and process it using machine learning algorithms and other techniques. After that, it can be funneled out to other services like Cosmos DB and Power BI.

Making a deep integration possible required a great deal of work on the part of both firms, however. Company representatives made many trips back and forth between Databricks’ office in San Francisco and Microsoft’s in Redmond. The partnership wasn’t without its challenges on either end, but both companies were committed to it for the sake of their joint customers.

Andreessen Horowitz cofounder Ben Horowitz, who sits on Databricks’ board, has a close relationship with Microsoft CEO Satya Nadella and helped facilitate the two companies’ collaboration.

Microsoft had to work through concerns about what it would mean for the company to deeply integrate Databricks with Azure systems, including those for incident management, handling support requests, and other functions. What would it mean for Microsoft to offer its own guarantees about the functionality of another company’s software on its cloud?

Databricks, meanwhile, had to spend time getting its product’s compliance features in line with what Microsoft needed.

“It’s been great, but it’s meant that we had to button up in advance, and invest in these things that we would have maybe invested in anyway a few years down the line, but we had to do those investments up front much more and collaborate with the Microsoft team to do this,” Ghodsi said.

Doing that work offered dividends for Databricks’ non-Azure business, however. The company had to do a lot of work to prepare for the European Union’s General Data Protection Regulation (GDPR) as part of its partnership with Microsoft. That accrued to the company’s software more broadly as well.

Rohan Kumar, a Microsoft corporate vice president for Azure Data, told VentureBeat that this is one of the fastest transitions he’s seen for a service going from preview to general availability, largely due to how robust Azure Databricks was when the companies originally released it.

Looking towards the future, Ghodsi said this experience changed his view on what cloud partnerships could be, given how well customers responded to a service this deeply integrated with Microsoft’s other offerings.

VB Daily - get the latest in your inbox

Thanks for subscribing. Check out more VB newsletters here.

An error occured.