Aporia and Databricks partner to enhance real-time monitoring of ML models

Machine learning (ML) observability platform Aporia today announced a strategic partnership with Databricks. According to the companies, the collaboration aims to empower customers who utilize Databricks’ lakehouse platform, AI capabilities and MLflow offerings by providing them with advanced monitoring features for their ML models.

Organizations can now monitor their ML models in real-time by deploying Aporia’s new ML observability platform directly on top of Databricks, eliminating the need for duplicating data from their lakehouse or any other data source.

Moreover, the integration with Databricks streamlines the monitoring process, according to the companies, allowing for the analysis of billions of predictions without the need for data sampling, making changes to production code or incurring hidden storage costs.

“This means monitoring billions of predictions, visualizing and explaining ML models in production becomes effortless,” Aporia CEO Liran Hason told VentureBeat. “Aporia supports all use cases and model types, providing flexibility for ML teams to tailor the platform to their specific needs.”

Real-time monitoring, customization

The new offering allows monitoring for anomalies such as drift, bias, degradation and data integrity issues and triggers live alerts to popular communication channels, ensuring timely notifications.

The platform also provides real-time customizable dashboards and metrics, enabling each ML stakeholder to prioritize their key areas of interest and translate data science metrics into tangible business impact.

This is crucial in industries including lending, hiring and healthcare, Hason said, and promotes a fair and transparent landscape in automated decisions.

“Organizations would now be able to manage all ML models under a single hub, regardless of deployment,” said Hason. “This centralization enhances collaboration, facilitates communication and streamlines model management, fostering continuous learning and efficient team workflows.”

Streamlining data monitoring with ML Observability

Organizations have traditionally encountered challenges when monitoring large volumes of data, often necessitating data duplication from their data lake to their monitoring platform. However, said Hason, this approach leads to inaccuracies, overlooked issues, drift, false positive alerts and difficulties in ensuring fairness and bias monitoring.

The new integration with Databricks addresses these pain points by allowing organizations to monitor all their ML models on Databricks swiftly, within minutes.

Additionally, the integration maximizes the benefits of existing database investments — even for use cases that involve processing extensive volumes of predictions, such as recommendation systems, search ranking models, fraud detection models and demand forecasting models.

“There is no need to duplicate data onto a separate monitoring environment,” Hason explained. “This ensures a single source of truth derived directly from your data lake, simplifying data management and accelerating insights-to-actions. The integration enhances the effectiveness of ML model monitoring and provides flexibility and freedom for organizations to leverage their existing ML and data infrastructure to its full potential.”

Numerous use cases

The company said the new ML observability platform will support many use cases, including enhancing recommendation systems through real-time performance monitoring.

Organizations can leverage Aporia to improve their search ranking algorithms, gaining insights into click-through rates and enhancing search results. In addition, Aporia’s real-time monitoring helps detect and prevent fraudulent activities, bolstering security and fostering customer trust.

Furthermore, the platform ensures accurate predictions in supply chain management and retail by monitoring demand forecasting models, enabling teams to optimize their response to deviations from a forecasted demand. The platform’s observability capabilities will also assist financial institutions in monitoring credit risk models, ensuring accurate and unbiased credit assessments while identifying potential biases.

The Databricks delta connector establishes a connection between Aporia and an organization’s Databricks delta lake, linking training and inference datasets to Aporia, Hason explained.

The platform distinguishes itself in monitoring large-scale data by effortlessly handling billions of predictions without resorting to data sampling, said Hason. This ensures a comprehensive and precise assessment of model performance, which is particularly beneficial for organizations grappling with substantial data volumes.

“No critical insights go unnoticed, guaranteeing thorough monitoring,” he added.

Unleashing the power of data for informed decision-making

Hason said that the partnership will assume a crucial role in propelling the wider adoption of observability in the AI and ML landscape, as it addresses existing demand and nurtures a deeper comprehension and acknowledgment of observability as a pivotal element in AI and ML.

He said that the combination of a robust observability platform and a scalable data platform offers a compelling proposition for organizations investing in AI and ML. The enterprises are developing a unified tool that enhances observability at scale, empowering organizations to make informed decisions and optimize their AI initiatives.

“The partnership is specifically designed to deliver a centralized, end-to-end, cost-effective solution, empowering organizations to make confident data-driven decisions,” added Hason.

Organizations can monitor all production data in minutes, ensuring a rapid time-to-value. This accelerated implementation quickly unlocks the benefits of the investment.

“These new functionalities can save organizations valuable resources that would otherwise be spent on troubleshooting and rectifying issues,” said Hason.