Giving business users a single view of data across operations was a challenge for optical and optoelectronics company Zeiss, a €12bn-a-year revenue business with products including contact lenses, and lenses for spectacles and for cameras.
Its other interests include semiconductor manufacturing, where its lenses are used in extreme ultraviolet lithography (EUV) machines, industrial quality control and research, and medical technology. Zeiss technology is used in applications from wind turbine quality control to brain surgery and blockbuster movies.
But with critical data held mostly in enterprise systems, demand from the business for more timely information put increased demands on the underlying enterprise resource planning (ERP) infrastructure.
Zeiss employs 46,000 people in 50 countries, and although it operates both on-premise and cloud systems, its operations are based around a “strong SAP core”, according to Javier Caycho, the company’s head of data foundation and integration.
Zeiss’ transactional data sits in its SAP ERP systems which, Caycho says, “was firmly on-premise”. This included SAP’s Business Warehouse as a single hub for SAP data across the Zeiss businesses, but this has been supplemented by an Azure data lake, as well as technology from Databricks.
According to Caycho, “tens of thousands of users consume data from our platforms”, a figure that has almost doubled in just two years. Zeiss’ data team replicates several hundred, business critical SAP tables in real time. And data integration has grown from a handful of use cases to real-time access to data across the business, but “without placing additional load on transactional systems”, explains Caycho.
This is a growing challenge, as enterprises want better access to data for real-time decision-making and, potentially, for AI.
“Operational systems are typically sized to accommodate their operational needs: their primary purpose for existence,” says Adam Ronthal, vice-president analyst covering data management and analytics at Garner.
“The increased demand for near real-time analytics on current data naturally increases the load on such systems. Enterprises need to protect the core operational SLAs [service-level agreements], and that either means increased capacity and expense, or different technologies, like in-memory databases, that can accommodate these requests more efficiently.”
Zeiss has largely taken the second approach.
Data migration
As a business, Zeiss migrated its underlying business warehouse databases to SAP Hana 10 years ago, which allowed the business to introduce real-time replication of ERP tables directly into Hana, using SAP Hana Smart Data Integration, with user access via SAP Analytics Cloud.
“This makes SAP data far more available and far easier to consume than hitting the critical ERP systems directly,” Caycho says.
Originally, a business warehouse fed the SAP Analytics Cloud and process data mining solution. These connected directly and consumed hundreds of tables in real time, Caycho recalls. The Azure Data Lake consumed “a comparable volume” from the business warehouse system, this time as files. However, the pressure started to show.
“The picture was a strong, on-premise SAP core feeing both our own reporting and a growing, wider ecosystem, part of it on-premise, part of it in the cloud. That’s exactly where the strain showed: this fan out of high volume, real-time consumption sat on top of the business warehouse, creating high consumption of system resources. And demand continued to grow,” says Caycho.
This called for a new approach, he added: “At Zeiss, data has grown in importance because the business increasingly treats it as a strategic asset rather than a byproduct of operations. In the past, data either lived inside individual systems and departments, or it was centralised but slow to reach.”
If someone wanted access to a new reporting system they would gave to request this through a central team, which took time. The business now faced pressure from two directions: rising demand demand for new data products and reports, and a data platform that was consuming more and more IT resources, and struggled to scale.
Forward with FeRDI?
Zeiss’ data team identified three challenges with its SAP BW system: growing demand, which required upgrades to the BW Hana database; high resource consumption by the Business Warehouse; and the increasing efforts needed to monitor performance across both the BW and the downstream systems it supports.
In addition, Zeiss was devoting time and resources to build the data pipelines into Azure, in ensuring data consistency and in maintaining snapshots of the source data.
To tackle all this, Zeiss decided to move in two directions. First, it built a a federated model, based on Azure, where business units owned their own data, but with a shared governance model. Second, it developed real-time data integration for the business’ SAP systems.
“We can serve live data direct from our transactional core, instead of every system querying those critical systems directly,” adds Caycho.
This led to FeRDI, for Federated Real-time Data Integration. The system takes real-time data from all Zeiss transactional systems and feeds that into SAP Hana Cloud, which, in turn, delivers data to SAP Datasphere, as the primary source for analytical systems.
FeRDI also delivers data to some transactional systems. Datasphere also takes Business Warehouse data and shares both sources to Azure Data Lake and a Databricks deployment, all without persistent storage.
The objectives for FeRDI, Caycho says, were initially fairly constrained: solving the existing system’s high resource consumption, by moving real-time data integration from the Business Warehouse and on to Hana Cloud and Datasphere. But, he adds, Zeiss quickly realised they had built something more than a replacement for the previous data hub.
“In effect, we now had a single view of Zeiss’ transactional world in Hana Cloud, with data from our different ERP [systems] in the same state as the source system,” he says. “That realisation is what expanded the scope.”
As FeRDI is cloud based, Zeiss was also able to meet new demands for data more quickly and more efficiently. “We could keep adding use cases without worrying about capacity,” Caycho says. “And because we hold the data in the same state as the source systems, we are we able to go beyond serving analytical consumers and start serving transactional systems too.”
New data views
This led to new ways of using data across the business that were not part of the original plan. Zeiss was able to create cross-source reporting, using SAP Analytics Cloud on top of Datasphere to combine and visualise data across different ERP systems.
The business also created real-time data accessibility as a shared service – this offers an alternative to building interfaces between transactional systems and has reduced the need to build redundant interfaces. By bringing enterprise data together in one place, Zeiss was able to analyse data quality. Both data quality and observability are benefits of FeRDI, even though they were not part of the project’s original scope.
“Even though Hana Cloud is very stable and we have deep experience in real-time replication, we didn’t want to take reliability for granted,” Caycho recalls. “So, we set out to build proper data observability to guarantee a dependable flow of data that is always available to our consumers.”
Zeiss used the classic observability pillars, Caycho says: data freshness and latency, volume, and lineage and schema drift. This allows the data team to pick up table changes as soon as they happen in the transactional systems, before they affect replicated data or downstream users.
Zeiss can now also measure data quality against business rules and detect anomalies in ERP data without needing to access the source ERP systems themselves. And the business can compare data tables for accuracy.
“It’s a good example of how real-time consolidation doesn’t just move data faster, it lets us actively check the consistency and integrity of the business itself,” Caycho says.
Business benefits
Zeiss has also seen wider benefits. The Hana Cloud channel has grown to 3.6TB of data, over 21 million tables. More than 6,000 people now either access the system directly or benefit from it. Zeiss estimates that the deployment is saving €2m a year.
Application developers and data engineers now have a single way to gain access to data without the need to interrogate individual systems or even write source code.
“Before, when a business user needed something, a good part of the work went into the plumbing: where do we get this data from, how often can we extract it without slowing down the source system, and how do we reconcile it once we have it. Now that work is already done,” Caycho says. “That shortens delivery times noticeably.”
And business users can view operational reporting from multiple ERP systems, in real time, rather than data that was updated overnight, he says, adding: “In operations, a few hours can matter.”
Other enterprises are seeing similar benefits. “Integrating all sources of data can provide a more complete picture of the data and analytics landscape,” says Gartner’s Ronthal. “There will be several natural emerging centres of data gravity. Enterprises need to understand the touch points and which sources of data need to be combined to deliver a holistic picture of the business.”
But investment here is also future proofing the business.
Soon, FeRDI will also support Zeiss’ work with AI. The company did not start FeRDI as an AI project, but its benefits – including data quality, observability and integration – are prerequisites for AI.
In the future, Caycho’s team plans to build foundational data products that can connect through SAP Joule to Microsoft 365 Copilot. Business users could then interrogate company data in natural language. “That’s the direction we’re working towards,” he says.
But there are already real benefits to business users, even without AI. “The clearest benefit is that people get answers faster, and they get to ask better questions,” says Caycho. “The data is there, in one place, and it’s up to date.”
Not all of Zeiss’ systems provide real-time data yet, although this is the goal. “But the direction is clear,” says Caycho. “Data has moved from a support function to the foundation the business relies on to make faster, better-informed decisions and increasingly, to power our AI solutions.”
He adds that it is now easier, cheaper and quicker to give users access to data from the business’ transactional systems.
Other enterprises have seen similar benefits from data integration, despite the time and effort needed. Areas, a catering business based in Spain, has used SAP Datasphere and SAP Analytics Cloud to reduce the time it takes to calculate its daily sales across the business. Meanwhile, London’s Heathrow Airport is using Salesforce’s Data Cloud to bring together previously siloed data, through a zero-copy architecture.
