How to trust the data
Knowing who and what to trust, it seems, has increasingly been on all our minds lately. One outcome of an ever more complex world is that the need to trust and search for evidence has become an essential activity in our daily lives. And, knowing if you can trust the data in your business systems and reports can be equally challenging.
The need for data lineage
I first heard of the need for “data lineage” in the late nineties when the Business Intelligence (BI) market began to really take off. A user and customer posed the question: “How do I know that’s a real number?”
He was referring to variance numbers flashing red on his dashboard. “What I need to know”, he continued, “is how that number was generated”. He went on; “If it’s derived from a number of data points in a source system, where did they come from? What transformations and calculations has it been subjected to?”
We discussed what would be needed to increase his confidence in the integrity of the data being reported. As ever, being a demanding user, he said he wanted a right mouse click or just to be able to hover over the numbers in question.
Today, I know his needs can be met more easily with the rise of the more popular data governance and catalogue vendors such as Alation, Collibra and Solidatus
They are responding to an everyday reality that businesses must adhere to a constantly shifting regulatory landscape, objectives and priorities. Compliance relies entirely on accurate and timely reporting and most importantly, the ability to trust your data.
The role of metadata
On a personal level we are much more aware of the possibility of fake news and questionable advertising. The digital revolution has exploded data volumes in all directions, accelerating the need for even greater levels of assurance.
Tech Target neatly summed up where we are in their recent article: Seeking truth in crisis times shows importance of metadata (techtarget.com) which draws parallels between what’s required in media and business. In both sectors we are encouraged to focus on leveraging all the metadata we can.
In the media world it’s easy to understand why the date and time of a picture or the accurate location of a video provides greater confidence in its validity.
How should we provide the same kind of metadata that provides users with the ability to trust the data across the enterprise? Presented with this objective across the entire data landscape, there is one system category that presents a significant challenge that cannot be ignored: the Enterprise Resource Planning (ERP) system.
Why ERP’s are challenging
ERP’s have grown up from roots first planted firmly in the manufacturing sector. Here, the balancing act of matching manufacturing processes and inventory costs with demand volumes requires constant tuning.
Mainstream enterprise software makers like SAP, Oracle and Microsoft, latched onto the idea that business processes could be captured and automated across many sectors and all departments from Human Resources to Purchasing, Sales and Accounting. Today, you’re not a slick enterprise, can’t function or be competitive unless you continually strive to fully maintain your ERP systems.
ERP’s harbour rich data sources for just about every data intensive project imaginable. Why then, are such sources hard to leverage? We believe there are a number of key reasons:
- ERP applications weren’t originally designed to be truly open
- They’re comprised of large, complex customized and opaque data models
- They provide hardly any tools for data analysts to analyze, subset and share metadata
In addition, when we realise that a typical SAP implementation can include one hundred thousand or more tables, it’s no wonder analysts and data architects often struggle to know where to start. As a result, project delays and cost overruns are common. Thankfully there is an answer:
How to unlock ERP & CRM metadata
There are some categories of software that have been developed over time to be truly vendor agnostic. None more so than more general data “discovery” tools. Whilst there are many of these around, none that we can find, apart from our product Safyr, has been developed specifically to analyze, curate and share ERP & CRM Metadata.
The reason we built Safyr in the way that we have is that what’s needed is the ability to unlock metadata for data intensive projects. Not just to scan, lift and shift the metadata, but to connect sufficiently well to be able to access and interpret highly relevant business and technical sources of metadata and improve the ability to trust the data
The challenge and how we have been able to meet it was summed up well by Bloor Research when they first performed :
“Understanding complex ERP/CRM as a part of this process is non-trivial. Fortunately, Silwood’s aim has always been to hide the complexity of the underlying environment so that you don’t need detailed technical knowledge of your ERP/CRM deployment in order to reach this understanding”
Using Safyr for JD Edwards to help users trust the data
Having heard from my ERP customers over the years about their struggles to sufficiently “sweat the asset” of legacy applications, finally being able to self-service their need to discover, analyze and document their key systems has been a game changer. One of our recent customers summed things up nicely:
“We are currently going through an Enterprise Digital Transformation initiative and use JD Edwards as our ERP.
We have begun efforts to assess the current state of particular areas of our data and have used Safyr to help us understand the intended purpose of certain fields in the system and compare that to how they are being used or misused in the system. This will help us to identify areas of opportunity to better leverage the system as intended.
We plan to leverage the data models and pull those into our integrated data modeling tool once we begin to design a data warehouse in the upcoming year. We have had a very positive experience working with the Safyr team thus far and look forward to working with them more as we begin to further leverage their tool in the upcoming months.”
Bethany Bruns Director of Data Governance, Henny Penny
After all, trust begins with shining a light on all the available evidence does it not?