Transformation to a data driven business needs metadata

Introduction

The theme at Enterprise Data World 2016 is “transforming to a data driven business” and Silwood Technology are delighted to be sponsoring the event again this year.

This is a broad subject that covers a multitude of disciplines, for example, data governance, enterprise information management, master data, business intelligence and analytics as well as data architecture, data quality and more.

Data models underpin all computer software

I was musing on this topic and how it relates to us at Silwood in the context of our specialist area.

This is the enabling our customers and partners to accelerate project delivery by giving them access to the metadata which make up the data models which underpin their large, complex packages applications.

It occurred to me that one of the critical components for this transformation would need to be an understanding of the data models which underpin those systems and solutions which create or provide the data for the business.

Without this it is almost impossible to derive enough meaning from the data to deliver the necessary integration, analytics or governance.

 What is meant by ‘Data Driven Business’?

To find a deeper connection however, it seemed reasonable to try to understand what is meant by “Data Driven Business”. A quick internet search actually left me not much the wiser, mainly because the term seems to be used to fit the given circumstances. This is in contrast to it being a phrase which has a widely accepted meaning, against which the degree to which products, services and solutions fit are measured.

Perhaps a alternative term could be Data-driven decision management as per Techtarget’s definition. They suggest that this approach is taken to try to gain competitive advantage or a significant improvement in for example, productivity. They quote a study from the MIT Center for Digital Business which found “that organizations driven most by data-based decision making had 4% higher productivity rates and 6% higher profits”.

For me I think that in many cases, data has had the potential to drive businesses and organisations to the extent that it has been able to since businesses started recording transactions.

Will machines make decisions for us?

Whether in the past decisions have been taken on the back of what the data suggests of course is an entirely different matter and probably the topic of a whole new blog article.

Now however, there is a difference.

There is more data available from more sources, there are much smarter methods for accessing, moving, analysing and understanding it.

There are also ways of automating decision making based on the data through the use of algorithms, triggers and programming.

Perhaps ultimately, the definition of a truly data driven business will be one where there is little or no manual, human intervention at all!

So what about metadata?

Metadata represents the data foundation layer of an application, whether that is a large package such as one from SAP, an in house developed application or database, a social media feed such as Twitter, data coming from sensors and logs or even unstructured information.

In some cases the term metadata can also represent the data structures which are used by an integration, master data, data warehouse or other ‘intermediary’ platform.

To enable a data driven business it would make sense therefore to be able to access, understand and use the metadata, or underlying data models, in all source applications in order to make the task of moving, integrating, analysing, mastering and governing data easier to achieve.

Not having that information is like having cupboards or a fridge full of unlabelled food items and then trying to create a meal. It might be exciting, however the results are variable and to achieve something tasty is likely to take longer than necessary. Further it would also be almost impossible to repeat.

Finding the metadata from many of the sources typically found in an IT ecosystem can be relatively easy. There may be documentation or relevant data models available on the internet. There may be specialists who can find and understand it for you and there are data modelling, metadata repository  and other software tools which can deliver that information quite easily and quickly in a meaningful way. This would typically be the case for smaller application packages or in house developed systems and for external sources where the data structures are well known and publicised.

When it comes to large complex and often highly customised packages from vendors such as SAP, Oracle and increasingly Salesforce however these tools and techniques are simply not adequate.

There are several reasons for this. Firstly, the data models are extremely large. For example SAP Business Suite has over 90,000 tables, so even before tables are added and extended to reflect the customer’s needs it is impossible to navigate and find what you are looking for using the methods outlined above in any reasonable timeframe.

Secondly, for most packages the rich metadata about table and field descriptions, relationships, etc., is not stored in the database System Catalogue so even if a third party tool can reverse engineer it or connect at that level, there is very little there of any use to a data professional. The useful information is stored in the Data Dictionary tables.

Thirdly, there are no tools, except for Safyr, which address this problem. The main packaged application vendors for the most part ignore this pressing need. Philip Howard of Bloor Research discussed this in his paper on “Data and Applications”.

Information and Data Management software makers do refer to metadata, however it is normally the metadata which is at the heart of their own solution rather than the metadata outside the confines of their own system.

See us present at EDW 2016

Conclusion

If this blog has sparked some interest and you are attending EDW, why not come and say hello to us at Booth 509.

I am also presenting this topic in a talk there on Tuesday 19th April.

If you are not going to San Diego and would like more information please do take a look around our website.

Roland Bullivant
Silwood Technology