What Is ‘Real-time Data Warehousing’?

I get asked the following questions often:

What is real-time data warehousing?

Is there such a thing as real-time?

Who defines ‘real-time’ anyway?

Is real-time really just ‘near-time’?

It comes down to this concept: the ability to have valuable, quality data in your data warehouse as quickly as that data is generated within your operational system can be very powerful and beneficial to your organization.  This concurrence is the crux of real-time data warehousing: getting production data into your data warehouse as it’s created.  Easy to say, harder to do. But, since we are neck deep in the digital age, everything is possible. Imagine what it would mean to have these capabilities in a physical warehouse… It’s like having scissor lifts that can store and deliver items being created in another country, in real time. So, in a way we are lucky to have this digital problem to solve.

There is an excellent whitepaper at The Data Warehousing Institute which discusses in detail real-time data warehousing, data integration, data federation and data virtualization.  It does a nice job of covering some options and provides some key thoughts to consider if your organization is working its way toward real-time data warehousing.  I highly recommend it.

In addition to the insights provided in this whitepaper, I would also propose that real-time data warehousing can be that final step which allows an organization to achieve true BI excellence. In my experience, it’s one of the last hurdles a company has to jump.

While many organizations have a data warehouse, most of them still do not get 100% of their decision-making data from their data warehouses. The lack of real-time data is one of the key reasons that is true – the data warehouses are incomplete, even if just by a few hours of data age.

For example, if you run a high-turn inventory or manufacturing system, you cannot have inventory levels for either raw materials or finished goods which is hours old. For that reason, many organizations still will go to the transaction system for some of its time-sensitive inventory data. That’s the data freshness needed to make real decisions.  If it’s hours old in the data warehouse, it’s nice and all, but it’s academic.  Nobody trusts it for heavy-duty decisions.

If a company is swapping back and forth between the data warehouse and transactional systems for information, it is possible (certain?) to run into training and tools issues. Training questions arise such as when should I use the data warehouse vs. the operational system?  Why? If I use the operational data, do I use the native query and reporting tools or do I use the enterprise BI tools? Does this organization have to now support multiple tools?  Which ones?  Is it faster and easier to use the native tools instead of the enterprise BI tools? Will data conflicts between data warehouse data and operational data arise?

Moving to real-time data warehousing and providing a source for all data needs can alleviate these issues organizations face. Truly utilizing one enterprise BI tool, training and supporting that one tool can be a tremendous advantage. It is a challenge to accomplish, but the article at TDWI does provide some good insight to get moving in the right direction.  We have also helped clients achieve that ‘last-mile’ of data warehouse efficiency so that the DW can be relied upon as the platform-of-record for all decisions, mundane and major alike.

If you are working towards a real-time data warehouse but are struggling to close the loop on the real-time part of the equation, we can help.  Please email me and I’ll be happy to get in touch with you.

Any other questions?  Please ask away in the comments.

###

MIPRO Consulting is a nationally-recognized consulting firm specializing in PeopleSoft Enterprise (particularly Enterprise Asset Management) and Business Intelligence. You’re reading MIPRO Unfiltered, its blog. If you’d like to contact MIPRO, email is a great place to start, or you can easily jump over to its main website. If you’d like to see what MIPRO offers via Twitter or Facebook, we’d love to have you.

More business intelligence posts.