Subscribe via E-mail

Your email:

Follow Me

Our Blog

Current Articles |  RSS Feed

Big Data Part 4


“Big Data” is about volume, but it’s more than that... Big data is also about Variety and a couple other characteristics that will be described in follow up blogs…

CPG companies are no stranger to variety. In addition to their own internal variety
of data residing in databases such as Access, Excel, Oracle, main frames, Teradata, DB2, Netezza so forth. You have multiple
applications such as trade promotion management applications, supply chain, manufacturing, planograms, CRM applications, forecasting and a slew of others.

In addition, the variety of data coming in from point of sale (POS) sources include retailer files that include EDI 852 files, EDI 867 files, AS2, flat files, other EDI files, retailer portal downloads and syndicated data from AC Nielsen, IRI, NPD and others. Most companies are also buying competitive market data, demographic information, surveys, weather trends, currency conversion information, and might even be trying to integrate emerging market data. In addition, you might have space information, displays and diagrams that are unstructured or semi-structured.

Those are all examples of various data sources that have existed over the years. Some of these sources are newer than others. But the newest variety of data is coming in via the web. These sources are coming from various applications that track your “Social Reputation,” clicks, and media presence to name a few.

Marketing teams also have ads, including print, on-line ads, tv commercials, radio spots. They might also have online targeted marketing on social media that include offers on web sites, mobile offers, YouTube videos, etc. All these sources are in different data formats containing different information. All of this adds up to a lot of variety.

Big data just got bigger with more variety from the internet. In these last two blogs we discussed volume and variety, but it's also about velocity and one other key characteristic that will be discussed in the next two blogs. Watch for our next blog, Big Data Part 5 on velocity.

Sign up for a Demo \u0026amp\u003B See How BlueSky Integration Studio Integrates Big Data

Data Marts and Data Warehouses and Big Data


Data marts and Data warehouses are often confused. Simply put, an enterprise data warehouse is the union of all marts. But that depends greatly on the underlying architecture.


A data mart can be stand alone reporting solution or it can be soundly integrated into an enterprise data warehouse.


Relational Solutions has been building enterprise data warehouses since the mid 90’s and pioneered the concept of an incremental, iterative approach.


This approach allows companies to get a fast ROI (return on investment) that will address
immediate needs of the business users. It will also provide a foundation that will allow you to get incremental benefits as new data is integrated. The design withstands the test of time and lets the data warehouse grow with your business and with the evolution of new data sources, including #bigdata.


Unfortunately, most data marts were designed as one off reporting solutions. When designed as
a stand alone, they are often referred to as a “stove pipe” or “silo” of informations.


Today, I hear some so-called, expert, CPG industry analysts use these terms as if this is some new concept. These are not new concepts or new terms. They are just new to these so-called experts. These "experts" are finally understanding what we’ve been preaching to them about the importance of architecture for years.


Data warehousing consultants have used these terms since the 90’s. They’re used to describe stand-alone reporting solutions. Typically these stand alone solutions are developed by individual teams or departments.

These groups develop “silo’s” or “stove pipe” reporting databases to achieve a specific goal that they were unable to get financial approval for. If they have a need for something that you can't get approval for, you resort to building something on your own. It happens in every company and every department.


That said, all data marts are not created equal. Some are in access, some in spreadsheets, some are in SQL Server or Oracle. Some are silo's and some are not. Data marts do not have to be silos. Designed correctly, a data mart can be integrated and should be fed from a single staging area where business rules are applied. Thus, a sound data warehouse is the union of all marts being fed by a single source.


Having an infrastructure that stages the data, cross references the data, cleanses it, harmonizes the data, and feeds it into a data model that then feeds subject specific marts
offers the best growth potential. The shared dimensions from one data mart to another provides consistency from department to department.


Relational Solutions are experts in data modeling and offer customized classes and consulting services in this area. Data modeling techniques vary depending on the database target. Data modeling is a big topic that involves too much description for this blog.


In short, designing the data model correctly allows business rules to be applied and data
to be accessed easily by the users. This design also maintains consistency from department to department. It also provides IT with a manageable solution that is designed to evolve over time to accommodate new data sources and new user requirements.

Companies who have a properly designed data warehouse can integrate internal data, outside data and even Big Data.


My next blog will start to explain big data and what makes various data sources different


Learn Mor4e about Relational Solutions Services.



All Posts