Subscribe via E-mail

Your email:

Follow Me

Our Blog

Current Articles |  RSS Feed

Before Big Data


Before Big data we had mainframes, ERP systems and data warehouses (and by the way, we still do). You could make the claim that big data started in the 1950’s with IBM’s “Big Iron” and “Big Data Processing” to handle mixed work-loads.

Or you could say big data started when Oracle coined the acronym, VLDB back in the 90’s to describe "very large databases." I can't decide which is bigger, “very large” or “big.”

Or was it Teradata, who back in 1992, built the first system over 1 terabyte for Walmart? This was the biggest implementation for its time. Teradata called their platform "massively parallel processing" or "MPP." I don’t know about you, but I definitely think “massive” sounds a lot bigger than “big.”  

Big, large, massive…regardless of the adjective used, database companies have been in the “big data” business for years. But “big data” today is NOT the same as “big data” from ten years ago. Today, there has been a full blow explosion of data.

One of my customers recently said “If you ask 50 people what “Big Data” is, you’ll get 50
answers.” My goal in this blog is to first explain the evolution of big data and in a series of blogs and help clarify some of the confusion.

Big Data encompasses many areas, starting with internal data in both transactional and analytical systems.

In transactional systems, the data is constantly changing and being updated. In analytical
systems, the data warehouse is typically updated once a day or sometimes more frequently. In analytical systems such as the enterprise data warehouse, companies are analyzing data, attempting to learn more about their customer, their buying patterns, their behaviors and how to best market to them.

Simply put, analytical systems are designed to help “manage” your business.
Transactional systems are designed to “run” your business.

The big data explosion started with data from applications designed to run your business. Mainframes were first on the scene. ERP (Enterprise Resource Planning) applications, really took off from the 80's & 90’s. Companies like SAP, Oracle, Microsoft, Sas, Infor, JDA & JDE all offer ERP solutions. They include applications for manufacturing, logistics, invoicing, order placement, call centers, etc. These applications also have reports associated with each of
their applications.

In the early 90’s companies started getting serious about using the data to improve knowledge, business processes and profits. All the buzz words back them evolved from Decision Support Systems (DSS) to Executive Information Systems (EIS) to data warehousing and business intelligence. Unfortunately, I’m old enough to remember these things.

Over the past ten years, we started seeing cooperation and data sharing between partners. I once felt like a missionary in the consumer goods space back in the 90’s trying to explain the value of sharing point of sale data. Retailers thought I was crazy and manufacturers said it will never happen. But today, they finally get it. Today they understand the value. Some more than others, but it’s finally caught on.

As more and more outside partners and vendors began offering new data and insights, we were
able to leverage that data through an architecture that allows new data sources to be integrated within a company’s existing data warehouse. That’s why Relational Solutions always stress the importance of a solid infrastructure.  

Some of these outside data sources include point of sale data, EDI files, syndicated data from companies like IRI, Nielsen and NPD, panel data, demographic data, currency conversions, weather trends and other sources.

New data sources are being shared every including loyalty data and emerging market data. Wholesalers, distributors, brokers and other selling partners are also starting to share data (not just reports).

Most companies just bought reports in the past. They didn’t understand the full value of having an infrastructure in place to leverage all data including future data, but that is starting to change. In the past, business users who had budget, would just go out and buy reports. Infrastructure didn’t matter to them and they didn't understand it. But as the market evolves, companies and people are maturing in their understanding of how important it is to have a big data infrastructure and more and more we see IT involved in those decisions.

Now, the latest evolution of “Big Data.” Combined, it is all big data, but in the pure sense of how software companies refer to Big Data today, they are mainly talking about unstructured data on the web.

See my next blog, "Big Data Part 1" as a follow up to this blog, “Before Big Data.” 



Currently, there are no comments. Be the first to post one!
Post Comment
Website (optional)

Allowed tags: <a> link, <b> bold, <i> italics