One misconception is that retailers send clean data. This simply is not the case. The cleansing and validation of the data is critical. Invalid data will give you invalid results. Retailers often send incomplete data. They also send duplicate data. In addition, they tend to “recast” previously sent data. Thus, how does one distinguish between duplicate and recast data? How do you identify missing fields? How is it rectified and re-loaded? What are the business rules? How is it modeled for reporting that is specific to Point-of-Sale data? Can it be integrated with internal data such as shipments, forecasts, budgets, etc.? How do you identify and analyze promotions? How can you predict problems before they impact your bottom line? Do you have the time and knowledge to shift through all this data to find only the nuggets that contain actionable information?
Because of the complexity, most CPG manufacturers simply buy summarized data from third party data providers. But by having an enterprise demand signal management solution in house or in a hosted environment, manufacturers can get more timely information and often save money by purchasing only the data they aren’t already getting from the retailers.
Building a demand signal management solution is not an easy task. Most IT departments of large companies are accustomed to building a data warehouse from internal data such as orders, shipments and finance. But building a demand signal management solution for POS data is completely different, although the fundamental methodologies should be the same.
With POS, you are dealing with 3rd party data that you have little to no control over. It comes in by the bucket full. Some of its daily, some weekly and some monthly yielding various levels of granularity. A lot of data from many retailers and other data sources, in varying formats supporting multiple feature sets. How do you consolidate it all, automate the process and build it so that it is both scalable and flexible and in a way that you won’t outgrow it? How do you align granularity levels and account for unlike calendars?
The Data integration process of transferring data from source to target can be painfully time consuming, tedious, and expensive. Many developers want to hand-code much of this work through the use of SQL scripts and stored procedures. This process is time consuming, difficult to maintain and support, impossible to manage, error prone, and not optimized for performance. What if something needs to be changed? Is any of it documented?
A true Enterprise demand signal management solution handles the data integration, cleansing and management processes. A good demand signal management process flow should be easy to use, have database specific API tie-ins, make simple tasks easy and complex tasks possible, require little to no coding, take advantage of in-memory processing for maximum performance and provide a fast ROI for your POS project.
A true demand signal management solution has pre-defined processes set up to automate the extraction, transformation, cleansing, synchronization and management processes. Because every company is different, the demand signal management solution should be open and allow for customization and enhancements of these processes.
Time and time again we see vendors leave their customers completely dependent on them. If they want to add a new retailer, they need to license a new module. If they want to add a new data source, they have to license a new data integration tool and contract services. Since demand signal management is a “process” and not a “product” companies should have the option to make future enhancements and management decisions without being forced to go back to the vendor whenever they need changes.
You should have the control (if you choose) to run processes, stop them, view the log files, and even monitor the job in real-time. An enterprise solution should support many databases so as your data grows, you are not throwing away your old solution. It should also support many business intelligence tools and offer the option to have the solution hosted or behind your firewall.