Business intelligence (BI) and data warehousing is about structured, high-quality data and publishing reports with irrefutable numbers. The technology is mature and proven: SQL, relational databases, OLAP and proprietary enterprise software. On the other hand, big data is about messy unstructured data; experimental data science and the big three Vs: volume, velocity and variety. The technology is rooted in the open source movement and it is still evolving: Hadoop, Spark, NoSQL, Python and R. At a glance, BI and big data look like opposite ends of the data management spectrum. However, in this post, I will argue that BI professionals are in the best position to take on the new challenges posed by big data projects.
Consider Lolë, a women’s apparel company based in Montreal. The company recently implemented an inventory system using RFID tags. Each article destined for a retail store is tagged when it is manufactured in China, resulting in drastic changes in supply chain and operational efficiencies. An inventory management process where orders are counted and then manually verified by scanning bar codes and counting, has been replaced with an automated process in which RFID scanners are placed in distribution centers and retail stores. As a result, Lolë staff are far more precise at managing inventory and faster at shipping the right merchandise to the intended store.
Lolë will use about two million RFID tags this year, creating a massive amount of data, which will be stored and used in innovative ways. For example, using RFID scanners in its stores, Lolë will track what items are tried on together and at what time of day. In addition, it can track what items are tried on, but not purchased. This RFID data will be combined with point-of-sale data about what actually gets purchased and what gets returned. Furthermore, store data will be combined and analyzed with website data to track purchase behaviors. Plus, discussions on social media sites will be tracked to help identify trends. This opens up the possibility of making marketing decisions based on actual data and consumer behavior, rather than guesses.
Machine algorithms will eventually be used to make suggestions about what should be discounted, the timing of promotions, and how to adjust prices to optimize inventory turnover. All of this is very exciting to a small apparel company trying to keep pace with larger and better-funded competitors.
The big data scenario above and all others pose challenges—and those challenges are best solved by BI professionals. Sure, big data is technically complex, but the hardware and software challenges are small compared to the business challenges. Any big data project will demand skill sets that are integral to the successful BI professional. You’ll want to hire BI professionals for big data analytics projects for these five reasons:
- They are expert at identifying relevant data to collect to meet business process goals. Data is collected and stored in a variety of ways; BI professionals gather relevant data from various locations, ensuring data quality and reducing technical roadblocks.
- After the data is collected, presenting it in a comprehensible format to the person or team making the decisions is essential. BI professionals create reports and dashboards that are simple enough to understand, yet comprehensive enough to clearly identify the opportunities present. They use the data to tell a story that resonates with decision makers.
- Making the data actionable and removing barriers to advise companies on how to take action is the role of BI professionals. Implementing changes, processes, marketing strategies and other projects identified from the data need to be well managed to not disrupt the day-to-day workflow. For example, few business processes can be impacted when changes are made and when they are made, they need to be done quickly. BI professionals can assist with limited downtime and help prioritize projects based on company operations, objectives and goals.
- Creating a feedback loop between data and operational activities helps with implementation and assists with making necessary changes. No matter what data-based system is implemented, experimentation has to take place to see results. BI professionals plays a critical role in building trust among team members to know it is okay to share concerns about the project and provide honest feedback.
- The benefits of using business intelligence have to be significant to justify data collection and the cost of storing and analyzing the data. BI professionals conduct cost/benefit analysis and figure return on investment to identify the information to collect, what system to use and how the data should be used.
Looking back at the Lolë case, the BI professional helped determine whether running promotions on its website to reduce store inventory was possible with the data it collected. The BI professional identified the best and worst case scenarios and the likely impact of each so Lolë could make sound decisions.
BI professionals have both project management experience and technology expertise to guide companies in meeting their organizational challenges. The elements I have learned for success in using big data is to start small, keep expectations realistic and continuously build on the successful projects.
Thanks to David Currie for contributing this article. David is a long-time business analytics consultant. He blogs about business intelligence and big data at davidpcurrie.com.
