Recently, we wrote about five reasons you might want to build custom ETL (extract, transform and load) from the ground up and not use off-the-shelf (OTS) tools. In this post we take the opposite side of the field and discuss the case for using OTS ETL tools.
Over the last two decades, organizations worldwide have been adopting data integration tools at an increasing rate, and for good reasons. Below, we list a few of the important ones and explain how even the smallest organizations can benefit.
Data integration tools have evolved beyond basic transformation
Remarkable new data integration technology has emerged to accommodate the shift to cloud architectures and big data. The architecture of data integration solutions is expanding as is the number and variety of options for storing and accessing information.
The role of data integration has evolved beyond the simple task of extract and load. Large organizations face large and complex challenges. It’s unlikely you will find a company of significant size without at least one OTS data integration tool, and often several OTS tools plus custom-built solutions.
These organizations may be leveraging OTS tool capabilities like hybrid integration, data governance integration, CI/CD, AI, advanced orchestration, high availability, on-demand scaling and more. The features that data integration teams are looking for are way beyond basic transformation features!
Today’s data integration tools. Too complex for your company?
Does this mean that OTS data integration tools are now so complex and weighty that small- to medium-sized organizations will be overwhelmed with the complexity? Have they been priced out of the market? Wouldn’t some custom SQL be a suitable solution for them?
We think the answer to these questions an emphatic “No!” The key is having the correct expectations and selecting the appropriate tool.
Small organizations now have many of the same options as large companies for how to implement their information systems. In 2021, Gartner estimated that nearly half of all organizations have at least one cloud-based data source. This approach calls for a hybrid data integration solution to deal with data that is on-prem and in the cloud.
The implication is that many small- and medium-sized organizations need to address the same issues as the larger ones, but with fewer employees to deal with them. Smaller teams usually have less time to implement best practices for testing, deployment and monitoring. That is, they have all the same problems but with less time to address them.
5 reasons for using an OTS data integration tool
Let’s look at the five key reasons you should use an off-the-shelf data integration tool to address these problems.
- You’re a smaller organization with less mature development processes and will benefit from the productivity and workflow structure that OTS tools provide.
- Your workload varies based on business requirements and your development team is small, so using an OTS tool allows you to scale up with outside help as demand increases.
- You want to benefit from the well-developed workflow, orchestration, CI/CD or other integrations (like data governance), reliability and monitoring of processes that an OTS tool can provide.
- You need to support a variety of data sources, on-prem and off, and desire to do so using documented and supported connectors. Selecting an OTS tool that supports a flexible data fabric is desired.
- You don’t want your entire data integration process to exist as SQL code that only a few key people can read and maintain. OTS tools allow the visual examination of pipelines and often provide automated documentation.
These are only five reasons. Likely there are quite a few others that apply to your specific situation. We believe there are many, many more reasons to use an OTS data integration tool than to write custom ETL.
While the choice of which ETL tool to use may initially seem difficult, it can usually be simplified by focusing on a few aspects unique to your business. For example, looking at factors like the kinds of data sources you will be integrating, the need for real-time data, or the industry you operate in will help you narrow the field. The choice of high-quality tools is larger than ever and there is a solution for every team and every budget.
Next time, we will discuss the emerging promise of “ETL-less” data integration.
Are you considering how best to integrate your data? Are you stymied by which of the many off-the-shelf solutions is best suited to your needs? We can help narrow the field of choices and select the best tool and approach for your business.
