The initial IT needs that led to the development and use of Extract, Transformation and Load (ETL) tools and technologies are at an inflection point. On the one hand, the future of IT departments and CIOs’ careers depends on providing a greater range of integration options that support diverse source and target data structures comprised of more complex data elements. Bulk, batch-based workflows are giving way to real-time integration needed to support customers’ ever-increasing expectations of real-time responses to their queries and requests.
On the other hand, the stability and consistency ETL once offered is now a liability to any business attempting to deliver information fast enough to stay in step with their customers. Known for being deceptively simple to set up yet very high maintenance to maintain, ETL tools and its supporting technologies are slowing companies down from excelling at serving customers. Overcoming these challenges can accelerate any IT Team’s speed of response and quality of data delivered.
How IT Teams and CIOs Are Overcoming ETL’s Challenges
ETL’s legacy of enabling integration options includes more than its fair share of challenges too. From its steep learning curve, source and target data modeling constraints, problems with data complexity, and dependence on log files, ETL tools aren’t scaling to the quickly-emerging integration needs of many CIOs. Presented below are several strategies CIOs and their IT teams are using to overcome ETL’s challenges and excel in attaining reliable, high-performance integration strategies:
- Shifting workloads to more flexible integration architectures to break free of rigid ETL routines that are time-consuming to modify and use. The CIO of one of the leading financial services providers for commercial real estate loans told me the proliferation of new legal and regulatory compliance requirements makes it a certainty his firm will move beyond ETL in 2017. He commented that the level of reporting required in the new nations the company is opening offices in makes ETL impractical given how complex the data structures are supporting legal and regulatory Purpose-optimized integration is the focus of his firm going forward.
- CIOs are piloting newer tools and technologies that provide the option of real-time integration and federation to supplant their base of ETL tools. Another CIO of a local manufacturing company which produces electronic replacement components for airlines globally says that many of their transactions with suppliers and buyers are fine staying in batch mode. The challenge is configuring ETL to support real-time integration with their new pricing and supplier management systems. “We’re in a business that is dependent on price and availability, and before ETL was working out OK. But, now we have competitors globally that can deliver real-time quotes and we have to step up our response times to win more business.”
- ETL upgrades look deceptively easy yet turn into incredibly complex challenges, leading more IT teams to evaluate purpose-optimized integration approaches. This insight comes from a software engineer who has done several ETL upgrades at one of the world’s leading global distributors of computer hardware, software and services. Defining transformations when data structures have nearly all the same elements is quick for an experienced software engineer to do. She said that ETLs’ advantages end there. Whenever there is any change to application logic, data sources, or data structures the engineer says her and her team resort to ad-hoc scripting to get the ETL system to work. She says it’s easy to set up when everything is known, but the minute data changes or there are new requirements, ETL becomes extremely high maintenance. As a result, this distributor is looking at purpose-optimized approaches in integration to support its traditional pick, pack and ship business.
- The database load phase of many ETL workflows often becomes slow and difficult to troubleshoot, forcing IT teams to optimize hardware while searching for a more efficient approach. The more complex the business process and supporting data structures, the more challenging it is to get fast ETL performance. Instead of investing in more hardware, CIOs and their teams are choosing to evaluate lower cost integration frameworks and apps that are less expensive and lower maintenance. They’re getting greater integration performance and the same stability and consistency they achieved with ETL.