Best Practices in Data Warehouse Modernization


Lighted Ways Tech
Shop Your Best Moments here. The easiest way to find your things!- CHECK EVERYTHING ON AMAZON

Importance Of DW Modernization | hashedin.com

DW modernization is truly happening. 

Irrespective of what strategy is in play in your DW modernization approach, the real truth is that it all requires substantial modifications to the logical and systems architectures of the extended data warehouse environment and there are many commercial and open-source options for that.


Existing enterprise data warehouses are now bursting at the seam choked by the weightiness of new structured and unstructured data sources. As with most data-driven platforms, inadequate organizational support limit Data Warehouse Modernization. Success in modernization depends on the team, which may suffer if technical deficiencies due to inadequate staffing for data warehousing and its related disciplines, insufficient proficiencies on new technologies and procedures, or the lack of experience with big new data types and analytics.

Funding for modernization could be restrained by the cost of implementing the latest technologies and hardware and software upgrades. Modernization could be threatened by the poor quality of data or metadata whether it is focused on big new data, traditional enterprise data, or both. Utilizing new architectures to a solution existing in the system necessitates a generous amount of retrofitting when the current DW architecture was designed for standard reports and OLAP only. Similarly, shifting to the complex, multi-platform system architectures atypical of modern DWs could be stymied by the complexity of architecting a modern, complicated environment and the intricacy in the management of a multi-platform DW environment.

Data warehouse platform limitations under a current warehouse can be a sizeable obstacle when the existing DW environment cannot scale up to big data or take in data fast enough to leverage big volumes or streaming data. Moreover, modernizing the ecosystem all around a warehouse needs the purchase or upgrading of many tool types otherwise, the resulting modernization is restricted by a deficiency of tools for purposes of analyzing big new data types or for integrating and management of big new data types.

Traditional IT Transformation
Traditional IT Transformation | trivadis-training.com

Best DW Modernization Practices

We continually noticed that DW modernization can assume a lot of forms at varying scopes. The possibilities boil down to the most common categories, to wit.

System modernization –

This system modernization includes upgrades and areas for hardware and software servers or tools, at one end, and at the other end, organizations are adding up fresh data platforms and analytics means to their extended data warehouse environments to accommodate the latest data types, huge volumes of data, and workloads processing.

Arbitrary modernization-

On one hand, it is helpful that many modernization tasks are centered on the business requirements of a particular project, thus attaining the level of data warehouse-to-business alignment. On the other hand, interviewed DW professionals have taken notice that random tasks tend to pop up erratically in which some demand a prompt reaction whereas other tasks can be doubled into the usual cycles for an uninterrupted modernization process.

Modernizations of non-DW –

Surveyed and interviewed users keep on talking about how the warehouse hardly ever totally gets modernized. It is because DW is frequently modernized to better collect and provide data for fresh or evolving business processes, integrated data solutions, reports (dashboard styles), and analytics (advanced forms such as data mining, graphs, and statistics).

Optimized modernization –

Significantly, this is an important productivity concern for the reason that an average DW professional often spends up to 30 percent of his or her time on performance fine-tuning and other similar optimizations. In one scenario, modern tools from both vendor and open-source communities have turned into an expert at automated optimization, particularly for SQL-based queries. Conversely, the emergent figure of stand-alone platforms in users’ extended DWEs has pushed up the amount and intricacy of cross-platform queries, which are not accordingly easily optimized.

Continuity in modernization –

The concern here is to nurture uninterrupted enhancement in a well-thought-out approach that guarantees respectable benchmarks and quality – and sanity. Numerous successful DW professionals have spoken well on exactly how they stick to a regular, trimestral cycle for applying necessary updates to the main DW. This one is comparable to the regulated release cycles as seen in software companies and open-source incubation projects, but more so often it is in line with the urgent requirements of the enterprise. Surely, the trimestral cycles work fine with small-to-midsize tasks, from fine-tuning current data models (performance or extending a customer view) to rolling out a new area (employees or locations). On the other hand, the trimestral cycle can also be applied to all-inclusive modernization tasks, for example, the implementation of a new tool or platform or rolling over a group of users from one tool or platform to another.

Disruptive modernization –

As one respondent aptly put it, “we have started a new BI program that will rebuild the entire DW on a new platform.” A modernization project can be dramatic, upsetting, and disruptive to users mainly when it involves the rip-and-replace approach of key data sets, platforms, or tools. Business managers will have to be involved in formulating and following the modernization plan. Modernization must be carefully planned as a multiphase and multiyear project. At this level of modernization, as mentioned above, the modernization aspect includes not only the DW but the entire technology stack, and the modernization of that scale will also be accompanied by changes to business processes, and sometimes even the whole business model. Thus, the multi-phase plan is not just for the warehouse or the technology it encompasses, but the plan must also outline how the business users and processes will migrate and roll over as the modernization move forwards. 

Strategies in Modernization

Data is captured and managed on the data warehouse platform, which naturally consists of a DBMS (Database Management System, or an equivalent, such as Hadoop), an operating system, networking, server hardware, and so on. Therefore, the warehouse and its platform are two distinctive but related layers. Hence, the distinction is significant because DW modernization may hit only the data, the platform, or both. Data is not something to be replaced, instead, modernization may extend, consolidate, remodel, and improve data. But in many cases, however, replacing the DW platform could be viable if not a practical modernization approach when the platform is deficient or no longer a decent fit for business and its technological goals.

Rip and replace is an inherently expensive and disruptive modernization exercise which is why many organizations prefer to just modernize and improve the existing platform. A growing number of organizations complement the existing DW platform by adding up other standalone platforms to the DWE or combined update and complement strategies.

With these concerns in mind, it is not surprising that many users are taking into consideration DW platform replacements as they plan their DW modernizations.

Augmenting (but do not replace) the current DW’s main platform by adding up supplementary data platforms and tools –

More survey respondents, (42%) selected this strategy. Choosing this strategy is consistent with the movement’s multiplatform DW environments, which is one of today’s strongest trends in data warehousing. This is a non-disruptive task from a business perspective for it preserves current investments in data warehousing, and when done properly, it prolongs the life of a costly and functional system. TDWI has for years seen users deploy DW appliances and columnar databases as part of their warehouse augmentation and modernization policy and just recently, Hadoop has turned out to be the leading analytics and data platform for such policies.

Replacing DW’s existing main platform –

Here, 15% of organizations describe DW modernization as replacing the existing data warehouse’s main platform, which of course is conservative. For a minority of users, this approach is fully appropriate despite the disruption and expenses that may incur. Typical of this approach are those with a deficient or outmoded DW platform. 

Determining strategy on a case-to-case approach –

Arbitrary modernization, as discussed earlier, has its rightful place in DW modernization but should be controlled to reduce its havoc and turmoil. From a different perspective, considering that modernization hits several strata of the overall technology stack, plus has varied business drivers, hence it is inevitable that a certain amount of case-to-case analysis is compulsory.

No strategy yet, although one is needed –

A few interviewed users (14%), grumbled that the continuous modernization of the DW was one practice after the other due to business impatience, resulting too often in a disorganized DW that endures many architectural verrucae, has inconsistent data quality, and is difficult to optimize and maintain.


Modern Data Warehouse
Modern Data Warehouse | beyondkey.com

Data have evolved from an important to an extremely critical part of entire organizations in terms of predicting and tracking particularly important business processes. A modern DW is a basic requirement to make this happen.

DW modernization can combine streaming and other unstructured data resources with existing data warehouse investments. It further optimizes DW storage and provides query-able archives and rationalizes the DW for greater simplicity and lower cost. 

Additionally, DW modernization provides better query performance to enable complex analytical applications resulting in improved business insights into operations for real-time decision-making.







IBM | Big data at the speed of business

TDWI | Data Warehouse Modernization in the Age of Big Data Analytics

Saxon | Challenges and Benefits Associated with Modernizing the Data warehouse

Previous Post Next Post