Data migration obstacles

Technology is constantly evolving and business needs are always changing. Given this state of consistent change, a majority of organizations will need to take on a data migration project at some point. In our recent 2017 global data benchmark report, we found that 35 percent of organizations will start a data migration this year. Data migrations are costly projects: they require a good deal of planning, and involve a number of stakeholders from across your business. A data migration is typically one step in launching a new system of some sort—be it a customer relationship management platform, an advanced business intelligence (BI) reporting tool, or consolidation of multiple sources into a data warehouse. After taking the time to vet the various options and investing a significant amount of money into a shiny new system, focusing your time and resources to preparing the data for a migration can often feel a little less exciting. While it is not cheap, the cost of a data migration is only a fraction of the money you may spend on purchasing and implementing a new system—but a successful data migration is critical to set your business up for success.

Data quality – According to our study, 44 percent of U.S. organizations say that data quality issues caused delays for their migration projects. Data quality is essential to a successful data migration. When organizations undertake a data migration, they often unearth data quality issues that may not have been noticeable in their existing systems because of workarounds or system shortcomings, but that become glaring when it’s time to switch to a new system. Poor data quality slows down the entire migration process, and sometimes bad data will be outright rejected by the new system as newer technologies often have stricter standards surrounding the quality of data entering the system.

Lack of standardization – To easily switch from an old system to a new one, data should be formatted the same way. Standardization is important to ensure that all of the data you are migrating from one or more existing systems is in the correct format to be understood in the new system. If, for example, the new system requires that names be listed as (last, first) and you have a mix of formatting for your existing data, that lack of standardization can create significant issues and slow down the migration process.

Insufficient labor – Many organizations rely heavily on manual processes when conducting a data migration. This entails teams of SQL experts coding queries to extract all the existing data and prepare it for migration—often without understanding the business context or the state of data quality. Considering that migrations involve moving hundreds of thousands, if not millions of records, the amount of manpower required is immense. We found that for 29 percent of organizations, insufficient labor was a major obstacle that delayed their data migrations. Even for organizations that have the software and tools needed to process the data, successful data migrations still involve a great deal of effort on the behalf of the IT team and many other stakeholders across the business.

Not enough time - As mentioned earlier, another big issue that many organizations encounter is that they run out of time. A data migration is often a timely process, and when companies do not proactively take steps to analyze their data at the outset, they are often surprised at the sheer volume of information that must be migrated. Many organizations also fail to build in time for an initial analysis or the many user acceptance tests that are typically conducted to see whether the data works in its new environment. Discovering that you have more data than expected or finding that the data fails in the new system can significantly delay the data migration time frame. When a user acceptance test fails, for example, the team must start over again, build something new, and test it, repeating the process until something works. Often, the migration timeline is determined by deadlines imposed by the specific business objective driving the adoption of a new system.

Ineffective tools – There are many different solutions and tools that organizations use to complete a data migration. Some organizations require robust, comprehensive tools, but many organizations can combine some lighter-weight tools that are less expensive. Often, the tools designed specifically for the migration lack data quality capabilities that are essential to success. Ensuring that you have a data migration tool to address data quality needs can help make your migration more effective and more efficient.

Poor communication – A lack of communication among the stakeholders in a data migration is common and can lead to tremendous issues. In our study, more than half of the respondents (51%) agreed that improved communication would have helped to reduce delays in the process. Generally, the IT and business teams will meet at the beginning of the process to discuss scope and expectations of the new system, but then IT will handle the full migration with minimal involvement from business users across the organization. Without business input throughout the process, IT completes the required technical specifications of the migration, but has little sense of whether or not the results will be practical and useful for the business at large.

Budget restrictions – Two of the most common reasons why organizations’ data migrations fail are that they run out of money or run out of time. This is because companies frequently overlook the budgeting and forecasting stage of the migration. Investing the money it takes to successfully complete a migration is necessary for achieving the desired outcomes or any potential return on investment from the new system. Many organizations end up setting themselves up for failure by never determining what an accurate budget might look like and underestimating how much a data migration might cost. Many businesses also ignore best practices, and their costs end up being higher in the end. Any perceived savings an organization tries to achieve by skimping on planning and preparatory investments ends up coming back to bite them later on, and can often have repercussions beyond just the budget.

Despite so many potential obstacles, a well-prepared organization can achieve a successful data migration. Planning effectively and getting the right people, processes, and tools in place can help you overcome these challenges and deliver the migration on time and on budget. Following the best practices outlined in the next section can help you avoid these pitfalls and set you up for a more seamless data migration.

The secret to data migration success

If there is one true secret to data migration success, it’s all in the planning. Planning for your data migration can be an intensive process, but it will create a better experience for all involved, and it’s the best way to ensure your new system will perform as desired. Knowing your specific needs and what you wish to get out of your data migration will help you understand the right team of people to put together, what your process will look like, and which tool or combination of tools will adequately fit your needs.

There are two main approaches to data migrations: big bang or phased.² In some instances, organizations require the big bang approach and need to transfer all their data to the new system, or series of systems in one fell swoop. This approach provides the benefit of a shorter timeline, which can potentially lead to lower costs. The big bang approach is often necessary for organizations that don’t have the option of simultaneously working in both systems and can’t afford any downtime for normal business operations. Organizations often plan for big bang migrations over the course of a number of months, and then execute the full migration over a weekend or holiday. The phased approach moves data in smaller segments over a longer period of time. The process involves less risk and provides ample time for business users to learn and acclimate to the new system, which can help with adoption. Sometimes organizations divide their migration and utilize both approaches, so perhaps a large chunk of essential business data is migrated via big bang, while other elements are migrated in a phased approach.