A large insurance company was facing significant roadblocks in leveraging data for critical insights. Their current infrastructure struggled to keep pace with the demands of new analytical and predictive models, hindering the timely delivery of valuable information. Additionally, a lack of integrated access to data on members, prospects, and their enrollment/claim history created major hurdles. This resulted in missed service level agreements (SLAs), impacting the productivity of both data scientists and business users. Furthermore, onboarding new data sources, such as those from IoT devices, proved to be a cumbersome and time-consuming process.
Capability: Data Engineering
Core Business: Claims Processing
Solution: Hadoop to BigQuery Migration
Tech Stack: Cloudera, GCP, BigQuery, Composer, Dataproc
Building a Scalable Data Foundation
The insurance company addressed their data challenges by implementing a comprehensive data modernization strategy. A key component was the automation of data migration using the ASCENDION AVA+ Low-Code Data Modernization Platform. This platform streamlined the conversion of Hive scripts to BigQuery, a highly scalable cloud data warehouse. Additionally, containerization with Docker images ensured automated deployment and efficient resource utilization.
To further enhance data accessibility and usability, the solution leveraged parametrization of datasets and labels. This standardized approach simplified data exploration and analysis. The team also designed high-performing data pipelines by implementing partitions and clustering techniques within BigQuery tables. Finally, they capitalized on BigQuery ML’s embedded Python capabilities to develop and deploy lead scoring models directly within the data warehouse, enabling faster analysis and decision-making.
The results:
- 60% TCO savings by migrating from Hadoop to BigQuery
- 4000+ data engineers/data scientists with robust data platform
- 5000+ hours effort savings in one year