Single Household View: Data Platform Architecture, Framework and Code Templates for Data Ingestion

Leading UK Telecom

Business Challenge​

A leading UK telecom wanted to have a single household view of their customers. The data resided in many distributed systems. The company chose Google Cloud as a strategic data and AI platform. The challenge was to ingest data from various sources and convert them into reusable data assets to support diverse analytics use cases.

 

Our Approach​

The first phase focused on building reusable architecture and templates, and a migration to GCP to create a data lake with reusable Data sets. We created architecture and developed event-driven batch ingestion using GCS, Cloud Functions and BigQuery. Deployment of GCP resources was organized through a combination of Terraform scripts and CI/CD applications.

 

Result

Best practice scalable, stable and cost-efficient architecture: streaming data platform, batch data processing systems, data storage, database, workflow manager, business analytics, repository management, distribution and orchestration.