Designing Intelligent Data Engineering Architectures for Automated Data Cleansing and Enrichment

Abstract

The exponential growth of data volume, variety, and velocity has made manual data preparation a critical bottleneck in analytics and machine learning pipelines. This paper explores the design of intelligent data engineering architectures that leverage machine learning (ML) and automation to perform scalable and accurate data cleansing and enrichment. We propose a layered, modular architecture that integrates rule-based systems with ML models for tasks such as anomaly detection, entity resolution, and semantic enrichment. The discussion includes a review of existing methodologies, a detailed blueprint for system design, and an analysis of key implementation challenges and performance metrics. The proposed framework aims to significantly reduce time-to-insight and improve data quality for downstream applications.