Data quality frameworks for real-time data pipelines

Abstract

This study analyzes trends in developing data quality frameworks for real-time data pipelines, which are crucial for various industries today. The paper acknowledges the significant benefits of real-time data processing but also
highlights potential issues with data quality, integrity, completeness, and timeliness. Existing architectures designed for micro-batch processing and homogenous data sources are still used, despite the challenges they pose in these varied environments. The paper proposes methodologies like machine learning and AI for automating data quality control. Intersystem synergism is therefore central to developing robust solutions for real-time data quality control