Data quality frameworks for real-time data pipelines - Girish Ganachari (North American Journal of Engineering Research, 2024)
This study analyzes trends in developing data quality frameworks for real-time data pipelines, which are crucial for various industries today. The paper acknowledges the significant benefits of real-time data processing but alsohighlights potential issues with data quality, integrity, completeness, and timeliness. Existing architectures designed for micro-batch processing and homogenous data sources are still used, despite the challenges they pose in these varied environments. The paper proposes methodologies like machine learning and AI for automating data quality control. Intersystem synergism is therefore central to developing robust solutions for real-time data quality control