Key takeaways
Data quality is judged not by whether the value looks clean, but by whether it can be used for the intended purpose. Real-time alerts, research analytics, and greenhouse gas reporting require different accuracy and completeness. Prior to collection, quality goals and inspection plans must be established and device installation, calibration, time frames, units, communications, and maintenance responsibilities must be established. Simply deleting outliers after a problem occurs does not provide quality control.
Quality items to check every day
First check that the units and time zone are correct and that the timestamps on different devices are aligned. Completeness is calculated as the actual number of collections compared to the expected number of collections and indicates physically impossible ranges, rapid changes, and long-repeating fixed values. Differences from adjacent sensors or reference equipment, interference factors such as temperature and humidity, and changes after calibration are also viewed. Automated checks are fast, but can flag rare real-world events as errors, requiring human verification of raw data and field records.
The original data will not be overwritten. Make corrections and exclusion flags separate fields, leaving the rules and versions applied. Device replacements, firmware changes, location moves, cleaning and power interruptions are also recorded on the same time base. Quality reports don't just highlight normal collection rates, they also show missingness, exclusions, potential for re-collection, and usage restrictions.
Field application checklist
Set quality goals and allowable missingness for each intended use.
Automatically checks unit, time, range, rate of change, fixed value, and drift.
Raw data, correction values, reasons for exclusion, and manual review history are stored separately.
FAQ
Can I delete outliers right away?
It is better to leave a flag and reason for exclusion than to delete it. You need the raw data so you can review it again when the rules change.
Does higher collection rate mean better quality?
Integrity is only one factor. Even if values are collected continuously, they may not be fit for purpose if they have drift, fixed values, or timing errors.
Rationale and Limitations
We referenced the principles of unit, time, completeness, range, rate of change, fixed value, automatic check and manual verification presented in US EPA's Air Sensor Quality Assurance document. Specific performance criteria and calibration procedures for livestock methane sensors must be established separately based on device specifications and application methodology.