Hydro-meteorological data quality assurance and improvement
Abstract
Advances in measurement equipment and data transfer enabled easy and economic automatic monitoring of various hydro-meteorological variables. The main characteristic of such automatic monitoring systems is that they do not rely on human activities, but only on electronic devices. Even if those electronic devices are of highest quality and accuracy, and properly tuned to specific problem, the reliability of measured values relyieson many other factors and unexpected or undesired occurrences, like modification of measurement micro location, power supply shortages or surges, etc. The sampled and acquired data values have to be additionally checked, validated and sometimes improved or cleared before further use. This paper presents an innovative approach to data validation and improvement through the framework generally applicable to all hydrological data acquisition systems. The proposed framework can incorporate any number of validation methods and can be easily customized according to t...he characteristics of every single measured variable. The framework allows for the self-adjustment and feedback to support self-learning of used validation methods, same as expert-controlled learning and supervision. After data validation, for low-scored data, its value quality can be improved if redundant data exist, so framework has the data reconstruction module. By applying different interpolation techniques or using redundant data value the new data is created same as accompanying metadata with the reconstruction history. After data reconstruction, the framework supports the data adjustment, the post-processing phase where the data is adjusted for the specific needs of each user. Every validated and sometimes improved data value is accompanied with a meta-data that holds its validation grade as a quality indicator for further use. .
Keywords:
data quality assurance / validation / data improvement / interpolationSource:
Journal of Serbian Society for Computational Mechanics, 2009, 3, 1, 228-249Publisher:
- Univerzitet u Kragujevcu - Fakultet inženjerskih nauka, Kragujevac
Collections
Institution/Community
GraFarTY - JOUR AU - Branisavljević, Nemanja AU - Prodanović, Dušan AU - Arsić, Miomir AU - Simić, Zoran AU - Borota, J. PY - 2009 UR - https://grafar.grf.bg.ac.rs/handle/123456789/223 AB - Advances in measurement equipment and data transfer enabled easy and economic automatic monitoring of various hydro-meteorological variables. The main characteristic of such automatic monitoring systems is that they do not rely on human activities, but only on electronic devices. Even if those electronic devices are of highest quality and accuracy, and properly tuned to specific problem, the reliability of measured values relyieson many other factors and unexpected or undesired occurrences, like modification of measurement micro location, power supply shortages or surges, etc. The sampled and acquired data values have to be additionally checked, validated and sometimes improved or cleared before further use. This paper presents an innovative approach to data validation and improvement through the framework generally applicable to all hydrological data acquisition systems. The proposed framework can incorporate any number of validation methods and can be easily customized according to the characteristics of every single measured variable. The framework allows for the self-adjustment and feedback to support self-learning of used validation methods, same as expert-controlled learning and supervision. After data validation, for low-scored data, its value quality can be improved if redundant data exist, so framework has the data reconstruction module. By applying different interpolation techniques or using redundant data value the new data is created same as accompanying metadata with the reconstruction history. After data reconstruction, the framework supports the data adjustment, the post-processing phase where the data is adjusted for the specific needs of each user. Every validated and sometimes improved data value is accompanied with a meta-data that holds its validation grade as a quality indicator for further use. . PB - Univerzitet u Kragujevcu - Fakultet inženjerskih nauka, Kragujevac T2 - Journal of Serbian Society for Computational Mechanics T1 - Hydro-meteorological data quality assurance and improvement EP - 249 IS - 1 SP - 228 VL - 3 UR - https://hdl.handle.net/21.15107/rcub_grafar_223 ER -
@article{ author = "Branisavljević, Nemanja and Prodanović, Dušan and Arsić, Miomir and Simić, Zoran and Borota, J.", year = "2009", abstract = "Advances in measurement equipment and data transfer enabled easy and economic automatic monitoring of various hydro-meteorological variables. The main characteristic of such automatic monitoring systems is that they do not rely on human activities, but only on electronic devices. Even if those electronic devices are of highest quality and accuracy, and properly tuned to specific problem, the reliability of measured values relyieson many other factors and unexpected or undesired occurrences, like modification of measurement micro location, power supply shortages or surges, etc. The sampled and acquired data values have to be additionally checked, validated and sometimes improved or cleared before further use. This paper presents an innovative approach to data validation and improvement through the framework generally applicable to all hydrological data acquisition systems. The proposed framework can incorporate any number of validation methods and can be easily customized according to the characteristics of every single measured variable. The framework allows for the self-adjustment and feedback to support self-learning of used validation methods, same as expert-controlled learning and supervision. After data validation, for low-scored data, its value quality can be improved if redundant data exist, so framework has the data reconstruction module. By applying different interpolation techniques or using redundant data value the new data is created same as accompanying metadata with the reconstruction history. After data reconstruction, the framework supports the data adjustment, the post-processing phase where the data is adjusted for the specific needs of each user. Every validated and sometimes improved data value is accompanied with a meta-data that holds its validation grade as a quality indicator for further use. .", publisher = "Univerzitet u Kragujevcu - Fakultet inženjerskih nauka, Kragujevac", journal = "Journal of Serbian Society for Computational Mechanics", title = "Hydro-meteorological data quality assurance and improvement", pages = "249-228", number = "1", volume = "3", url = "https://hdl.handle.net/21.15107/rcub_grafar_223" }
Branisavljević, N., Prodanović, D., Arsić, M., Simić, Z.,& Borota, J.. (2009). Hydro-meteorological data quality assurance and improvement. in Journal of Serbian Society for Computational Mechanics Univerzitet u Kragujevcu - Fakultet inženjerskih nauka, Kragujevac., 3(1), 228-249. https://hdl.handle.net/21.15107/rcub_grafar_223
Branisavljević N, Prodanović D, Arsić M, Simić Z, Borota J. Hydro-meteorological data quality assurance and improvement. in Journal of Serbian Society for Computational Mechanics. 2009;3(1):228-249. https://hdl.handle.net/21.15107/rcub_grafar_223 .
Branisavljević, Nemanja, Prodanović, Dušan, Arsić, Miomir, Simić, Zoran, Borota, J., "Hydro-meteorological data quality assurance and improvement" in Journal of Serbian Society for Computational Mechanics, 3, no. 1 (2009):228-249, https://hdl.handle.net/21.15107/rcub_grafar_223 .