Online time data series pre-processing for the improved performance of anomaly detection methods
Apstrakt
The number of automated measuring and reporting systems used in water distribution and sewer systems is dramatically increasing and, as a consequence, so is the volume of acquired data. Since the real time data is likely to contain a certain amount of anomalous values and since the probability of equipment malfunction is high, it is essential to equip the SCADA with automatic procedures that will detect the problems and assist the user in monitoring and data management. A number of anomaly detection techniques and methods exist that can be used with varying success. Some of those techniques in some cases are applicable to the online usage (inspection of the incoming data streams) but usually are more suitable for the offline data processing since they require frequent expert's involvement in parameter adjustment. The aim of this paper is to explore the online and offline data pre-processing techniques that could be used to remove redundant information and reduce the total volume of acq...uired data whilst preserving all the necessary data series features that could be used for anomaly detections. The paper explores the usefulness of different pre-processing techniques as a tool for improving the anomaly detection methods. The methodology developed is tested on several sets of real-life data, with different anomaly detection procedures including statistical, model-based and data mining approaches. The results obtained demonstrate the effectiveness of the suggested methodology.
Izvor:
Integrating Water Systems - Proceedings of the 10th International on Computing and Control for the W, 2010, 99-Kolekcije
Institucija/grupa
GraFarTY - CONF AU - Branisavljević, Nemanja AU - Kapelan, Zoran AU - Prodanović, Dušan PY - 2010 UR - https://grafar.grf.bg.ac.rs/handle/123456789/281 AB - The number of automated measuring and reporting systems used in water distribution and sewer systems is dramatically increasing and, as a consequence, so is the volume of acquired data. Since the real time data is likely to contain a certain amount of anomalous values and since the probability of equipment malfunction is high, it is essential to equip the SCADA with automatic procedures that will detect the problems and assist the user in monitoring and data management. A number of anomaly detection techniques and methods exist that can be used with varying success. Some of those techniques in some cases are applicable to the online usage (inspection of the incoming data streams) but usually are more suitable for the offline data processing since they require frequent expert's involvement in parameter adjustment. The aim of this paper is to explore the online and offline data pre-processing techniques that could be used to remove redundant information and reduce the total volume of acquired data whilst preserving all the necessary data series features that could be used for anomaly detections. The paper explores the usefulness of different pre-processing techniques as a tool for improving the anomaly detection methods. The methodology developed is tested on several sets of real-life data, with different anomaly detection procedures including statistical, model-based and data mining approaches. The results obtained demonstrate the effectiveness of the suggested methodology. C3 - Integrating Water Systems - Proceedings of the 10th International on Computing and Control for the W T1 - Online time data series pre-processing for the improved performance of anomaly detection methods SP - 99 UR - https://hdl.handle.net/21.15107/rcub_grafar_281 ER -
@conference{ author = "Branisavljević, Nemanja and Kapelan, Zoran and Prodanović, Dušan", year = "2010", abstract = "The number of automated measuring and reporting systems used in water distribution and sewer systems is dramatically increasing and, as a consequence, so is the volume of acquired data. Since the real time data is likely to contain a certain amount of anomalous values and since the probability of equipment malfunction is high, it is essential to equip the SCADA with automatic procedures that will detect the problems and assist the user in monitoring and data management. A number of anomaly detection techniques and methods exist that can be used with varying success. Some of those techniques in some cases are applicable to the online usage (inspection of the incoming data streams) but usually are more suitable for the offline data processing since they require frequent expert's involvement in parameter adjustment. The aim of this paper is to explore the online and offline data pre-processing techniques that could be used to remove redundant information and reduce the total volume of acquired data whilst preserving all the necessary data series features that could be used for anomaly detections. The paper explores the usefulness of different pre-processing techniques as a tool for improving the anomaly detection methods. The methodology developed is tested on several sets of real-life data, with different anomaly detection procedures including statistical, model-based and data mining approaches. The results obtained demonstrate the effectiveness of the suggested methodology.", journal = "Integrating Water Systems - Proceedings of the 10th International on Computing and Control for the W", title = "Online time data series pre-processing for the improved performance of anomaly detection methods", pages = "99", url = "https://hdl.handle.net/21.15107/rcub_grafar_281" }
Branisavljević, N., Kapelan, Z.,& Prodanović, D.. (2010). Online time data series pre-processing for the improved performance of anomaly detection methods. in Integrating Water Systems - Proceedings of the 10th International on Computing and Control for the W, 99. https://hdl.handle.net/21.15107/rcub_grafar_281
Branisavljević N, Kapelan Z, Prodanović D. Online time data series pre-processing for the improved performance of anomaly detection methods. in Integrating Water Systems - Proceedings of the 10th International on Computing and Control for the W. 2010;:99. https://hdl.handle.net/21.15107/rcub_grafar_281 .
Branisavljević, Nemanja, Kapelan, Zoran, Prodanović, Dušan, "Online time data series pre-processing for the improved performance of anomaly detection methods" in Integrating Water Systems - Proceedings of the 10th International on Computing and Control for the W (2010):99, https://hdl.handle.net/21.15107/rcub_grafar_281 .