The amount of environmental data is increasing, and the data would be valuable to the society if they are delivered to the right processes at the right time. In the seminar, we show examples of available data, how they are produced and processed, and how the data can be used in new innovative applications. This presentation is part of the Environmental Data for Applications Seminar held on the 23rd of September 2015. The seminar was organised by the MMEA (Measurement, Measuring and Environmental Assessment) research programme under the Cleen Ltd (SHOK). The presentations are based on the research results related to environmental data interoperability. The participants included key players and partners in the field of environmental monitoring in Finland. More info at www.mmea.fi

1. Quality Control and Measurement Uncertainty
Ympäristötiedosta palveluihin – seminaari, 23.9.2015
Mauno Rönkkö (UEF),
Okko Kauhanen (UEF), Markus Stocker (UEF), Mikko Kolehmainen (UEF),
Harri Hytönen (Vaisala), Olli Ojanperä (Vaisala), Esko Juuso (UO),
Markku Ohenoja (UO), Ville Kotovirta (VTT), Maija Ojanen (VTT),
Petri Koponen (VTT), Teemu Näykki (SYKE), Jari Koskiaho (SYKE),
Niina Kotamäki (SYKE), Jari Silander (SYKE),
Hanna Huitu (LUKE), Jussi Nikander (LUKE),…

2. 24.9.2015Mauno Rönkkö 2
Contents
1. Sources for Uncertainty in Environmental Monitoring
2. Quality Flagging by Nordic Meteorological Institutes
3. Extended Quality Flagging Scheme to Environmental Data
4. Automatic Monitoring – case Väänteenjoki
5. Water Quality Monitoring and MUkit
6. Conclusion

3. Sources for Uncertainty
in
Environmental Monitoring
24.9.2015Mauno Rönkkö 3

4. 1. Case #1: Incomplete Understanding [1/8]
24.9.2015Mauno Rönkkö 4
http://www.mmea.fi

5. 1. Case #2: Indirect Measurements [2/8]
•The Finnish Environmental Institute (SYKE) monitors
total phosphorus of lakes and rivers in Finland.
•Currently, there is no device that measures total phosphorus of a lake.
•The amount of total phosphorus is
(1) estimated based on the amount of
suspended solids
(2) which is estimated based on
measured turbidity at
(3) a given location on a lake/river.
24.9.2015Mauno Rönkkö 5
http://wwwi3.ymparisto.fi/i3/sakylapyhajarvi/sakylapyhajarvi.htm

6. 1. Case #3: Heterogeneous Measurement
Methods [3/8]
24.9.2015Mauno Rönkkö 6
http://www.vaisala.com/en/products/
automaticweatherstations/Pages/default.aspx
http://www.biltema.fi/fi/Toimisto---Tekniikka/Kellot-
ja-Lampomittarit/Lampomittari/Langaton-saaasema-
84086/

7. 1. Case #4: Sampling [4/8]
24.9.2015Mauno Rönkkö 7
Is this a good enough sample?

8. 1. Case #5: Inconsistent Treatment of
Measurement Errors [5/8]
24.9.2015Mauno Rönkkö 8
ERRORS (random errors + systematic errors):
-Devices are individuals
-Measurements drift over time
-Devices are positioned badly
-Devices are used in non-optimal conditions
-Measurement noise is too large for measured quantity
-Person measuring affects the measurements
-Environmental conditions vary and affect measurements
-Several different measurement devices used to get a dataset
-Spatially and temporally different measurements are not preprocessed
-Wrong devices are used in a specific measurement method
-Different measurement methods with different devices are used
-No calibration
-Standards and not used or they are used improperly
-Measurement data is treated with wrong statistical methods
…

9. 1. Case #5: Inconsistent Treatment of
Measurement Errors [6/8]
24.9.2015Mauno Rönkkö 9
ERRORS (the less recognized but most important):
-Non-synchronized measurement clocks
-Cognitive pitfalls

10. 1. Case #6: Semantically Inconsistent
Interoperability [7/8]
24.9.2015Mauno Rönkkö 10

11. 1. Case #7: Poorly Understood Uncertainties and
Validities [8/8]
24.9.2015Mauno Rönkkö 11

12. Quality Flagging
by
Nordic Meteorological Institutes
24.9.2015Mauno Rönkkö 12
F. Vejen (ed), C. Jacobsson, U. Fredriksson, M. Moe, L. Andresen, E. Hellsten, P.
Rissanen, T. Palsdottir, and T. Arason. Quality Control of Meteorological Observa-
tions. Automatic Methods Used in the Nordic Countries. Climate Report 8/2002,
Norwegian Meteorological Institute, 2002.

13. 2. Why bother?
24.9.2015Mauno Rönkkö 13
THIS IS WATER CONSUMPTION!?

14. 2. Why bother?
24.9.2015Mauno Rönkkö 14
IS THIS WATER
CONSUMPTION!?

15. 2. Why bother?
24.9.2015Mauno Rönkkö 15
THIS IS WATER CONSUMPTION!

16. 2. Data to information
24.9.2015Mauno Rönkkö 16
Measurement device
Server
Data storage
Data analysis and refinement
Human operator

17. 2. Quality checks
24.9.2015Mauno Rönkkö 17
QC0 QC1
QC2
HQC
Measurement device
Server
Data storage
Data analysis and refinement
Human operator

18. 2. Quality checks
24.9.2015Mauno Rönkkö 18
QC1
QC2
HQC
Measurement device
Server
Data storage
Data analysis and refinement
Human operator
QC0: real-time
quality control on
individual data points
about range, step and
consistency

19. 2. Quality checks
24.9.2015Mauno Rönkkö 19
QC2
HQC
Measurement device
Server
Data storage
Data analysis and refinement
Human operator
QC0: real-time
quality control on
individual data
points about range,
step and consistency
QC1: real-time quality
control on individual data
points using statistical
methods, including missing
and expected values

20. 2. Quality checks
24.9.2015Mauno Rönkkö 20
HQC
Measurement device
Server
Data storage
Data analysis and refinement
Human operator
QC0: real-time
quality control on
individual data
points about range,
step and consistency
QC1: real-time quality
control on individual data
points using statistical
methods, including
missing and expected
values
QC2: non-real-time quality
control on data sets
including spatial and
temporal analysis with
corrective computations

21. 2. Quality checks
24.9.2015Mauno Rönkkö 21
Measurement device
Server
Data storage
Data analysis and refinement
Human operator
QC0: real-time
quality control on
individual data
points about range,
step and consistency
QC1: real-time quality
control on individual data
points using statistical
methods, including
missing and expected
values
QC2: non-real-time quality
control on data sets
including temporal and
spatial analysis with
corrective computations
HQC: non-real-time
quality inspection
including visualization;
the final word

22. 2. Measurement Data
24.9.2015Mauno Rönkkö 22
556 2014-09-27T09:00:30 62.8925, 27.678333 22.6 42
time
humiditytemperature
location
device-id

23. 2. Measurement Data with a Quality Flag
24.9.2015Mauno Rönkkö 23
556 2014-09-27T09:00:30 62.8925, 27.678333 22.6 42 9330
time
quality flag
humiditytemperature
location
device-id

24. 2. The Flag Values
24.9.2015Mauno Rönkkö 24
556 2014-09-27T09:00:30 62.8925, 27.678333 22.6 42 9330
time
quality flag
humiditytemperature
location
device-id
C = 1000EHQC + 100EQC2
+ 10EQC1 + EQC0

25. 2. Multiple Data Points
24.9.2015Mauno Rönkkö 25
556 2014-09-27T09:00:30 62.8925, 27.678333 22.6 42 9330
C = 1000EHQC + 100EQC2
+ 10EQC1 + EQC0
556 2014-09-27T09:00:30 62.8925, 27.678333 15.3 55 4000

26. Extending
the Quality Flagging Scheme
to Environmental Data
24.9.2015Mauno Rönkkö 26
M. Rönkkö, O. Kauhanen, M. Stocker, H. Hytönen, V. Kotovirta, E. Juuso, M. Kolehmainen.
Quality Control of Environmental Measurement Data with Quality Flagging.
IFIP Advances in Information and Communication Technology, 2015, Volume 448,
Environmental Software Systems. Infrastructures, Services and Applications, pages 343-350.

27. 3. Generic Interpretation
24.9.2015Mauno Rönkkö 27
Flag Original interpretation Generic interpretation
0 No check performed Value not checked
1 Observation is ok Approved value
2 Suspected small difference Suspicious value
3 Suspected big difference Anomalous value
4 Calculated value Corrected value
5 Interpolated value Imputed value
6 (Not defined originally) Erroneous value
7 (Not defined originally) Frozen value
8 Missing value Missing value
9 Deleted value Deleted value

28. 3. Example Service Architecture
24.9.2015Mauno Rönkkö 28

29. 3. Quality Control of Water Consumption Data
24.9.2015Mauno Rönkkö 29

30. 3. Quality Control of Water Consumption Data
24.9.2015Mauno Rönkkö 30
QC1: measured and checked once a minute
QC2: runs every 2 hours, used for spotting leaks and malfunctions
HQC: done once a month, aims at resolving frozen data values

31. Automatic Monitoring
- case Väänteenjoki
24.9.2015Mauno Rönkkö 31

32. 24.9.2015Mauno Rönkkö 32
4. Automatic monitoring – case Väänteenjoki
• The challenge in automating water quality monitoring is that the measurement data
have not only significant seasonal variation, but also erroneous values
• Thus, without proper quality control and reliable uncertainty estimation, the data has
little value
• As a solution, we have implemented a computation service based on an Enterprise
Service Bus Architecture. The service provides means for online quality control and
integration of uncertainty estimation
• Case study: In the Karjaanjoki River
Basin the Väänteenjoki site equipped
with an OBS3+ turbidity sensor
(Campbell Scientific inc.)
• OBS3+ sensor emits a near-infrared
light into the water, measures the
light that scatters back from the
suspended particles, and transforms
this information into turbidity values
in Nephelometric Turbidity Units
(NTU)

33. 24.9.2015Mauno Rönkkö 33
4. Automatic monitoring – case Väänteenjoki
• The “raw” turbidity recorded by the OBS3+ sensor had to be calibrated against the
turbidity determined from water samples taken near the sensor
• Calibration equation was determined by linear regression between the values of the
water samples and the simultaneous values recorded by the sensor
• Then, because turbidity does not denote the content of substance in water, the
calibrated turbidity data had to be converted to concentrations of susp.solids and
total P
• We have implemented a computational service that automates and integrates
uncertainty estimation to the sequence of operations

34. Water Quality Monitoring
and MUkit
24.9.2015Mauno Rönkkö 34

35. 24.9.2015Mauno Rönkkö 35

36. 24.9.2015Mauno Rönkkö 36

37. 24.9.2015Mauno Rönkkö 37

38. 24.9.2015Mauno Rönkkö 38

39. Conclusion
24.9.2015Mauno Rönkkö 39

40. 5. Conclusion [1/2]
•What you cannot measure, you cannot control.
•Sources for uncertainties
Incomplete understanding, Indirect measurements, Heterogeneous
measurement methods, Sampling, Inconsistent treatment of
measurement errors, Semantically inconsistent interoperability, Poorly
understood uncertainties and validities
•Quality Flagging
– scheme by the Nordic Meteorological Institutes
– Quality checks at various stages; Real-time and non-real-time checks
•Quality Flagging of Environmental Data
– Generic interpretation
– ESB based architecture
24.9.2015Mauno Rönkkö 40

41. 5. Conclusion [2/2]
•Automatic monitoring – case Väänteenjoki
– proper quality control and reliable uncertainty estimation required
– implemented a computation service based on an ESB
•Water quality monitoring and MUkit
– Based on the Nordtest TR 537 guide and on the standard SFS-EN ISO
11352
– Automated turbidity measuring system
for ”real-time” uncertainty estimation using AutoMUkit
– Several international publications available!
24.9.2015Mauno Rönkkö 41

42. Mauno Rönkkö
mauno.ronkko@uef.fi
tel. +358 40 355 2202
www.uef.fi