Inspire Compliant Weather Data

INSPIRE Compliant Weather Data
Roope Tervo, Mikko Visa, Jukka Pakarinen | Finnish Meteorological Institute

Overview
Data Models
Producing Data Sets
Experiences
2/15/2017
INSPIRE Compliant Weather Data | Roope Tervo,
Mikko Visa, Jukka Pakarinen
2

• Finnish Meteorological Institute opened its data in 2013.
• Basically everything FMI owns was opened.
• The very same data portal works as Open Data and INSPIRE
portal.
FMI Open Data
2/15/2017
3
Meta data
Services
ISO19115 WFS WMS
CSW
Grid Series
Observations
Time Series
Observations
Data
Models O&M
Simple
Feature
GRIB
NetCDF GeoTiff

2/15/2017
INSPIRE Compliant Weather Data | Roope Tervo, Mikko Visa, Jukka
Pakarinen
4
Data Sets
• Way beyond INSPIRE Data Specification scope
• Instantaneous weather and marine observations
• AWS, soundings, mast, air quality, sun radiation, marine, lightning…
• Radiation observations by Radiation and Nuclear Safety Authority
(STUK)
• Road weather observations
• Model data from 6 different weather and marine models
• Weather radar images
• Climatological data
• Observation time series, analysis, climatological reference…

Overview
Data Models
Producing Data Sets
Experiences
2/15/2017
11

Data Models
o Observations and point
forecasts as GML
o The same data is published in
three different formats.
o Gridded data is provided in
appropriate binary format (grib,
NetCDF, HDF…)
o WFS members contains the
metadata ‘envelope’ with a link
to a actual data
o WCS will replace this
‘arrangement’ at some point
2/15/2017 12
Pakarinen
o Every XML document as self-explanatory as possible

om:phenomenonTime
• Tells start and end time of returned data
2/15/2017
13
Data Models
most important elements

om:procedure
• Describes data producing procedure
om:parameter
• Changing parameters in the process
• i.e. analysis time of the numerical model
2/15/2017
14
Data Models

om:featureOfInterest
• Describes location(s) or area of returned data
• Several different kind of identifiers
• target:SurfaceWeatherTargetArea  area (grid data)
• fmisid  FMI id for observation station
• wmo  WMO id for observation station
• geoid  http://geonames.org id for location
• name  Human readable name
• region  county of the station
2/15/2017
15
Data Models

2/15/2017
16
om:observedProperty
• Describes meteorological parameters in returned data
• Defined as xlink
• Describes label, basePhenomenon and units of measure of
every parameter in the reponse
• Tip: use the same service without parameter definitions to
query all possible parameters
• http://data.fmi.fi/fmi-apikey/.../meta?observableProperty=forecast
• Note that all parameters are not available in all data sets
Data Models

om:result
• Data in required format in observations and point forecasts and link to a
external binary data in grid formatted data
2/15/2017
17
Data Models

Data Models
gmlcov:MultiPointCoverage
2/15/2017 18
gml:rangeSet
gml:doubleOrNilReasonTupleList
The data is listed for every
point defined in domain set.
gml:domainSet
gmlcov:simpleMultiPoint
The coverage is
defined as a list of
points in 4
dimensional grid (lat,
lon, height, time).
gmlcov:rangeType
The parameters
listed in range set
are defined in
separate element.
Pakarinen

Cons
- Not intuitive
- No natural
structure of XML
 XSLT and
Xpath don’t work
Pros
+ Compact
+ Efficient
+ Small file size
+ Works for many
data types
2/15/2017 19
Data Models
Pakarinen

Data Models
wml2:MeasurementTimeseries
2/15/2017 20
wml2:MeasurementTimeseries
One member contains time
series of one parameter and
one location
Pakarinen

Cons
- Lots of repetition
- Large file size
- Heavy for DOM-
based parsers
- Don’t work i.e. for
thunder strikes
Pros
+ Intuitive
+ Easy to use
+ XSLT & XPath
works
2/15/2017 21
Data Models
wml2:MeasurementTime
series
Pakarinen

Data Models
SimpleFeature
2/15/2017 22
SimpleFeature
One member contains one
time step of one parameter
and one location
Pakarinen

Cons
- Huge file size
- Heavy for DOM-
based parsers
Pros
+ Intuitive
+ Easy to use
+ Wide application
support
2/15/2017 23
Data Models
SimpleFeature
Pakarinen

2/15/2017 24
Data Models
file size comparison
81.7
52.9
1.81.3 1.2 0.2
0
10
20
30
40
50
60
70
80
90
Document Size
[MB]
Compressed
DocumentSize
[MB]
Pakarinen

2/15/2017 25
Data Models
Popularity Comparison
80
19.8
0.2
0
10
20
30
40
50
60
70
80
90
Downloads[%]
Pakarinen

Data Type Data Format
Observations wml2:MeasurementTimeseries
SimpleFeature
Point Forecasts wml2:MeasurementTimeseries
SimpleFeature
Lightning Observations gmlcov:MultiPointCoverage
SimpleFeature
Grid Forecasts XML Envelope + Grib2/NetCDF
Radar Images GeoTiff / PNG images
METAR IWXXM
2/15/2017 26INSPIRE Compliant Weather Data | Roope Tervo, Mikko Visa, Jukka Pakarinen

• Aviation weather reposts are delivered as IWXXM
• New data model coming into use in aviation defined by
ICAO and WMO
• Consists of the same elements than other messages
• om:phenomenonTime, om:procedure, om:featureOfInterest,
om:result
• Content of the METAR is in om:result part as
• extracted into XML elements
• original, “old fashion”, METAR string
2/15/2017
27
Data Models
aviation observations IWXXM / METARS

• Once published, content and structure of IWXXM
messages will not change
• Messages will contain a digital signature
• Use GetPropertyValue to download only original
METAR string
• http://data.fmi.fi/fmi-
apikey/…/wfs?request=GetPropertyValue&storedquery_id=f
mi::avi::observations::finland::latest::iwxxm&valuereference=
wfs:FeatureCollection/wfs:member/avi:VerifiableMessage/av
i:metadata/avi:MessageMetadata/avi:source/avi:Process/avi:
input
2/15/2017
28
Data Models

• Messages have been automatically created based on
the original METAR/SPECI code messages.
• Messages are NOT an authoritative aviation
weather report. They SHOULD NOT be used as a
weather report to be used in flight planning or other
aviation related use.
2/15/2017
29
Data Models

• WFS response contains the same meta data information
than in observations and point forecasts
• om:result contains gmlcov:RectifiedGridCoverage
• Basically the same with multipointcoverage
• domainSet defines the grid (now as regular grid)
• rangeSet contains data as an external link to the binary content
• rangeType defines the parameters
2/15/2017
30
Data Models
binary data

• Note that fileReference points often to an other service
• Can be used directly, but
• It is always good practice to consult WFS for available
times and parameters
2/15/2017
31
Data Models
binary data
Client
WFS
WMS
FMI Data
Server
Available radar images?
Available weather model outputs?

• Area, time and weather parameters may be defined in
the request
• For weather models, intersection of requested area and
available data area is returned
• For radar images, all images which intersects requested
spatial and temporal space are returned
2/15/2017
32
Data Models
binary data
Requested area
Radar images
Radar images Requested area
Model coverage
Weather models

Radar images
• The reference points to a “original” gray scale GeoTiff image
• Images can also be downloaded as color images
• Remove &styles=raster from the data request
• But then information is lost
• Used SLD-files can be downloaded from
https://github.com/fmidev/opendata-resources/tree/master/sld
• Consult om:parameter element for single radar measurement
parameters
• Scanning angle, bin count and bin length
2/15/2017
33
Data Models
binary data

Numerical models
• The reference points to (often a subset) the model output in
appropriate format
• GRIB2 for weather models
• NetCDF for marine models
• Note that whole model output can be large
• Up to 17 GB
• Download always only area, parameters and time range of
interest
2/15/2017
34
Data Models
binary data

ef:EnvironmentalMonitoringFacility
Data Models
Environmental Facilities
(blueprint)
2/15/2017 35
ef:EnvironmentalMonitoringNetwork
(AWS | Radar | ….)
• Activity time
• Procedure
• Observed property (same with
data products)
Pakarinen
ef:observingCapability 1…*

Overview
Data Models
Producing Data Sets
Experiences
2/15/2017
36

Services
2/15/2017
37
Data (NFS) Databases
GeoNetwork
CSW
GeoServer
WMS
SmartMet
Server
WFS & WCS & WMS

Open Data Service
Cluster
S1 S2 S3
Client Data Service
Cluster
S1 S2 S3
Load Balancer
Configuration
Data
(NFS)
Configuration
(NFS)
Database
2/15/2017 38
Pakarinen

Weather Data – Volumes
2/15/2017 39
• In-situ weather measurement 1 TB
• Weather radar data 50 TB
• NWP model gridded data (FMI)
• HIRLAM 120 TB
• AROME 230 TB
• Satellite image data (FMI)
• Globsnow 32 TB
• Other 50 TB
• Climate model data (FMI) 29 TB
• Other models
• SILAM 100 TB
• Tuuliatlas 21 TB
• Other 300 TB
Pakarinen

Producing INSPIRE Data Products
Observations
2/15/2017 40
Pakarinen
Database
SmartMet
Server
SmartMet
Server WFS

Point Forecasts
2/15/2017 41
Pakarinen
File
System
SmartMet
Server
SmartMet
Server WFS

Grid Forecasts 1/2
File
System
SmartMet
Server
SmartMet
Server WFS
2/15/2017 42
Pakarinen

Grid Forecasts 2/2
File
System
SmartMet
Server
SmartMet Server
Download
2/15/2017 43
Pakarinen

SmartMet Server
• Data and product server for MetOcean data
• High capacity & availability
• FMI installation handles over 30 000 000 requests each day
• Data is extracted and products generated on-demand
• INSPIRE Compliant
• Operative since 2008
• FMI client services (since 2008)
• Finnish Meteorological Institute (FMI) Open Data Portal (since 2013)
• Going to be used at Copernicus C3S Climate Data Store (ECMWF)
• Open source
2/15/2017
44

SmartMet Server
• Several input sources
• GRIB-, NetCDF-, etc. files (multi-dimensional grid data)
• PostGIS database (vectors)
• Point database (point observations)
• Several output interfaces and formats
• WMS, WFS 2.0
• JSON, XML, ASCII, HTML, SERIAL
• GRIB1, GRIB2, NetCDF
2/15/2017
45

SmartMet Server
Usage
2/15/2017
46
FMI Open Data
Portal & FMI
INSPIRE Data
Services
Backend for
clients’ web
services
Integration to
clients’
systemsBackend for
mobile
applications
Backend for
FMI Client
Services
Backend for
FMI public
pages
• Basis of most FMI product generation

• Published in 2016 in GitHub
• https://github.com/fmidev/smartmet-server
• MIT Licence
• Documentation in GitHub
• FMI will host the development
• Small contributions with pull requests
• In larger contributions, implementation plan is
recommended (in GitHub wiki)
• CLA (Contributor Licence Agreement) will be
required
2/15/2017
47
SmartMet Server
Open Source

Interested?
https://github.com/fmidev/smartmet-server
Source code and documentation
Follow:
https://github.com/fmidev
https://facebook.com/fmibeta
New releases will be updated here
2/15/2017
48

Overview
Data Models
Producing Data Sets
Experiences
2/15/2017
53

INSPIRE Data Sets
How to define a data set?
o All weather observations from
Finland?
 Would cause over 50 000 000
Observations (XML file size ~37 G)
o All observations from one
observation station?
 Would cause over 200 data sets
o Even one year’s observations cause
too large data set to handle
2/15/2017 54
Pakarinen

INSPIRE Data Sets
Meteorological data is a constant
flow of observations
FMI has one data set per data
type, i.e. one for ground weather,
observations, one for Hirlam
weather forecasts, etc…
Every data set have predefined
area and time range.
2/15/2017 55
Pakarinen

INSPIRE Data Sets
It is notable that data set
response depends on time it’s
requested
 Unique identifiers are not
reasonable
2/15/2017 56
Pakarinen

And a little over
450 000 data
downloads
per day
(5,2 req/s)
At the moment
over 11 000
registered users
2/15/2017 57
Some Experiences
Pakarinen

Practically
no client
supports
complex
features
Although standards
are followed, there’s
a gap between
provided data model
and clients’
capabilities
2/15/2017 58
Pakarinen
Some Experiences

GeoServer was
modified to support
stored queries in
WFS 2.0 (released
in version 2.7)
FMI published the
same data as simple
features to support
clients
2/15/2017 59
Pakarinen
Some Experiences

Industry is
happy to use
standardized
services
Amateur and
freelancer coders
would prefer simple
JSON API
2/15/2017 60
Pakarinen
Some Experiences

…but suites quite
well for exchanging
(subsets of) data.
2/15/2017 61
Pakarinen
Data format is too
verbose for clients to
use directly…
Some Experiences

For now,
very few have
been interested in
forecast models
as a grid data
Point forecasts,
observations and
radar images are the
most interesting data
types
2/15/2017 62
Pakarinen
Some Experiences

www.fmi.fi
http://ilmatieteenlaitos.fi/avoin-lahdekoodi
https://github.com/fmidev
https://en.ilmatieteenlaitos.fi/open-data
https://facebook.com/fmibeta
http://roopetervo.com
http://www.slideshare.net/tervo

Inspire Compliant Weather Data

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (19)

En vedette

En vedette (13)

Similaire à Inspire Compliant Weather Data

Similaire à Inspire Compliant Weather Data (20)

Plus de Roope Tervo

Plus de Roope Tervo (8)

Dernier

Dernier (20)

Inspire Compliant Weather Data