FME Workspace Template Collection for Reading Fundamental Geographic Data in Japan

CONNECT. TRANSFORM. AUTOMATE.
FME Workspace Template Collection for Reading
Fundamental Geographic Data in Japan
Takashi Iijima

FME Certified Professional, pragmatica inc.

Background: KSJ and FGD
Major Fundamental Geographic Datasets
provided by Japanese Governmental Organizations
KSJ: 国⼟土数値情報
(Kokudo Suuchi Jouho)
National Land Numerical Information
FGD: 基盤地図情報
(Kiban Chizu Jouho)
Fundamental Geospatial Data

Background: KSJ Overview
!  Provider: 国⼟土交通省省
-- Ministry of Land,
Infrastructure, Transport and Tourism (MLIT)
!  http://nlftp.mlit.go.jp/ksj/
!  Feature Types: 60+; Geographic Features, Land
Use, Political Designated Zones, Administrative
Areas, Public Facilities, Transportation Facilities
etc. (Point, Line or Area)
!  Format: XML conforming to GML
!  GML reader cannot read the data.

Background: FGD Overview
!  Provider: 国⼟土地理理院
-- Geospatial Information
Authority of Japan (GSI)
!  http://www.gsi.go.jp/kiban/etsuran.html
!  Feature Types: Vectorized Topographical Map
Elements (Point, Line or Area), Survey Reference
Point, Digital Elevation Model (DEM)
!  Format: XML conforming to GML
!  GML reader can read the data except DEM.

Project Overview
!  Purpose: To provide easy and flexible way to read
KSJ and DEM in FGD with FME to increase their
usability.
!  Approach: Create FME workspaces to read the
datasets with the XML reader and create features
having appropriate geometry and attributes.
!  Goal: FME Template Collection

Part 1. Reading KSJ
!  General Data Structure
!  Framework of Workspaces
!  Geometry Creation – Point
!  Geometry Creation – Line
!  Geometry Creation – Area
!  Resolving Incorrect Data with xfMap

<?xml version="1.0" encoding="UTF-8"?>
<ksj:Dataset>
<gml:boundedBy>
<gml:EnvelopeWithTimePeriod srsName="JGD2000 / (B, L)" frame="GC / JST">
<gml:lowerCorner>20.0 123.0</gml:lowerCorner>
<gml:upperCorner>46.0 154.0</gml:upperCorner>
<gml:beginPosition calendarEraName="西暦">1900</gml:beginPosition>
<gml:endPosition indeterminatePosition="unknown"/>
</gml:EnvelopeWithTimePeriod>
</gml:boundedBy>
<gml:Point gml:id="n1">
<gml:pos>34.98509412 139.86044116</gml:pos>
</gml:Point>
<ksj:BusStop gml:id="ED01_1">
<ksj:position xlink:href="#n1"/>
<ksj:busStopName>西下町</ksj:busStopName>
<ksj:busRouteInformation>
<ksj:BusRouteInformation>
<ksj:busType>1</ksj:busType>
<ksj:busOperationCompany>JRバス関東（株）</ksj:busOperationCompany>
<ksj:busLineName>フラワー号</ksj:busLineName>
</ksj:BusRouteInformation>
</ksj:busRouteInformation>
</ksj:BusStop>
</ksj:Dataset>
KSJ: General Data Structure
Envelope

Geometries

Features

GML: Point, Curve,
OrientableCurve, Surface

A feature links to a geometry
via "href" attribute.

Contains SRS name:
JGD2000 or Tokyo Datum

KSJ: Framework of Workspaces
Merge

Features

Geometries

Envelope (SRS)

Merge

Set coordinate system
Modify attribute names
XML Reader feature type
with Feature Type Specific xfMap

XML Reader feature type with Common xfMap
Creates geometries (Line, Area) with Custom Transformer

XML Reader feature type created
with Common xfMap

Add geometry to feature
Group By: fme_basename
Join On: geometry ID

Add SRS name to feature
Group By: fme_basename
Unconditional merging

KSJ: Geometry Creation - Point
Mapping rule for <Point>
<mapping match="Point">
<feature-type>
<literal expr="Point" />
</feature-type>
<attributes>
<attribute>
<name><literal expr="Point.id" /></name>
<value><extract expr="@gml:id" /></value>
</attribute>
</attributes>
<geometry activate="xml-point">
<data name="data-string">
<extract expr="./pos"/>
</data>
<data name="axis-order">
<literal expr="2,1" />
</data>
</geometry>
</mapping>
<Point> element example
<gml:Point gml:id="p1">
<gml:pos>36.02232800 139.00306800</gml:pos>
</gml:Point>
<gml:pos>36.01240300 139.02192100</gml:pos>
</gml:Point>
<gml:pos>36.04452300 139.05435900</gml:pos>
</gml:Point>
<gml:pos>35.98977700 139.07710600</gml:pos>
</gml:Point>
<gml:pos>35.99583200 139.07745600</gml:pos>
</gml:Point>
Points can be created simply with xfMap.

KSJ: Geometry Creation - Line
Read <LineStringSegment> elements as XML fragments.
Create Line geometries with a Custom Transformer.
Mapping rule for <Curve>
<mapping match="Curve">
<feature-type>
<literal expr="Curve" />
</feature-type>
<attributes>
<attribute>
<name><literal expr="Curve.id" /></name>
</attribute>
<attribute type="list">
<name><literal expr="LineStringSegment" /></name>
<value>
<extract expr="./segments/LineStringSegment" as-xml="true" />
</value>
</attribute>
</attributes>
</mapping>
<Curve> element example
<gml:Curve gml:id="c-85">
<gml:segments>
<gml:LineStringSegment>
<gml:posList>
35.91887573 139.92888998
35.91885861 139.92893361
35.91885472 139.92894500
35.91885472 139.92895778
35.91887972 139.92899667
35.91925667 139.92939778
35.91927722 139.92945889
</gml:posList>
</gml:LineStringSegment>
</gml:segments>
</gml:Curve>

KSJ: Geometry Creation - Line
KsjLineBuilder (Custom Transformer) creates Line geometries.

Surface element example
<gml:Surface gml:id="a00001">
<gml:patches>
<gml:PolygonPatch>
<gml:exterior>
<gml:Ring>
<gml:curveMember xlink:href="#_c00003"/>
</gml:Ring>
</gml:exterior>
<gml:interior>
<gml:Ring>
</gml:Ring>
</gml:interior>
<gml:interior>
<gml:Ring>
</gml:Ring>
</gml:interior>
</gml:PolygonPatch>
</gml:patches>
</gml:Surface>
KSJ: Geometry Creation - Area
Mapping rule for Surface
<mapping match="Surface">
<feature-type>
<literal expr="Surface" />
</feature-type>
<attributes>
<attribute>
<name><literal expr="Surface.id" /></name>
</attribute>
</attributes>
<structure
matched-attributes="yes"
matched-prefix="attributes"
cardinality="*/patches/PolygonPatch{}/exterior/Ring/curveMember{}
*/patches/PolygonPatch{}/interior{}/Ring/curveMember{}" />
</mapping>
Complicated hierarchy XML structure can be
mapped to nested-structured list attribute.
Create Area geometries with a Custom
Transformer.

OrientableCurve element example
<gml:OrientableCurve gml:id="_c00100002" orientation="-">
<gml:baseCurve xlink:href="#c00100002"/>
</gml:OrientableCurve>
An OrientableCurve represents a reversed Curve,
links to a Curve by its ID.
Mapping Rule for OrientableCurve
<mapping match="OrientableCurve">
<feature-type>
<literal expr="OrientableCurve" />
</feature-type>
<structure
cardinality="*/+" />
</mapping>

KsjAreaBuilder (Custom Transformer) creates Area geometries.

KSJ: Bus Stop (Point)

KSJ: River (Line and Point)

KSJ: Administrative Boundary (Area)

KSJ: Resolving Incorrect Data with xfMap
Case 1
Incorrect - wrong feature type/attribute names
<ksj:AdministrativeArea gml:id="gy2">
<ksj:are xlink:href="#sf2"/>
<ksj:prn>千葉県</ksj:prn>
<ksj:cn2>柏市</ksj:cn2>
…
Two Mapping rules for the same feature type

<mapping match="AdministrativeBoundary">
<feature-type>
<literal expr="AdministrativeBoundary" />
</feature-type>
<structure
cardinality="*/+" />
</mapping>

<mapping match="AdministrativeArea">
<feature-type>
<literal expr="AdministrativeBoundary" />
</feature-type>
<attributes>
<attribute>
<name><literal expr="AdministrativeBoundary.id" /></name>
</attribute>
<attribute>
<name><literal expr="bounds.href" /></name>
<value><extract expr="./are[@xlink:href]" /></value>
</attribute>
<attribute>
<name><literal expr="prefectureName" /></name>
<value><extract expr="./prn" /></value>
</attribute>
<attribute>
<name><literal expr="cityName" /></name>
<value><extract expr="./cn2" /></value>
</attribute>
…
Multiple mapping rules for the same feature type are allowed
if “match” attributes (XML element names) are different.
Correct
<ksj:AdministrativeBoundary gml:id="ec00100004">
<ksj:bounds xlink:href="#a00100004"/>
<ksj:prefectureName>千葉県</ksj:prefectureName>
<ksj:cityName>柏市</ksj:cityName>
…

KSJ: Resolving Incorrect Data with xfMap
Case 2
Correct
<gml:pos>36.02232800 139.00306800</gml:pos>
</gml:Point>
Incorrect - wrong GML element name
<gml:Point gml:id="pt0">
<gml:position>36.08985800 139.78870900</gml:position>
</gml:Point>
Mapping rule modified for incorrect data
<mapping match="Point">
<feature-type>
<literal expr="Point" />
</feature-type>
<attributes>
<attribute>
<name><literal expr="Point.id" /></name>
</attribute>
</attributes>
<geometry activate="xml-point">
<data name="data-string">
<extract expr="./pos"/>
<extract expr="./position" />
</data>
<data name="axis-order">
<literal expr="2,1" />
</data>
</geometry>
</mapping>

<extract> element in xfMap generates just an
empty string if the XML path specified by “expr”
attribute doesn’t exist. This mapping rule works
for both correct data and incorrect data.

Part 2. Reading DEM in FGD
!  DEM Types
!  Data Structure
!  Sequence of Elevation Values
!  Skeleton of Workspace
!  Process of Creating Raster
!  Implementation and Performance

DEM: DEM Types
Type Name
Cell Sizes
width x height

Columns x Rows
per a Dataset

Extent Sizes
width x height
5m Mesh
0.2 x 0.2 sec.
225 x 150
45 x 30 sec.

10m Mesh
0.4 x 0.4 sec.
1125 x 750
450 x 300 sec.

250m Mesh
11.25 x 7.5 sec.
320 x 320
3600 x 2400 sec.

Both 5m Mesh DEM and 10m Mesh DEM are covering all of Japanese
land area; 250m Mesh DEM is covering a special area only.
Ideally the workspace should be applicable to any type of
DEM.

DEM: Data Structure
<?xml version="1.0" encoding="Shift_JIS"?>
<Dataset>
<DEM><coverage>
<gml:boundedBy>
<gml:Envelope srsName="fguuid:jgd2011.bl">
<gml:lowerCorner>38.916666667 139.8</gml:lowerCorner>
<gml:upperCorner>38.925 139.8125</gml:upperCorner>
</gml:Envelope>
</gml:boundedBy>
<gml:gridDomain>
<gml:Grid>
<gml:limits>
<gml:GridEnvelope>
<gml:low>0 0</gml:low>
<gml:high>224 149</gml:high>
</gml:GridEnvelope>
</gml:limits>
</gml:Grid>
</gml:gridDomain>
<gml:rangeSet>
<gml:DataBlock>
<gml:tupleList>
海水面,-9999.
海水面,-9999.
海水面,-9999.
地表面,0.63
…
地表面,6.83
地表面,6.67
地表面,6.57
</gml:tupleList>
</gml:DataBlock>
</gml:rangeSet>
<gml:coverageFunction>
<gml:GridFunction>
<gml:startPoint>108 0</gml:startPoint>
</gml:GridFunction>
</gml:coverageFunction>
</coverage></DEM>
</Dataset>

Number of columns and rows:
num_columns = high{0} + 1 – low{0}
num_rows = high{1} + 1 – low{1}

List of (Type, Elevation)
SRS name: “jgd2000.bl” or “jgd2011.bl”
Extents: (ymin, xmin), (ymax, xmax)

Index of start point in the grid matrix:
start_col = startPoint{0}
start_row = startPoint{1}

DEM: Sequence of Elevation Values
0 1 2 3 4 5 6 7 …

0
1
2
3
4
5
6
Start Column Index = 6, Start Row Index = 2

Elevation values are put in order in the turn of going to Right-Lower
from Left-Upper.
Location of the starting cell is specified by start column, row index
described in the <GridFunction> element.

<gml:rangeSet>
<gml:DataBlock>
<gml:tupleList>
海水面,-9999.
海水面,-9999.
海水面,-9999.
地表面,0.63
…
地表面,6.83
地表面,6.67
地表面,6.57
</gml:tupleList>
</gml:DataBlock>
</gml:rangeSet>>

DEM: Skeleton of Workspace
Dataset

Extract Number of
Columns / Rows

Extract Coordinates of
Lower / Upper Corner

Create Raster
Origin = (0, 0)
Cell Sizes = 1 x 1
Width = Number of Columns
Height = Number of Rows

Resize, Move,
Set CoordSys

XML Reader
with Feature Paths

Scaler
Offsetter
CoordinateSystemSetter

AttributeSplitter
AttributeCreator

AttributeSplitter

NumericRasterizer parameter settings
Size Specification: RowsColumns
Number of Columns (cells): <num_columns>
Number of Rows (cells): <num_rows>
Ground Extents
Minimum X: 0
Minimum Y: 0
Maximum X: <num_columns>
Maximum Y: <num_rows>

Scaler

Offsetter

(0, 0)

(xmin, ymin)

DEM: Process of Creating Raster
Extract Start Column / Row Index

Serialize Elevation Values

Create 3D Grid Points

Transform 3D Points into a Raster

3D Point Coordinate Calculation
Index = (0-based Index in the Sequence)
+ (Start Row Index) * (Number of Columns)
+ (Start Column Index)
Column Index = Index % (Number of Columns)
Row Index = INT(Index / (Number of Columns))
X = (Column Index) + 0.5
Y = (Number of Rows) – 0.5 – (Row Index)
Z = Elevation

NumericRasterizer parameter settings
Size Specification: RowsColumns
Number of Columns (cells): <num_columns>
Number of Rows (cells): <num_rows>
Ground Extents
Minimum X: 0
Minimum Y: 0
Maximum X: <num_columns>
Maximum Y: <num_rows>

Extract Number of Columns / Rows

DEM: Implementation and Performance
1st Implementation: uses existing Transformers only.
About 110 sec. for rasterizing a 10m Mesh DEM (843,750 cells).
The ListExploder seems to consume long time.

2nd Implementation: uses a Python script to extract Elevation
values and create 3D Grid Points.
About 25 sec.: Performance has been improved considerably.
But there should be more efficient way.

import fmeobjects
class GridPointCreator(object):
def input(self, feature):
count = feature.getAttribute('_count')
numCols = int(feature.getAttribute('_num_columns'))
numRows = int(feature.getAttribute('_num_rows'))
startCol = int(feature.getAttribute('_start{0}'))
startRow = int(feature.getAttribute('_start{1}'))
tuples = feature.getAttribute('DEM.coverage.rangeSet.DataBlock.tupleList').strip()
index = startRow * numCols + startCol
for z in [float(t.split(',')[1]) for t in tuples.split()]:
x = index % numCols + 0.5
y = numRows - 0.5 - index / numCols
point = fmeobjects.FMEFeature()
point.setAttribute('_count', count)
point.setAttribute('_num_columns', numCols)
point.setAttribute('_num_rows', numRows)
point.setGeometry(fmeobjects.FMEPoint(x, y, z))
self.pyoutput(point)
index += 1

3rd Implementation: inserted a PointCloudCombiner to
transform 3D Points into a Point Cloud before rasterizing.
About 12 sec.: Performance has been improved further more.
Is it the best performance?

Special Thanks: The Point Cloud Technology was suggested by
Dale and Chris@Safe.

Final Implementation: updated the Python script so that it will
create a 3D Line having all the 3D Grid Points as vertices.
About 2.5 sec.: Quite faster than creating many point
features.
import fmeobjects
def create3DLine(feature):
numCols = int(feature.getAttribute('_num_columns'))
numRows = int(feature.getAttribute('_num_rows'))
startCol = int(feature.getAttribute('_start{0}'))
startRow = int(feature.getAttribute('_start{1}'))
tuples = feature.getAttribute('DEM.coverage.rangeSet.DataBlock.tupleList').strip()
index = startRow * numCols + startCol
coords = []
for z in [float(t.split(',')[1]) for t in tuples.split()]:
x = index % numCols + 0.5
y = numRows - 0.5 - index / numCols
coords.append((x, y, z))
index += 1
feature.setGeometry(fmeobjects.FMELine(coords))

Level
Processing Time
Comments

1st
110 sec. (100%)
Existing Transformers Only

2nd
25 sec. (23%)
Create 3D Grid Points with Python

3rd
12 sec. (11%)
Introduced the Point Cloud Technology

Final
2.5 sec. (2.3%)
Create 3D Line with Python

Tested with FGD 10m Mesh DEM: FG-GML-5338-41-dem10b-20090201.xml
FME 2014 SP2 WIN32, Windows 7 64-bit

Conclusion:
•  Python scripting is worth to consider for improving performance, especially
if the ListExploder looks inefficient.
•  Using a Point Cloud as input for the NumericRasterizer is very efficient.

Summary

Rasterizing Result:

Final Workspace

Thank You!
!  Questions?
!  For more information:
!  Takashi Iijima iijima@pragmatica.jp
!  Pragmatica Inc.
!  http://fme-memorandum-takashi.blogspot.jp/

FME Workspace Template Collection for Reading Fundamental Geographic Data in Japan

Recommandé

Recommandé

Contenu connexe

Similaire à FME Workspace Template Collection for Reading Fundamental Geographic Data in Japan

Similaire à FME Workspace Template Collection for Reading Fundamental Geographic Data in Japan (20)

Plus de Safe Software

Plus de Safe Software (20)

Dernier

Dernier (20)

FME Workspace Template Collection for Reading Fundamental Geographic Data in Japan