This document discusses how FME can be used to work with big data. It provides examples of using FME to load and export data from Amazon DynamoDB and MarkLogic databases. The DynamoDB demo shows how to spatially index and store any type of data in DynamoDB. The MarkLogic demos demonstrate using FME to convert data to XML/GML and load it into MarkLogic, and to query MarkLogic and export XML results for conversion to other formats. Big data provides opportunities for FME, and the cloud model is a natural fit for working with large datasets.

1. CONNECT. TRANSFORM. AUTOMATE.
Big Data Meets FME

2. Agenda
 What is Big Data
 Big Data Challenges
 FME and Big Data
 Big Data Technologies
 DynamoDB Workflow
 MarkLogic Workflow

4. Big Data and Cloud
Big Data needs big resources
 Big data stores
 Big processing power
 Big bandwidth
Cloud technology gives you this for fraction of
traditional cost!

5. Big Data and FME
 Big Data is a new data
“classification” for FME.
 Big Data is no different than
other data to FME
 FME Cloud is a natural fit for
data in the Cloud
FME makes it easy to leverage the power of Big Data

6. Big Data and FME Support
Amazon S3
 Limitless internet based
storage
Amazon RDS
 See blog article on Amazon RDS (PostGIS/SQLServer/Oracle)
Amazon DynamoDB
 NoSQL limitless database service
Amazon RedShift
 Petabyte scale database warehouse service.
Google BigQuery
 Superfast append only tables
MarkLogic
 Large XML based NoSQL database

7. Big Data Challenges
 Loading Data
 Lacks Spatial Support
 Big Data Analysis
 Querying and Exporting Data

8. Why Demo FME with
MarkLogic and DynamoDB?
Different from other
databases supported by
FME

9. Demo #1 – Limitless Spatial Database

10. DynamoDB
 NoSQL SSD-based database service
 No limit on size of database
 Specify the needed performance
 Autoscale thru Dynamic DynamoDB
 Amazon EMR (Hadoop) integration

11. Dynamodb Big Data Demo
Spatially locate and store
anything in DynamoDB!

12. Dynamodb Demo – Index Strategy
Generate GeoHash Index
for each feature and
Write to
GeoHashSpatialIndex

13. DynamoDB Demo –
Storing Vector, Raster, Lidar
Write small features
to DynamoDB
Write large features
to Amazon S3, link
to DynamoDB

14. DynamoDB Demo–
Storing Geocoded Images
Generate Geohash record
of picture location
Write Image to S3, link
to DynamoDB

15. DynamoDB Demo –
Spatially Locate and Store Any document or Web Resource
Generate Geohash
index
Write Document to
S3 and Link to
DynamoDB
location

16. DynamoDB Demo –
Retrieve any stored document
Write URI Link to
DynamoDB
Generate Geohash
index
location

17. What is ?
 NoSQL database – XML optimized
 Powerful search and analysis
 Native spatial support
 XML based data model (GML, XML, etc.)
 Deploy on Hadoop HDFS

18. FME and MarkLogic – A Natural Fit
 Convert data to XML/GML*
 Easily load XML into MarkLogic with FME
 Process and convert XML results
 FME 2014: New schema based GML Writer

19. Demo #1a - Loading MarkLogic
Convert GIS / CAD
data to GML (XML)
Compose REST request
to PUT to MarkLogic
database

20. 1.Convert GIS / CAD data into Valid GML
2.Generate Key Fields
3.Build insert message
4.Execute PUT REST call
MarkLogic accepts any valid XML – just PUT it!
Loading GIS to MarkLogic

21. Loading GIS to MarkLogic with FME

22. Demo #1b Exporting from MarkLogic
GET Query to find
URI’s for features
of interest
GET Query using URI’s to
get feature
XML/GML, then
Conversion to format of
choice (CAD, GIS …)
/WFS

23. Exporting XML from MarkLogic
1. Query database via GET request
2. Parse search result and compose GET feature request
3. Extract attributes and geometry from result
4. Validate and write XML Result

24. Exporting XML from MarkLogic
Search GET request to find URI based on query:
http://localhost:8003/v1/keyvalue?element=comment&value=AIXM.Chicago
Document Retrieval GET request based on URI:
http://localhost:8003/v1/documents?uri=/docs/myXML_653c46c3-fdfb-4837-ae1c-
49735dd29356.xml

25. AIXM from MarkLogic via FMEServer
http://UHURA/fmedatastreaming/Demos/QueryMarkLogicDB.
fmw?Element=airportCode&Value=CYVR
/AIXM

26. AIXM from MarkLogic via FMEServer

27. Summary
Big Data = big new opportunities
FME great for working with Big Data
Cloud model is a natural fit for Big Data
This is just the beginning - more to come!

28. Thank You!
 Questions?
 For more information:
 info@safe.com
 www.safe.com