Margriet Groenendijk - Open data is available from an incredible number of data sources that can be linked to your own datasets. This talk will present examples of how to visualise and combine data from very different sources such as weather and climate, and statistics collected by individual countries using Python notebooks in Analytics for Apache Spark.

1. Margriet Groenendijk, PhD
Developer Advocate for IBM Cloud Data Services
Connecting and Visualising Open Data from Multiple Sources
Data Driven Innovation Open Summit
Rome - 20 May 2016
@MargrietGr

2. Please Note
▪ IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion.
▪ Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making
a purchasing decision.
▪ The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material,
code or functionality. Information about potential future products may not be incorporated into any contract.
▪ The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.
▪ Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual
throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the
amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed.
Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
@MargrietGr

3. About me
• Developer Advocate at IBM Cloud Data Services, UK
• Data scientist
• Python, R, Cloudant, dashDB
• Research Fellow at University of Exeter, UK
• Worked with very large observational datasets and the output of
global scale climate models
• PhD at Vrije Universiteit Amsterdam, the Netherlands
• Explored large observational datasets of carbon uptake by forests
@MargrietGr

4. Outline

5. Connect and Visualise Data
@MargrietGr
But the first step - getting the data in, in a way you can use it - takes up
most of the time
I have spend most of my time just doing this for the last 10 years
In March I joined IBM and I started exploring better and easier ways of
data use and analysis

6. @MargrietGr
http://geoawesomeness.com/wp-content/uploads/2015/10/GoogeMaps-vs-OSM-Geoawesomeness.jpg
• Freely available
• Constantly updated by
local volunteers
• Data format needs
some processing

7. Weather and Climate Data
@MargrietGr
There is a lot of it and the files are large
Binary data format of grids in different
shapes and sizes
Clear understanding of where the data
comes from is important. Most of it is
generated by models or through
interpolation of observations

8. Census Data
@MargrietGr
Demographic, economic
an statistical data by
country
For US also by state and
city
Accessible through APIs

9. OpenStreetMap Data

10. OpenStreetMap is built by a community of mappers that contribute and maintain data
about roads, trails, cafés, railway stations, and much more, all over the world
Weekly updated
But… large files that can do with some processing to make the data easily accessible
@MargrietGr
https://www.openstreetmap.org
https://www.cloudant.com
use anywhereIBM Cloudant

11. Several data sources - world, continent, country, city or a user defined box
Several data formats for which free to use conversion tools exist - pbf, osm, json, shp
Example for the Netherlands:
@MargrietGr
wget -c http://download.geofabrik.de/europe/netherlands-
latest.osm.pbf
use anywhereIBM Cloudant

12. Extract the POIs with osmosis
@MargrietGr
osmosis --read-pbf netherlands-latest.osm.pbf
--tf accept-nodes
aerialway=station
aeroway=aerodrome,helipad,heliport
amenity=* craft=* emergency=*
highway=bus_stop,rest_area,services
historic=* leisure=* office=*
public_transport=stop_position,stop_area
shop=* tourism=*
--tf reject-ways --tf reject-relations
--write-xml netherlands.nodes.osm
(easy to install with brew on Mac)

13. Some cleaning up with osmconvert
Convert from osm to json format with ogr2ogr
@MargrietGr
osmconvert $netherlands.nodes.osm
--drop-ways --drop-author --drop-relations
--drop-versions >$netherlands.poi.osm
ogr2ogr -f GeoJSON $netherlands.poi.json
$netherlands.poi.osm points

14. Create an account on www.cloudant.com
(free trial available)
Upload to Cloudant with couchimport
@MargrietGr
export COUCH_URL="https://
username:password@username.cloudant.com"
cat $netherlands.poi.json | couchimport
--db poi-$netherlands --type json --jsonpath "features.*"
https://github.com/glynnbird/couchimport
IBM Cloudant

15. ▪ Cloudant screen shot…
@MargrietGr

16. ▪ Cloudant screen shot…
@MargrietGr

17. ▪ Cloudant screen shot…
@MargrietGr

18. Examples from https://docs.cloudant.com/geo.html
Easily accessible in Python notebook by with the requests package
@MargrietGr
use anywhere!IBM Cloudant

19. @MargrietGr
use anywhereIBM Cloudant
Weekly updates
Adapt the code and
automate it to run
weekly
Up to date database

20. Weather and Climate Data

21. Weather and Climate Data
@MargrietGr
There is a lot of it and the files are large
Binary data format of grids in different
shapes and sizes
http://www.cru.uea.ac.uk/data/
https://modelingguru.nasa.gov/docs/DOC-2312

22. https://developer.ibm.com/clouddataservices/2016/04/18/predict-temperatures-using-dashdb-python-and-r/
@MargrietGr
Weather and Climate Data
The below blog explains how to process some example data and load it
into a relation database (dashDB)
This data is now easily accessible

23. Load data into Python directly from dashDB
(credentials are easily found in dashDB)
@MargrietGr
from ibmdpy import IdaDataBase, IdaDataFrame
jdbc = "jdbc:db2://dashdb-entry-yp-
dal09-09.services.dal.bluemix.net:50000/BLUDB:user="
+ username + ";password=" + password
idadb = IdaDataBase(jdbc)

24. @MargrietGr
Average global temperature
import pandas as pd
temp = pd.read_csv("temperature.csv")
temp[0:5]

25. @MargrietGr
From 2D to 3D matrix
import numpy as np
# Determine the size of the 3D matrix
lats = np.unique(temp.latitude)
lons = np.unique(temp.longitude)
nt = 12
ni = len(lats)
nj = len(lons)

26. @MargrietGr
From 2D to 3D matrix
# Create and fill matrix by looping over the 3 dimensions
temperature = np.zeros(nt*ni*nj)
temperature.shape = [nt, ni, nj]
mo = -1
for mon in range(1,13):
mo = mo+1
la = -1
for lat in lats:
la = la+1
lo = -1
for lon in lons:
lo = lo+1
t = temp["temperature"][(temp["month"]==mon) &
(temp["latitude"]==lat) &
(temp["longitude"]==lon)]
temperature[mo, la, lo] = np.array(t)

27. @MargrietGr
import scipy
import matplotlib
from pylab import *
from mpl_toolkits.basemap import Basemap, addcyclic, shiftgrid,
maskoceans

28. @MargrietGr
import scipy
import matplotlib
from pylab import *
from mpl_toolkits.basemap import Basemap, addcyclic, shiftgrid,
maskoceans
# define the area to plot and projection to use
m =
Basemap(llcrnrlon=-180,llcrnrlat=-60,urcrnrlon=180,urcrnrlat=80,
projection='mill')

29. @MargrietGr
Global temperature mapimport scipy
import matplotlib
from pylab import *
from mpl_toolkits.basemap import Basemap, addcyclic, shiftgrid,
maskoceans
# define the area to plot and projection to use
m =
Basemap(llcrnrlon=-180,llcrnrlat=-60,urcrnrlon=180,urcrnrlat=80,
projection='mill')
# covert the latitude, longitude and temperatures to raster
coordinates to be plotted
t1=temperature[0,:,:]
t1,lon=addcyclic(t1,lons)
january,longitude=shiftgrid(180.,t1,lon,start=False)
x,y=np.meshgrid(longitude,lats)
px,py=m(x,y)

30. @MargrietGr
rcParams['font.size']=12
rcParams['figure.figsize']=[8.0, 6.0]
figure()

31. @MargrietGr
rcParams['font.size']=12
rcParams['figure.figsize']=[8.0, 6.0]
figure()
palette=cm.RdYlBu_r
rmin=-30.; rmax=30.
ncont=20
dc=(rmax-rmin)/ncont
vc=arange(rmin,rmax+dc,dc)
pal_norm=matplotlib.colors.Normalize(vmin = rmin, vmax = rmax, clip =
False)

32. @MargrietGr
Global temperature maprcParams['font.size']=12
rcParams['figure.figsize']=[8.0, 6.0]
figure()
palette=cm.RdYlBu_r
rmin=-30.; rmax=30.
ncont=20
dc=(rmax-rmin)/ncont
vc=arange(rmin,rmax+dc,dc)
pal_norm=matplotlib.colors.Normalize(vmin = rmin, vmax = rmax, clip =
False)
m.drawcoastlines(linewidth=0.5)
m.drawmapboundary(fill_color=(1.0,1.0,1.0))
cf=m.pcolormesh(px, py, january, cmap = palette)
cbar=colorbar(cf,orientation='horizontal', shrink=0.95)
cbar.set_label('Mean Temperature in January')
tight_layout()
show()

33. @MargrietGr

34. UN Census data
https://console.ng.bluemix.net/data/exchange

35. Census Data
@MargrietGr
Demographic, economic
an statistical data by
country
For US also by state and
city
Accessible through APIs

36. 36

37. 37

38. @MargrietGr

39. 39

40. 40

41. 41
——————————

42. @MargrietGr
import urllib
filelink=urllib.urlopen(“https://
console.ng.bluemix.net/data/exchange-api/v1/entries/
889ca053a19986a4445839358a91963e/data?
accessKey=xxxxxx")
popdf = pd.read_csv(filelink)
list(popdf)
['Country or Area', 'Year', 'Value', 'Value Footnotes']

43. @MargrietGr
popdf[0:10]

44. Combine and visualise

45. Combine and Visualise
▪ POI data in Cloudant
▪ Weather data in dashDB
▪ Census data
@MargrietGr

46. In the cloud: Data & Analytics on IBM Bluemix
@MargrietGr

47. https://www.datascientistworkbench.com
@MargrietGr

48. Key points
▪ There is lots of data freely available
▪ A lot of analysis tools are free, with examples in blogs and on Github
▪ There is still lots of preparation needed before doing any analysis or visualisation
▪ But this getting easier and easier
▪ API access of data
▪ Data storage, analysis and visualisation in the cloud
@MargrietGr

49. https://github.com/MargrietGroenendijk/notebooks
Thank you!
@MargrietGr
Margriet Groenendijk
Developer Advocate for IBM Cloud Data Services