2. How do you organise your data ?
The problem :
lots of dierent les, formats : csv, txt, xls, shp, asc ...
in dierent folders more or less well organised and easy to nd
lack of descriptions (source, stations, projection, last version ...)
reproducible science ?
easy sharing ?
easy backup ?
Do you need a database ?
not so dicult to built
enforce you to organize you data
save you a lot of time in the long run
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 2 / 25
3. How do you organise your data ?
The problem :
lots of dierent les, formats : csv, txt, xls, shp, asc ...
in dierent folders more or less well organised and easy to nd
lack of descriptions (source, stations, projection, last version ...)
reproducible science ?
easy sharing ?
easy backup ?
Do you need a database ?
not so dicult to built
enforce you to organize you data
save you a lot of time in the long run
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 2 / 25
4. How do you organise your data ?
The problem :
lots of dierent les, formats : csv, txt, xls, shp, asc ...
in dierent folders more or less well organised and easy to nd
lack of descriptions (source, stations, projection, last version ...)
reproducible science ?
easy sharing ?
easy backup ?
Do you need a database ?
not so dicult to built
enforce you to organize you data
save you a lot of time in the long run
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 2 / 25
5. How do you organise your data ?
The problem :
lots of dierent les, formats : csv, txt, xls, shp, asc ...
in dierent folders more or less well organised and easy to nd
lack of descriptions (source, stations, projection, last version ...)
reproducible science ?
easy sharing ?
easy backup ?
Do you need a database ?
not so dicult to built
enforce you to organize you data
save you a lot of time in the long run
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 2 / 25
6. How do you organise your data ?
The problem :
lots of dierent les, formats : csv, txt, xls, shp, asc ...
in dierent folders more or less well organised and easy to nd
lack of descriptions (source, stations, projection, last version ...)
reproducible science ?
easy sharing ?
easy backup ?
Do you need a database ?
not so dicult to built
enforce you to organize you data
save you a lot of time in the long run
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 2 / 25
7. How do you organise your data ?
The problem :
lots of dierent les, formats : csv, txt, xls, shp, asc ...
in dierent folders more or less well organised and easy to nd
lack of descriptions (source, stations, projection, last version ...)
reproducible science ?
easy sharing ?
easy backup ?
Do you need a database ?
not so dicult to built
enforce you to organize you data
save you a lot of time in the long run
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 2 / 25
8. How do you organise your data ?
The problem :
lots of dierent les, formats : csv, txt, xls, shp, asc ...
in dierent folders more or less well organised and easy to nd
lack of descriptions (source, stations, projection, last version ...)
reproducible science ?
easy sharing ?
easy backup ?
Do you need a database ?
not so dicult to built
enforce you to organize you data
save you a lot of time in the long run
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 2 / 25
9. What Is PostGIS ?
Open source object-relational database management system (ORDBMS)
Widely considered as the most full-featured open-source database system.
PostGIS = database extender for the PostgreSQL Database Management System
open source, freely available, fairly OGC compliant
In a nutshell it adds spatial functions such as :
distance, area, union, intersection
specialty geometry data types to the database
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 3 / 25
10. Installing PostgreSQL with PostGIS Functionality
Install PostgreSQL rst, then PostGIS extension (using Stack Builder)
http://www.postgresql.org/download/
http://postgis.refractions.net/download/
Windows
http://www.postgresql.org/download/windows
Mac
http://www.kyngchaos.com/software:postgres
Ubuntu
sudo apt-get install postgresql postgis
psql -d template1 -c alter user postgres with password 'postgres'
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 4 / 25
11. Creating a database
Using pgAdminIII interface
Right clic on database
or using SQL query
CREATE DATABASE blavet
WITH ENCODING=' UTF8 '
OWNER=postgres
LC_COLLATE=' fr_FR . UTF -8 '
LC_CTYPE=' fr_FR . UTF -8 '
CONNECTION LIMIT =−1;
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 5 / 25
12. Adding spatial extension
Using pgAdminIII interface
Right clic on extensions
or using SQL query
CREATE EXTENSION postgis
VERSION 2.0.1
Check the table spatial_ref_sys
Check also that you have lots of new functions
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 6 / 25
13. Organising a database
How would you organise a database ? Having :
Runo time series of dierent basins
Rainfall time series of dierent raingauges
Spatial description of those measurements
Other spatial informations (like soil type, geology...)
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 7 / 25
14. Adding tables
Table of basins
CREATE TABLE basins
(
id_basin serial NOT NULL ,
name text ,
city text ,
altitude numeric ,
area numeric ,
x_lambert2e numeric ,
y_lambert2e numeric ,
CONSTRAINT basins_pkey PRIMARY KEY ( id_basin )
)
WITH (
OIDS=TRUE
);
ALTER TABLE basins
OWNER TO postgres ;
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 8 / 25
15. Adding tables
Table of measures of runo
CREATE TABLE measures_runoff
(
id_runoff serial NOT NULL ,
id_basin integer NOT NULL ,
datehourmeasure timestamp with time zone NOT NULL ,
runoff numeric ,
timestep text ,
unit text ,
source text ,
CONSTRAINT measures_runoff_pkey PRIMARY KEY ( id_runoff ,
id_basin , datehourmeasure )
)
WITH (
OIDS=FALSE
);
ALTER TABLE measures_runoff
OWNER TO postgres ;
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 9 / 25
16. Adding tables
Table of raingauges
CREATE TABLE raingauges
(
id_raingauge serial ,
id_meteofrance integer ,
city text ,
name_raingauge text ,
altitude numeric ,
type_raingauge text ,
producer text ,
latitude_dec numeric ,
longitude_dec numeric ,
CONSTRAINT raingauge_pkey PRIMARY KEY ( id_raingauge )
)
WITH (
OIDS=TRUE
);
ALTER TABLE raingauges
OWNER TO postgres ;
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 10 / 25
17. Adding tables
Table of measures of rainfall
CREATE TABLE measures_rainfall
(
id_rainfall serial ,
id_raingauge integer ,
datehourmeasure timestamp with time zone ,
rainfall numeric ,
timestep text ,
unit text ,
CONSTRAINT rainfall_pkey PRIMARY KEY ( id_rainfall ,
id_raingauge , datehourmeasure )
)
WITH (
OIDS=FALSE
);
ALTER TABLE measures_rainfall
OWNER TO postgres ;
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 11 / 25
18. Adding tables
Table of antilope points
CREATE TABLE antilope_points
(
id_point numeric NOT NULL ,
x_lambert2e numeric ,
y_lambert2e numeric ,
CONSTRAINT antilope_points_pkey PRIMARY KEY ( id_point )
)
WITH (
OIDS=TRUE
);
ALTER TABLE antilope_points
OWNER TO postgres ;
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 12 / 25
19. Adding tables
Table of measures of antilope points
CREATE TABLE measures_rainfall_antilope
(
id_measure serial ,
id_point integer ,
datehourmeasure timestamp with time zone ,
rainfall numeric ,
units text ,
CONSTRAINT measures_rainfall_antilope_pkey PRIMARY KEY (
id_measure )
)
WITH (
OIDS=FALSE
);
ALTER TABLE measures_rainfall_antilope
OWNER TO postgres ;
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 13 / 25
20. Filling tables
Import les to database
COPY basins ( id_basin , name , city , altitude , x_lambert2e , y_lambert2e )
FROM ' .../ Intro_to_PostGIS / basins . csv ' with delimiter '; ' ;
COPY measures_runoff ( id_basin , datehourmeasure , runoff , timestep ,
unit ) FROM ' .../ Intro_to_PostGIS / runoff . csv ' with delimiter '
;' ;
COPY raingauges ( id_raingauge , id_meteofrance , city , name_raingauge ,
altitude , latitude_dms , longitude_dms , type_raingauge , producer ,
latitude_dec , longitude_dec ) FROM ' .../ Intro_to_PostGIS /
raingauges . csv ' with delimiter '; ' ;
COPY measures_rainfall ( id_raingauge , datehourmeasure , rainfall ,
timestep , unit ) FROM ' .../ Intro_to_PostGIS / rainfall . csv ' with
delimiter '; ' ;
COPY antilope_points ( id_point , x_lambert2e , y_lambert2e ) FROM ' .../
Intro_to_PostGIS / antilope_points . csv ' with delimiter '; ' ;
COPY measures_rainfall_antilope ( id_point , datehourmeasure , rainfall
, units ) FROM ' .../ Intro_to_PostGIS / antilope_rainfall . csv '
with delimiter '; ' ;
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 14 / 25
21. Adding internal spatial information
AddGeometryColumn
AddGeometryColumn ( schema_name , table_name , column_name , srid ,
type , dimension )
(Source : JUMP, Technical Report)
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 15 / 25
22. Adding internal spatial information
(Source : OpenGIS Simple Features Specication for SQL)
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 16 / 25
23. Geometry Constructors and Geometry Accessors
ST_MakePoint
ST_MakePoint ( x , y , z )
Full list here : Geometry Constructors
ST_X, ST_Y and ST_Z
ST_X ( geometry )
ST_Y ( geometry )
ST_Z ( geometry )
Full list here : Geometry Accessors
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 17 / 25
24. Geometry Editors
ST_SetSRID
ST_SetSRID ( geom , srid )
ST_Transform
ST_Transform ( geom , srid )
Full list here : Geometry Editors
A Spatial Reference System Identier (SRID) is a unique value used to unambiguously
identify projected, unprojected, and local spatial coordinate system denitions.
Large database of SRID created by the European Petroleum Survey Group (EPSG) is
widely used.
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 18 / 25
25. Spatial Relationships and Measurements
Spatial Relationships and Measurements
ST_Area ( geometry )
ST_Centroid ( geometry )
ST_Contains ( geometry , geometry )
ST_Distance ( geometry , geometry )
Full list here : Spatial Relationships and Measurements
Geometry Processing
ST_Buffer ( geometry , radius_of_buffer )
ST_Union ( geometry set g1field )
ST_Union ( geometry , geometry )
ST_Intersection ( geometry , geometry )
Full list here : Geometry Processing
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 19 / 25
26. Adding internal spatial information
Adding geometry columns
−−A d d i n g g e o m e t r y c o l u m n s
SELECT AddGeometryColumn ( ' basins ' , ' outlet_lambert2e ' , 2 7 5 7 2 , '
POINT ' , 2 ) ;
SELECT AddGeometryColumn ( ' basins ' , ' outlet_lambert93 ' , 2 1 5 4 , '
POINT ' , 2 ) ;
SELECT AddGeometryColumn ( ' basins ' , ' limits_lambert2e ' , 2 7 5 7 2 , '
POLYGON ' , 2 ) ;
SELECT AddGeometryColumn ( ' basins ' , ' limits_lambert93 ' , 2 1 5 4 , '
POLYGON ' , 2 ) ;
SELECT AddGeometryColumn ( ' raingauges ' , ' raingauges_lambert2e ' ,
2 7 5 7 2 , ' POINT ' , 2 ) ;
SELECT AddGeometryColumn ( ' raingauges ' , ' raingauges_lambert93 ' ,
2 1 5 4 , ' POINT ' , 2 ) ;
SELECT AddGeometryColumn ( ' antilope_points ' , '
antilope_points_lambert2e ' , 2 7 5 7 2 , ' POINT ' , 2 ) ;
SELECT AddGeometryColumn ( ' antilope_points ' , '
antilope_points_lambert93 ' , 2 1 5 4 , ' POINT ' , 2 ) ;
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 20 / 25
27. Adding internal spatial information
Filling geometry columns
−− F i l l g e o m e t r y c o l u m n s
UPDATE basins SET outlet_lambert2e=st_setsrid ( st_makepoint (
x_lambert2e , y_lambert2e ) , 2 7 5 7 2 ) ;
UPDATE basins SET outlet_lambert93=st_transform ( outlet_lambert2e
,2154) ;
UPDATE raingauges SET raingauges_lambert2e=st_transform (
st_setsrid ( st_makepoint ( longitude_dec , latitude_dec ) , 4326)
,27572) ;
UPDATE raingauges SET raingauges_lambert93=st_transform (
st_setsrid ( st_makepoint ( longitude_dec , latitude_dec ) , 4326)
,2154) ;
UPDATE antilope_points SET antilope_points_lambert2e=st_setsrid (
st_makepoint ( x_lambert2e , y_lambert2e ) , 2 7 5 7 2 ) ;
UPDATE antilope_points SET antilope_points_lambert93=st_transform
( st_setsrid ( st_makepoint ( x_lambert2e , y_lambert2e ) , 2 7 5 7 2 )
,2154) ;
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 21 / 25
28. Adding external spatial information
Using graphical interface
Using Quantum-GIS extension (like QGIS spit)
pgShapeLoader
...
Using command line with shp2pgsql
shp2pgsql -iIDd -s 2154 -W utf8 basins_lambert93.shp basins_polygon impo
psql -U postgres -W -h localhost -p 5432 -d blavet -f import.sql
OR
shp2pgsql -iIDd -s 2154 -W utf8 basins_lambert93.shp basins_polygon |
psql -U postgres -W -h localhost -p 5432 -d blavet
Filling geometry columns
UPDATE basins SET limits_lambert93=st_geometryn ( geom , 1 )
FROM basins_polygon
WHERE basins_polygon . id_basin=basins . id_basin ;
and limits_lambert2e ...
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 22 / 25
29. Exemple SQL of query
Select a chronique of runo
Calculate the area of one basin
Select raingauges inside one basin
Select average antilope rainfall time serie for each basins
Import shapele geology and get a table of statistics of geology for each basin
Check if there is double measures in runo and rainfall
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 23 / 25
30. Visualizing your data
Using Qgis to vizualize your data
Plugin in Qgis for query and shapele creation
Plugin in Qgis for importing data
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 24 / 25
31. Connect to your database with R
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 25 / 25
32. Connect to your database with R
A. de Lavenne (INRA, Rennes) University of Trento 17 décembre 2012 25 / 25