Contenu connexe Similaire à Mobile Mapping Spatial Database Framework Similaire à Mobile Mapping Spatial Database Framework (20) Plus de Conor Mc Elhinney Plus de Conor Mc Elhinney (10) Mobile Mapping Spatial Database Framework1. Mobile Terrestrial LiDAR Datasets
in a Spatial Database Framework
Dr. Conor Mc Elhinney
Postdoctoral Researcher
Mobile Mapping Group
7th MMT 16th June 2011
7. Survey based
LiDAR folder
Survey 10 Apr
Block 1
Block 2
Block 3
.
.
Block N
.
MetaData: Geo Bounds, date,
processing done
8. Survey based
LiDAR folder
Survey 10 Apr Survey 5 Dec
Block 1 Block 1
Block 2 Block 2
Block 3 Block 3
. .
. .
Block N
. Block N
.
MetaData: Geo Bounds, date,
MetaData: Geo Bounds, date,
processing done
processing done
9. Survey based
LiDAR folder
Survey 10 Apr Survey 5 Dec Survey 2 May
Block 1 Block 1 Block 1
Block 2 Block 2 ....... Block 2
Block 3 Block 3 Block 3
. . .
. . .
Block N
. Block N
. Block N
.
MetaData: Geo Bounds, date,
MetaData: Geo Bounds, date, MetaData: Geo Bounds, date,
processing done
processing done processing done
12. Survey based
Give me the data from Dublin city
LiDAR folder
13. Survey based
Do we have data in a given area?
LiDAR folder
14. Survey based
Give me 10mx5m cross sections at
5m intervals
LiDAR folder
20. Storage
PostgreSQL
PostGIS
Database
21. Storage
PostGIS PostgreSQL
Database
23. Storage
DB Index: 3D point
Spatial Database
30. Why is upload an issue
130GB
80Km – 1 way
1,300
million points
33. First tests
DB on powerful desktop
> 60m records
> 4hrs to upload
34. First tests
DB on powerful desktop
Small survey >40hrs
> 60m records
> 4hrs to upload time
upload
35. Our hardware
1 Processing Server
8 Intel Xeons, 2.8 GHz
32 GBs RAM
1 Storage Server
7TBs Raided Drives
38. Test files
LiDAR data over 66m rows
2 files:
10 columns -> 4.4Gbs
14 columns -> 6.8Gbs
39. Postgresql upload
1. Create Table
2. Load data
3. Create Geometry Column
4. Update Geometry Column
5. Create Index
6. Vacuum Table
40. Our upload process
1. Pre-process - Python
2. Create Table
3. Create Geometry Column
4. Create Index
5. Load data
41. Time per row
PostgreSQL
Exp 1
Exp 2
Exp 3
Copy 10
pg_bulkload columns
0.025 0.05 0.075 0.1
Time per row (ms)
Copy
0.25 0.5 0.75 1 1.25
14
pg_bulkload columns
0.025 0.05 0.075 0.1
Time per row (ms)
42. Time per row Exp 1
PostgreSQL
Exp 1
2
New
Exp 2
3
Exp 3
Copy 10
pg_bulkload columns
0.025 0.05 0.075 0.1
Time per row (ms)
Copy 0.25
0.25
0.5
0.5
0.75
0.75
1
1
1.25
1.25
14
pg_bulkload columns
0.025 0.05 0.075 0.1
Time per row (ms)
43. Row size impacts time Exp 1
PostgreSQL
1
Exp 2
New2
Exp 3
Exp 3
Copy 10
pg_bulkload columns
0.25 0.5 0.75 1 1.25
Time per KB (ms)
Copy 0.25
0.25
0.5
0.5
0.75
0.75
1
1
1.25
1.25
14
pg_bulkload columns
0.25 0.5 0.75 1 1.25
Time per KB (ms)
44. Row size impacts time Exp 1
PostgreSQL
1
Exp 2
New2
Exp 3
Exp 3
Copy 10
pg_bulkload columns
0.25 0.5 0.75 1 1.25
Time per KB (ms)
Copy 0.25
0.25
0.5
0.5
0.75
0.75
1
1
1.25
1.25
14
pg_bulkload columns
0.25 0.5 0.75 1 1.25
Time per KB (ms)
45. Row size impacts time Exp 1
PostgreSQL
1
Exp 2
New2
Exp 3
Exp 3
Copy 10
pg_bulkload columns
0.25 0.5 0.75 1 1.25
Time per KB (ms)
Copy 0.25
0.25
0.5
0.5
0.75
0.75
1
1
1.25
1.25
14
pg_bulkload columns
0.25 0.5 0.75 1 1.25
Time per KB (ms)
46. Row size impacts time Exp 1
PostgreSQL
1
Exp 2
New2
Exp 3
Exp 3
Copy 10
pg_bulkload
As row size
0.25 0.5 0.75 1 1.25
columns
Time per kb
Time per KB (ms)
Copy 0.25
0.25
0.5
0.5
0.75
0.75
1
1
1.25
1.25
14
pg_bulkload columns
0.25 0.5 0.75 1 1.25
Time per KB (ms)
49. 60
1.5 1.5
Copy Orig
Copy
Direct
50 Parallel
1 40 1
Time (hours)
30
0.5 0.5
20
0 10 0
0
0 500 1000 1500 2000
Rows (millions)
50. 60
1.5 1.5
Copy Orig
Copy
Direct New
Copy
50 Parallel
1 40 1
Time (hours)
30
0.5 0.5
20
0 10 0
0
0 500 1000 1500 2000
Rows (millions)
51. 60
1.5 1.5
Copy Orig
Copy
Direct New
Copy
50 Parallel
1 40 1
Time (hours)
30
0.5 0.5
20
0 10 0
0
0 500 1000 1500 2000
Rows (millions)
52. 60
>24hrs
1.5 1.5
Copy Orig
Copy
Direct New
Copy
50 Parallel
1 40 1
Time (hours)
30
0.5 0.5
20
0 10 0
0
0 500 1000 1500 2000
Rows (millions)
53. 60
>24hrs
1.5 1.5
Copy Orig
Copy
Direct New
Copy
50 PG_bulkload
Parallel
1 40 1
Time (hours)
30
0.5 0.5
20
0 10 0
0
0 500 1000 1500 2000
Rows (millions)
54. 60
4hrs
1.5 1.5
Copy Orig
Copy
Direct New
Copy
50 PG_bulkload
Parallel
1 40 1
Time (hours)
30
0.5 0.5
20
0 10 0
0
0 500 1000 1500 2000
Rows (millions)
56. Access “Mobile Mapping System LiDAR Data
Framework “
3D GeoInfo 2010
Spatial Database
57. Access “Mobile Mapping System LiDAR Data
Framework “
3D GeoInfo 2010
Spatial Database
58. Access “Mobile Mapping System LiDAR Data
Framework “
3D GeoInfo 2010
Spatial Database
59. Access “Mobile Mapping System LiDAR Data
Framework “
3D GeoInfo 2010
Spatial Database
65. Process
> 140km
Spatial Database
67. Conclusions
Store
Access
Automatically
Process
Visualise
68. Future work
Finalise DB schema
Formalise DB Upload / Access
Release Free Mobile Mapping
Spatial DB
Automated algorithms
69. Questions
Conor Mc Elhinney, Paul Lewis, Tim McCarthy
conormce@cs.nuim.ie