Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Geo-spatial Research for Impact: Beyond Analysis  Towards Synthesis Ramesh Raskar Assoc. Professor MIT Media Lab with Nik...
Poverty, Pop-density, Crop, Methane .. maps
Google Street View
Online User Photos: Social voting in the real world Phototourism by Seitz at al
Analysis: Perceived Safety Naik, Raskar, Hidalgo 2016
Analysis  Synthesis
Street Address: Assign | Adopt • 75% of the world population without street addresses • Timely ambulance delivery • Stimul...
Beyond Analysis  Towards Synthesis Actionable Insights Traffic Nudge Crisis Response Street Addr Govt Policies Large Scal...
Occasional Frequent Low Quality Census High Quality Satellite, Street view Phone, Mobility, Social, Financial AV, PlanetLa...
Capture Analyze Act Low Level Mid Level High Level
Capture Analyze Act Low Level Street-level Measure •Visual Census •Counting/Density Alert •Traffic Interventions Mid Level...
Capture  Present  Future Low Level Street-level Street View Crowdsourcing (mapillary) Traffic Cameras Autonomous Vehicle...
Analyze Present  Future Low Level Measure •Visual Census •Counting/Density Streetscore, Streetchange, Visual Census (Fei...
Socio-Economic Inference from Digital Patterns Satellite ImageryStreet-level Imagery Aerial Imagery Phone Mobility Data So...
400K Google StreetViews Socio Economic and Physical environment
4k Images, 200K pairwise comparisons Place Pulse 1.0 Naik, Raskar, Hidalgo 2016
1.7K Ranked 400K Predicted
Naik, Raskar, Hidalgo 2016
New York Boston Detroit Chicago Validated with Crime Stats streetscore.media.mit.edu
105 Kent Avenue, Brooklyn, NY August 2007 Streetscore = 1.8/10
105 Kent Avenue, Brooklyn, NY September 2014 Streetscore = 7.2/10
Urban Growth in New York 2007 - 2014
Raskar, Camera Culture, MIT Media Lab http://cameraculture.info MIT Media Lab Ilke Demir, Nikhil Naik
Capture Analyze Act Low Level Street-level Measure •Visual Census •Counting/Density Alert •Traffic Interventions Mid Level...
Example: Congestion Pricing (Nikhil Naik et al.) • Jakarta: Response of vehicles by types • CV + traffic cameras across th...
Street Address: Assign | Adopt • 75% of the world population without street addresses • Timely ambulance delivery • Stimul...
Street Addresses from Satellite Imagery İlke Demir, Forest Hughes, Aman Raj, Kaunil Dhruv, Suryanarayana M. Muddala, Sanya...
Thanks Nikhil Naik Tristan Swedish Praneeth Vepakomma Ilke Demir Forest Hughes Jatin Malhotra SuryaNarayana Murthy Kavnil ...
Pilot 1: Will people use street names? In use After 6 monthsCommunity assigned names + Signs
Pilot 2 Will SMEs use the labeling scheme? Measured relative efficiency for Pizza delivery
Pilot 3 Suitable for E-commerce work-flow?
What3words parrot.casino.failed Robocodes 75D. Road NE27. Dhule.MhIn Landmarks Green Park Poor Coverage Lat/lon Goal: Memo...
What3words: A: parrot.casino.failed B: issuer.lollipop.ripe - Irrelevant words based on lat/lon. Robocodes: 75D.NE27.Dhule...
Addressing Schemes Around the World London postal code system: Radial regions based on orientation and distance South Kore...
Robocode Scheme • 5 alphanumeric fields • Hierarchical and linear descriptors • To close the gap between physical addresse...
Design Choices Linear: similar addresses stored in a linear fashion Hierarchical: top-down structure for spatial encapsula...
Our Pipeline Satellite Images Predictions Road Segments Clustering Segmentation RegionsRoad IDsMarkers + Blocks Labeling
Satellite Images ç • Irregular urban structure • Illumination/weather/country • Different road types
Segmentation • Binary road masks • 19K*19K, 0.5m/pixel • SegNet
Extracting Road Segments • Orientation based median filtering • Road segments by orientation bucketing
NF NH NE Region Creation • Road graph: Node=intersection, edge=road, weight=length • Partition for max inter, min intra co...
Region and Road Naming • Cmax 𝑟𝑜𝑎𝑑𝑠 𝐴 → 𝐶𝐴 (downtown) • Orientation bucketing into N, S, W, E • Trace regions based on dis...
Offsetting and Meter Marking • 5 meter marker along the road • Odd/even based on RHR • Distance field of roads: block offs...
Unmapped Developing Country Regions follow natural boundaries
Street Address with Robocodes • From Satellite Imagery to Deployed Street Addresses • Generative address : linear, hierarc...
Occasional Frequent Low Quality Census High Quality Satellite, Street view Phone, Mobility, Social, Financial AV, PlanetLa...
Capture Analyze  Act Low Level Street-level Measure •Visual Census •Counting/Density Alert •Traffic Interventions Mid Lev...
Act Alert Assist Change Low/Mi d/High Level Alert about the state of people/economy/b uilt environment (e.g., predict crop...
Inaccessible Areas • To extend our format to cover areas that are not accessible by streets, we explored different impleme...
f(C,lat,lon) = hash(round(lat,3)) + dir(lat) . hash(round(lon,3)) +dir(lon) . C L-A-T-dir.L-O-N-dir.name.area Inaccessible...
Thanks! What next? Today Tomorrow Friday A month • Robocode.info • Join our presentation in CVPR WiCV. Friday 10am • Join ...
Bonus: Blackholes! Main aim: f(<place>)=robocode Base case: <place> = <house, street, city, country> “12C.NA14.PALO.CAUS” ...
Region Experiments • Experimented with (a) normalized min-cut, (b) Newman- Girvan, (c) modularity based partitioning. • Ex...
Output Maps and Tools • .osm maps with roads (meter marking and offsetting on the fly) • ID-tool of MapBox for on-demand i...
Analyze: Three Types of Outputs 1. Semantic 2. Objective Population Density 76000/sq. mile 3. Qualitative Assign a semanti...
Geo-spatial Research: Transition from Analysis to Synthesis
Prochain SlideShare
Chargement dans…5
×

Geo-spatial Research: Transition from Analysis to Synthesis

308 vues

Publié le

We need to transition from analysis to synthesis when it comes to large scale image based studies of satellite or street level images.

Large scale, image based studies have the ability to unlock the human potential and really address some of the most important societal problems. The question really is, are we going to do that through analysis or are we going to step up to the game and actually start doing synthesis? Are we only go to study and observations or are we going to go and actually make an impact in the society?
Can global image repositories help UN's sustainable development goals (SDGs)? help us understand the social determinants of health? Satellite imagery, Google street view and user contributed photos from a global image repository are being used for large scale image-based studies, visual census and sentiment analysis [Ermon][http://StreetScore.media.mit.edu]. But we need to go beyond simply relying on big data for investigating social questions via remote analysis. We need to transition from analysis to synthesis. For deployable social solutions, we need to consider the full stack of physical devices, organizational interests and sector-specific resources.

Image-based large studies allow us to predict poverty from daytime and nighttime satellite imagery which can influence critical decisions for aid and development planning. In project ‘StreetScore’, our group has shown that semantic analysis of street level imagery such as Google Streetview, can provide varied insights rich in urban perception; our recent project ‘StreetChange’ shows the benefits of time-series data in driving these insights (http://streetchange.media.mit.edu).

We have seen some amazing work and you'll hear from Stephano about poverty mapping my glove previous collaborators to a population density crop maps, Betaine. So we had been, that's been fantastic progress in, in using a global industry, uh, in, in these areas that are taken from satellites or drones and then a street level imagery is also very widely available, either very structured like Google street view, but also from a user contributor photos and to that Nikki like and others in my group have been working on can we do a sentiment analysis of, of this imagery in this case, sentiment analysis of the perceived safety just for Google Street and main street and then create kind of citywide maps of a perceived safety that can be used by city planners and urban planners. So, which is great. But coming back to analysis versus synthesis opportunities, I'm going to give you a flavor of one of the projects we worked on a which is street addresses.

Publié dans : Technologie
no profile picture user

  • Identifiez-vous pour voir les commentaires

  • Soyez le premier à aimer ceci

Geo-spatial Research: Transition from Analysis to Synthesis

  1. 1. Geo-spatial Research for Impact: Beyond Analysis  Towards Synthesis Ramesh Raskar Assoc. Professor MIT Media Lab with Nikhil Naik Ilke Demir
  2. 2. Poverty, Pop-density, Crop, Methane .. maps
  3. 3. Google Street View
  4. 4. Online User Photos: Social voting in the real world Phototourism by Seitz at al
  5. 5. Analysis: Perceived Safety Naik, Raskar, Hidalgo 2016
  6. 6. Analysis  Synthesis
  7. 7. Street Address: Assign | Adopt • 75% of the world population without street addresses • Timely ambulance delivery • Stimulate digital economy via eCommerce • Protect property rights of the marginalized • Crisis response (Haiti, 48 hrs to coord aid)
  8. 8. Beyond Analysis  Towards Synthesis Actionable Insights Traffic Nudge Crisis Response Street Addr Govt Policies Large Scale Visual Study Visual Census Socio-eco observations Sentiment maps
  9. 9. Occasional Frequent Low Quality Census High Quality Satellite, Street view Phone, Mobility, Social, Financial AV, PlanetLabs etc Sense Process Respond
  10. 10. Capture Analyze Act Low Level Mid Level High Level
  11. 11. Capture Analyze Act Low Level Street-level Measure •Visual Census •Counting/Density Alert •Traffic Interventions Mid Level Aerial Understand •Geolocalization •Crowds •3D Modeling Assist •Crisis Response •Agriculture •Insurance/Hedge Funds High Level Satellite Predict • Economic activity Change •Street Addr •Government planning
  12. 12. Capture  Present  Future Low Level Street-level Street View Crowdsourcing (mapillary) Traffic Cameras Autonomous Vehicles Cloud Dashcams Mid Level Aerial Commercial airplanes Amateur Drones Autonomous Drones High Level Satellite Meter-resolution Limited coverage for high-res Sub-meter, real- time, entire Earth (Planet Lab, OneWeb) Ack: Nikhil Naik
  13. 13. Analyze Present  Future Low Level Measure •Visual Census •Counting/Density Streetscore, Streetchange, Visual Census (Fei-Fei Li) Generative modeling of cities Mid Level Understand •Geolocalization •Crowds •3D Modeling DeepRoadMap per (Urtasun et al.) Design Suggestions High Level Aggregate and Predict • Economic activity Jean and Ermon Jayachandran Deforestation Study Real-time data analysis. Actionable information
  14. 14. Socio-Economic Inference from Digital Patterns Satellite ImageryStreet-level Imagery Aerial Imagery Phone Mobility Data Social Networks, Photos Device Activity Built Environment Human Activity 84 countries, no data Ack: Nikhil Naik
  15. 15. 400K Google StreetViews Socio Economic and Physical environment
  16. 16. 4k Images, 200K pairwise comparisons Place Pulse 1.0 Naik, Raskar, Hidalgo 2016
  17. 17. 1.7K Ranked 400K Predicted
  18. 18. Naik, Raskar, Hidalgo 2016
  19. 19. New York Boston Detroit Chicago Validated with Crime Stats streetscore.media.mit.edu
  20. 20. 105 Kent Avenue, Brooklyn, NY August 2007 Streetscore = 1.8/10
  21. 21. 105 Kent Avenue, Brooklyn, NY September 2014 Streetscore = 7.2/10
  22. 22. Urban Growth in New York 2007 - 2014
  23. 23. Raskar, Camera Culture, MIT Media Lab http://cameraculture.info MIT Media Lab Ilke Demir, Nikhil Naik
  24. 24. Capture Analyze Act Low Level Street-level Measure •Visual Census •Counting/Density Alert •Traffic Interventions Mid Level Aerial Understand •Geolocalization •Crowds •3D Modeling Assist •Crisis Response •Agriculture •Insurance/Hedge Funds High Level Satellite Aggregate and Predict • Economic activity Change •Street Addr •Government planning
  25. 25. Example: Congestion Pricing (Nikhil Naik et al.) • Jakarta: Response of vehicles by types • CV + traffic cameras across the city to detect types + number of vehicles • Understand traffic flows before/after congestion pricing is introduced • Adjust rates for different types of vehicles/ in different areas
  26. 26. Street Address: Assign | Adopt • 75% of the world population without street addresses • Timely ambulance delivery • Stimulate digital economy via eCommerce • Protect property rights of the marginalized • Crisis response (Haiti, 48 hrs to coord aid)
  27. 27. Street Addresses from Satellite Imagery İlke Demir, Forest Hughes, Aman Raj, Kaunil Dhruv, Suryanarayana M. Muddala, Sanyam Garg, Barrett Doo, Ramesh Raskar IJCG 2018
  28. 28. Thanks Nikhil Naik Tristan Swedish Praneeth Vepakomma Ilke Demir Forest Hughes Jatin Malhotra SuryaNarayana Murthy Kavnil Dhruv Aman Raj Barrett Doo Praveen Gedam Anna Roy Cesar A. Hidalgo Guan Pang Jing Huang Daniel Aliaga Manohar Paluri Pierre Roux Yael Maguire Leo Tsourides Divyaa Ravichandran Sanyam Garg Sai Sri Sathya Grace Kermani Tobias Tiecke Andreas Gros Santanu Bhattacharya Kabir Rustogi Will Marshall
  29. 29. Pilot 1: Will people use street names? In use After 6 monthsCommunity assigned names + Signs
  30. 30. Pilot 2 Will SMEs use the labeling scheme? Measured relative efficiency for Pizza delivery
  31. 31. Pilot 3 Suitable for E-commerce work-flow?
  32. 32. What3words parrot.casino.failed Robocodes 75D. Road NE27. Dhule.MhIn Landmarks Green Park Poor Coverage Lat/lon Goal: Memorable Geocoding Physical 1050 Market St, Fresno, CA, US .
  33. 33. What3words: A: parrot.casino.failed B: issuer.lollipop.ripe - Irrelevant words based on lat/lon. Robocodes: 75D.NE27.Dhule.MhIn 76C.NE27.Dhule.MhIn - Hierarchical and linear addresses. Google Maps: Near Green Park Near Green Park - No street names or numbers Point vs Edges for Geometric Queries
  34. 34. Addressing Schemes Around the World London postal code system: Radial regions based on orientation and distance South Korea streets: Meter markers Japan block system: Hard to decipher Dubai addressing: Uses districts Berlin numbering: Zigzag house pattern
  35. 35. Robocode Scheme • 5 alphanumeric fields • Hierarchical and linear descriptors • To close the gap between physical addresses and automated geocoding Road naming scheme: - distance from the center - orientation in odd parity i.e. WB14 Region naming scheme: - orientation wrt downtown - distance from downtown i.e. WB House numbering scheme: - meter markers on the road - block letters from the road i.e. 38K WB14 “I7 Hacker Way, Menlo Park, CA, US”
  36. 36. Design Choices Linear: similar addresses stored in a linear fashion Hierarchical: top-down structure for spatial encapsulation Compressible: 5x4 max (chars x words) Universal: independent of local language Inquirable: useful for geometric, proximity-based, and type-ahead queries Extendible: dynamically modifiable for new places Robust: flexible for overestimation and noise StructuralDesignParameters forefficientcomputerimplementation Linear: closer addresses are given related names Hierarchical: top-down subdivision of the world Memorable: short and alphanumeric, easily convertible Intuitive: with a sense of direction and distance Topological: consistent with road topology Inclusive: with local names (city, state) Physical: consistent with natural boundaries SemanticDesignParameters foruserfriendliness Machine Needs Human Needs
  37. 37. Our Pipeline Satellite Images Predictions Road Segments Clustering Segmentation RegionsRoad IDsMarkers + Blocks Labeling
  38. 38. Satellite Images ç • Irregular urban structure • Illumination/weather/country • Different road types
  39. 39. Segmentation • Binary road masks • 19K*19K, 0.5m/pixel • SegNet
  40. 40. Extracting Road Segments • Orientation based median filtering • Road segments by orientation bucketing
  41. 41. NF NH NE Region Creation • Road graph: Node=intersection, edge=road, weight=length • Partition for max inter, min intra connectivity, using normalized min-cut. • 𝑛 𝑚𝑎𝑥 = 𝑐𝑒𝑖𝑙 𝑟𝑜𝑎𝑑𝑠 88
  42. 42. Region and Road Naming • Cmax 𝑟𝑜𝑎𝑑𝑠 𝐴 → 𝐶𝐴 (downtown) • Orientation bucketing into N, S, W, E • Trace regions based on distance to CA • Orientation bucketing into major axes • Trace roads based on order
  43. 43. Offsetting and Meter Marking • 5 meter marker along the road • Odd/even based on RHR • Distance field of roads: block offset
  44. 44. Unmapped Developing Country Regions follow natural boundaries
  45. 45. Street Address with Robocodes • From Satellite Imagery to Deployed Street Addresses • Generative address : linear, hierarchical, and intuitive • Human friendly rather than machine friendly
  46. 46. Occasional Frequent Low Quality Census High Quality Satellite, Street view Phone, Mobility, Social, Financial AV, PlanetLabs etc Sense Process Respond
  47. 47. Capture Analyze  Act Low Level Street-level Measure •Visual Census •Counting/Density Alert •Traffic Interventions Mid Level Aerial Understand •Geolocalization •Crowds •3D Modeling Assist •Crisis Response •Agriculture •Insurance/Hedge Funds High Level Satellite Aggregate and Predict • Economic activity Change •Street Addr •Government planning Pervasive Recording, Incentives, Distributed ownership, Privacy, Full proof authenticity, Equality
  48. 48. Act Alert Assist Change Low/Mi d/High Level Alert about the state of people/economy/b uilt environment (e.g., predict crop yield from satellite imagery, predict insurance price from street view) Assist in acting on information by providing suggestions based on data (e.g., design optimal congestion pricing based on detected cars, design crisis response in hurricanes) ?
  49. 49. Inaccessible Areas • To extend our format to cover areas that are not accessible by streets, we explored different implementations to cover such areas, which are 26*5 m away from any street. • Geocoding as a function (excluding the version field): f (info, lat, lon) = x.y.z.t • For places with roads, info={road network, city, country} f (R, C) = x.y.city.country • Extreme case: only reliable information is latitude/longitude! 52
  50. 50. f(C,lat,lon) = hash(round(lat,3)) + dir(lat) . hash(round(lon,3)) +dir(lon) . C L-A-T-dir.L-O-N-dir.name.area Inaccessible Areas: Blackholes! • Linear hashing: • 26 letters + 10 digits • 100m x 100 m granularity • Last letter is the hemisphere • Range: 359.999, longitude: 7PRZ W • Hierarchical hashing: • Enlarge the grid from to 1 km x 1 km • Using two floating points = three letters • Within each cell, re-hash it to a 36 x 36 grid = one letter • New resolution: 30m, represented by five letters 53 f(C,lat,lon) = hash(round(lat,2)) + hash(lat - round(lat,2)) + dir(lat) . hash(round(lon,2)) + hash(lon - round(lon,2)) + dir(lon) . C LlatLlatHlatDlat .LlonLlonHlonDlon . name . Ocean /Continent /etc
  51. 51. Thanks! What next? Today Tomorrow Friday A month • Robocode.info • Join our presentation in CVPR WiCV. Friday 10am • Join us with your new ideas at SIGGRAPH 2018 Maps & Urban Data session. Code: https://github.com/facebookresearch/street-addresses Paper: https://research.fb.com/publications/generative-street- addresses-from-satellite-imagery/
  52. 52. Bonus: Blackholes! Main aim: f(<place>)=robocode Base case: <place> = <house, street, city, country> “12C.NA14.PALO.CAUS” No street: <place> = <lat, lon, city, country> “F12.HN3.PALO.CAUS” No city/country: <place> = <lat, lon, other info (ocean, dessert, etc.)> “JK3.3DF.PAC.OCEA”
  53. 53. Region Experiments • Experimented with (a) normalized min-cut, (b) Newman- Girvan, (c) modularity based partitioning. • Experimented with image based methods (superpixels, region growing) and the dual of the road graph. • Evaluated with urban rules (geography, population, road distribution)
  54. 54. Output Maps and Tools • .osm maps with roads (meter marking and offsetting on the fly) • ID-tool of MapBox for on-demand inverse/forward geocoding • rtree extension for efficient spatial querying • Experimental mobile app for self navigation • 21.7% decrease in arrival time using Robocodes
  55. 55. Analyze: Three Types of Outputs 1. Semantic 2. Objective Population Density 76000/sq. mile 3. Qualitative Assign a semantic label to each pixel Label road quality as “Bad” or “Good”

×