Location analytics by Marc Planaguma at Big Data Spain 2014

•

0 j'aime•865 vues

http://www.bigdataspain.org/2014/conference/location-analytics While the implementation of analytic operations on distributed computing frameworks has been widely describing, enabling the computational core of a Big Data system with capabilities for supporting geospatial querying on data is yet a challenging issue. This session aims to target that specific aspect by reviewing how researchers at BDigital Technology Centre have designed and implemented a stack for advanced Machine Learning on Urban Data by providing a way to geoquery massive amounts of HDFS data from Spark processes without hindering the overall system performance.

Technologie

SPATIAL DATA ANALYTICS
MARC PLANAGUMÀ
LEAD DATA ENGINEER AND RESEARCHER BDIGITAL

Ciudad 2020
Exploratory lines and further work
Joana Simoes, BDigital
27/05/2014
Scaling up a in a world of geolocated data
Marc Planagumà, BDigital Technology Centre

Spatial data analytics
Spatial analysis or spatial statistics includes any of
the formal techniques which studies entities using
their topological, geometric or geographic
properties

Spatial data analytics
Basic Spatial Types
Line Point Polygon
* plus others

Spatial data analytics
Operations
Contains Within

Spatial data analytics
Operations
Equals Disjoint

Spatial data analytics
Operations
Touches Overlaps
Crosses

Spatial data analytics
Operations
Distance

Spatial data analytics
Operations
Union Difference

Spatial data analytics
Operations
Intersection Symmetric Diff

Spatial data analytics
Complex operations
Buffering
Convex Hull

Spatial data analytics
Complex operations
Area

Spatial data analytics
Geospatial data mining techniques
Density-based clustering Hierarchy-based clustering

Spatial data analytics
Geospatial data mining techniques
Heatmaps

Spatial data analytics
Geospatial data mining techniques
Trajectory mining

Spatial data analytics
Geospatial data mining techniques
Geospatial querying
Spatial anomalies
Geographically weighted regressions
Self-organizing Maps
Agent-based modeling

Spatial data analytics
Our Datasets
Twitter (Point)
Meteo (Polygon)
Traffic status (Multi-Line-String)
Traffic incidence (Multi-Point)
Credit card ops (Multi-Point)
Demographic (Point & Polygon)
Shapefiles (Multi-Polygon)

Our Tools
Spatial data analytics
Queries
Clustering
Heat maps
Trajectory
Anomaly detection

Spatial data analytics
Old School Stack - Why Not?
PostGIS Hadoop + GIS
Kibana
Tools ElasticSearch

Spatial data analytics
Hadoop Stack - The new Classic
PostGIS Hadoop + GIS
Kibana
Tools ElasticSearch

Spatial data analytics
Complete Stack - Big Data GIS
PostGIS Hadoop + GIS
Kibana
Tools ElasticSearch

Do’s and Don’ts
Spatial data analytics
Geo Batch
Near Real-Time
Basic Geo Operations
Workflow Composition
Geo Streams
Real-Time
Advanced Geo Op.
Unattended Workflow

Moltes gràcies!
nom@bdigital.org
mplanaguma@bdigital.org
@djkram @bdigital

Contenu connexe

Tendances

The Role of Indoor Mapping in the “New Normal”

CARTO

Tips and tricks for Working with Demographic Data [CARTOframes & Python]

CARTO

Unlock the power of spatial analysis using CARTO and python [CARTOframes]

CARTO

How to Use Spatial Data Science in your Site Planning Process? [CARTOframes]

CARTO

Le rôle de l’intelligence géospatiale dans la reprise économique

CARTO

Encuentro Aporta 2016 - Mesa 2 - Miguel Arias

Datos.gob.es

CARTO en 5 Pasos: del Dato a la Toma de Decisiones [CARTO]

CARTO

Tb geo dev_presentation_nov_5

Todd Barr

Scaling Spatial Analytics with Google Cloud & CARTO

CARTO

Think Spatial: Don't Ignore Location in your Models! [CARTOframes]

CARTO

This webinar focuses on how using CARTO's tech stack to embed Location Intelligence in a variety of web and mobile apps. Our speakers, Román Jiménez and Víctor Velarde, showcase the latest products and features available for developers to reduce production time of custom-built apps and deliver superior user experiences. Watch the recorded at: https://go.carto.com/airship-location-intelligence-web-apps-user-experience-recorded

Embedding Location Intelligence in Web Apps that Enhance User Experience [Air...

CARTO

Market analysis through Consumer Behavior Pattern Insights

CARTO

Stepping up your Spatial Analysis with Mastercard Retail Location insights 3.0

CARTO

Understanding Retail Catchment Areas with Human Mobility Data

CARTO

Applying Spatial Analysis to Real Estate Decision-Making

CARTO

Can Kanye West Save Gap? Real-Time Consumer Social Media Segmentation On CARTO

CARTO

Indoor Mapping: Why Precision Matters

CARTO

Understanding short term rental & tourism trends through geospatial data [CAR...

CARTO

CARTO BUILDER: from visualization to geospatial analysis

Jorge Sanz

Introduction to GIS

Ahmed Ali Rajpoot

Tendances (20)

The Role of Indoor Mapping in the “New Normal”

Tips and tricks for Working with Demographic Data [CARTOframes & Python]

Unlock the power of spatial analysis using CARTO and python [CARTOframes]

How to Use Spatial Data Science in your Site Planning Process? [CARTOframes]

Le rôle de l’intelligence géospatiale dans la reprise économique

Encuentro Aporta 2016 - Mesa 2 - Miguel Arias

CARTO en 5 Pasos: del Dato a la Toma de Decisiones [CARTO]

Tb geo dev_presentation_nov_5

Scaling Spatial Analytics with Google Cloud & CARTO

Think Spatial: Don't Ignore Location in your Models! [CARTOframes]

Embedding Location Intelligence in Web Apps that Enhance User Experience [Air...

Market analysis through Consumer Behavior Pattern Insights

Stepping up your Spatial Analysis with Mastercard Retail Location insights 3.0

Understanding Retail Catchment Areas with Human Mobility Data

Applying Spatial Analysis to Real Estate Decision-Making

Can Kanye West Save Gap? Real-Time Consumer Social Media Segmentation On CARTO

Indoor Mapping: Why Precision Matters

Understanding short term rental & tourism trends through geospatial data [CAR...

CARTO BUILDER: from visualization to geospatial analysis

Introduction to GIS

En vedette

Data warehouse modernization programme by TOBY WOOLFE at Big Data Spain 2014

Big Data Spain

Intro to the Big Data Spain 2014 conference

Big Data Spain

Big Data Web applications for Interactive Hadoop by ENRICO BERTI at Big Data...

Big Data Spain

http://www.bigdataspain.org/2014/conference/dataflows-the-abstraction-that-powers-the-big-data-technology Dataflows are an omnipresent abstraction across many big data technologies due to its suitability for representing programs in a way that is easy to parallelize. All dataflow models---such as those of Spark or MapReduce---are stateless, which facilitates achieving fault tolerance, a crucial property when running at large-scale. However, this stateless dataflow models have a negative impact on the programming models they expose, which need to adapt to match the stateless nature of the underlying platforms. With the “democratization of data”, different types of users with different skills want answers from their big datasets, but sometimes they lack the skills required to write programs adapted to these specific frameworks: A familiar programming model becomes crucial to open big data value to a broader set of users. Video https://www.youtube.com/watch?v=blY7EleXW6U

Dataflows: The abstraction that powers Big Data by Raul Castro Fernandez at ...

Big Data Spain

The top five questions to ask about NoSQL. JONATHAN ELLIS at Big Data Spain 2012

Big Data Spain

Big Data the potential for data to improve service and business management by...

Big Data Spain

http://www.bigdataspain.org/2014/conference/state-of-play-data-science-on-hadoop-in-2015-keynote Machine Learning is not new. Big Machine Learning is qualitatively different: More data beats algorithm improvement, scale trumps noise and sample size effects, can brute-force manual tasks. Session presented at Big Data Spain 2014 Conference 18th Nov 2014 Kinépolis Madrid http://www.bigdataspain.org Event promoted by: http://www.paradigmatecnologico.com Slides: https://speakerdeck.com/bigdataspain/state-of-play-data-science-on-hadoop-in-2015-by-sean-owen-at-big-data-spain-2014

State of Play. Data Science on Hadoop in 2015 by SEAN OWEN at Big Data Spain ...

Big Data Spain

We live in an age of an over-abundance of data stores choices and we also live in an age where the data has more value than ever.We think that data has no value until it’s shown and everybody is nowadays looking for simple ways to get insights from its data through self-explanatory visualizations. Would it be nice if we were able to create awesome graphs regardless the underlying technology that stores our data? Session presented at Big Data Spain 2015 Conference 15th Oct 2015 Kinépolis Madrid http://www.bigdataspain.org Event promoted by: http://www.paradigmatecnologico.com Abstract: http://www.bigdataspain.org/program/thu/slot-12.html

Getting the best insights from your data using Apache Metamodel by Alberto Ro...

Big Data Spain

NoSQL databases have emerged as a response to some perceived problems in the RDBMSs: agile/dynamic schemas; and transparent, horizontal scaling of the database. The former has been promptly targeted with the introduction of unstructured data types, but scaling a relational databases is still a very hard problem. As a consequence, all NoSQL databases have been built from scratch: their storage engines, replication techniques, journaling, ACID support (if any). They haven't leveraged the previously existing state-of-the-art of RDBMSs, effectively re-inventing the wheel. Isn't this sub-optimal? Wouldn't it be possible to construct a NoSQL database by layering it on top of a relational database? Session presented at Big Data Spain 2015 Conference 16th Oct 2015 Kinépolis Madrid http://www.bigdataspain.org Event promoted by: http://www.paradigmatecnologico.com Abstract: http://www.bigdataspain.org/program/fri/slot-37.html

ToroDB: Scaling PostgreSQL like MongoDB by Álvaro Hernández at Big Data Spain...

Big Data Spain

Talk by Gordon Guthrie, Senior Software Engineer at Basho Summary A review of the CAP Theorem and the difficulties of resolving conflicts in highly distributed systems. Covering the issues and various theories on how to resolve including the use CRDTs in Riak Details CRDTs are used to replicate data across multiple computers in a network, executing updates without the need for remote synchronisation. This leads to merge conflicts in systems using conventional eventual consistency technology, but CRDTs are designed such that conflicts are mathematically impossible. Under the constraints of the CAP theorem they provide the strongest consistency guarantees for available/partition-tolerant (AP) settings. The CRDT concept was first formally defined in 2007 by Marc Shapiro and Nuno Preguiça in terms of operation commutativity, and development was initially motivated by collaborative text editing. The concept of semilattice evolution of replicated states was first defined by Baquero and Moura in 1997, and development was initially motivated by mobile computing. The two concepts were later unified in 2011. Basho has worked with the EU and Marc Shapiro's team to push CRDTs into distributed systems. Riak v2.x is the first commercial product to include this functionality

Convergent Replicated Data Types in Riak 2.0

Big Data Spain

CloudMC: A cloud computing map-reduce implementation for radiotherapy. RUBEN ...

Big Data Spain

The ultimate business success of Big Data in business will depend on our ability to successfully bring about the realignment and placement of Big Data into a more generalized architectural framework, one that coalesces strategic, technical and management elements of data warehousing (DW 3.0), business intelligence, textual analysis and statistical analysis into a coherent, synergistic and usable whole. Session presented at Big Data Spain 2015 Conference 16th Oct 2015 Kinépolis Madrid http://www.bigdataspain.org Event promoted by: http://www.paradigmatecnologico.com Abstract: http://www.bigdataspain.org/program/fri/slot-26.html

Big Data, analytics and 4th generation data warehousing by Martyn Jones at Bi...

Big Data Spain

At LinkedIn, we ingest more than 1 Trillion events per day pertaining to user behavior, application and system health etc. into our pub-sub system (Kafka). Another source of events are the updates that are happening on our SQL and No-SQL databases. For e.g. every time a user changes their linkedIn profile, a ton of downstream applications need to know what happened and need to react to it. We have a system (DataBus) which listens to changes in the database transaction logs and makes them available for down stream processing. We process ~2.1 Trillion of such database change events per week. We use Apache Samza for processing these event-streams in real time. In this presentation we will discuss some of challenges we faced and the various techniques we used to overcome them. Session presented at Big Data Spain 2015 Conference 15th Oct 2015 Kinépolis Madrid http://www.bigdataspain.org Event promoted by: http://www.bigdataspain.org/program/thu/slot-3.html

Essential ingredients for real time stream processing @Scale by Kartik pParam...

Big Data Spain

Analyzing organization e-mails in near real time using hadoop ecosystem tools...

Big Data Spain

DKS EAP (Enterprise Analytical Portal) was first thought of as an integrated analytical portal; a conjunction of BI and BA products (ranging from advanced customer intelligence to marketing analytics) assembled under a unified interface, specifically designed to better assist business in their decision-making. We believe DKS EAP is of special interest for the Big Data community nowadays since it is currently leveraging technologies such as Hadoop, Cassandra, Spark and Storm to improve its analytic capabilities. Accordingly, the talk will briefly present DKS EAP as an integrated Big Data analytic environment, focusing on how the fast-growing needs for real-time, social media and geomarketing data analysis intensified its necessity to embrace Big Data technologies. Session presented at Big Data Spain 2015 Conference 15th Oct 2015 Kinépolis Madrid http://www.bigdataspain.org Event promoted by: http://www.paradigmatecnologico.com Abstract: http://www.bigdataspain.org/program/thu/slot-8.html#spch9.1

How to integrate Big Data onto an analytical portal, Big Data benchmarking fo...

Big Data Spain

Stratio presented its open source Lucene-based implementation of Cassandra’s secondary indexes at Cassandra Summit London 2014, which provided several search engine features. It used to be distributed as a fork of Apache Cassandra, which was a huge problem both for users and maintainers. Nowadays, due to some changes introduced at C* 2.1.6, we are proud to announce that it has become a plugin that can be attached to the official Apache Cassandra. With the plugin we have been able to provide C* with geospatial capabilities, making it possible to index geographical positions and perform bounding box and radial distance queries. This is achieved through Lucene’s geospatial module. Session presented at Big Data Spain 2015 Conference 15th Oct 2015 Kinépolis Madrid http://www.bigdataspain.org Event promoted by: http://www.paradigmatecnologico.com Abstract: http://www.bigdataspain.org/program/thu/slot-9.html

Geospatial and bitemporal search in C* with pluggable Lucene index by Andrés ...

Big Data Spain

Purpose of the talk: Describing the use of Machine Learning and Big Data Techniques to improve the performance of elearning students. Presenting an existing case of an elearning platform (iAdLearn¡ng) and the technology used behind the scenes, to make adaptive/high performance elearning a reality. Session presented at Big Data Spain 2015 Conference 15th Oct 2015 Kinépolis Madrid http://www.bigdataspain.org Event promoted by: http://www.paradigmatecnologico.com Abstract: http://www.bigdataspain.org/program/thu/slot-17.html

IAd-learning: A new e-learning platform by José Antonio Omedes at Big Data Sp...

Big Data Spain

The term 'Data Science' was first described in scientific literature about 15 years ago. It started to become a major trend in industry about 7 years ago. O'Reilly Media surveys the industry extensively each year. In addition we get a good birds-eye view of industry trends through our conference programs and publications, working closely with some of the best practitioners in Data Science. By now, the field has evolved far beyond its origins eclipsing an earlier generation of Business Intelligence and Data Warehousing approaches. Data Science is moving up, into the business verticals and government spheres of influence where it has true global impact. This talk considers Data Science trends from the past three years in particular. What is emerging? Which parts are evolving? Which seem cluttered and poised for consolidation or other change? Session presented at Big Data Spain 2015 Conference 15th Oct 2015 Kinépolis Madrid http://www.bigdataspain.org Event promoted by: http://www.paradigmatecnologico.com Abstract: http://www.bigdataspain.org/program/thu/slot-2.html

Data Science in 2016: Moving up by Paco Nathan at Big Data Spain 2015

Big Data Spain

Apache Spark has successfully built on Hadoop infrastructure to encompass real-time processing, moving from rigid Map-Reduce operations to general purpose functional operations distributed across a cluster of machines. However data storage has become a black box. The source data for a query has to be retrieved in full and sent through the analysis pipeline rather than processing the data where it is stored, as in traditional database systems. This introduces significant cost, both in network utilisation and in the time taken to produce a result. Session presented at Big Data Spain 2015 Conference 15th Oct 2015 Kinépolis Madrid http://www.bigdataspain.org Event promoted by: http://www.paradigmatecnologico.com Abstract: http://www.bigdataspain.org/program/thu/slot-14.html

A new streaming computation engine for real-time analytics by Michael Barton ...

Big Data Spain

Processing large-scale graphs with Google(TM) Pregel by MICHAEL HACKSTEIN at...

Big Data Spain

En vedette (20)