Data pipeline and data lake for autonomous driving

•

0 j'aime•443 vues

Yu Huang

Data pipeline, data lake, autonomous driving

Ingénierie

DATA PIPELINE AND DATA LAKE IN
AUTONOMOUS DRIVING
YU HUANG
SUNNYVALE, CALIFORNIA
YU.HUANG07@GMAIL.COM

OUTLINE
• TESLA
• GOOGLE WAYMO
• PLUSAI
• ALIBABA CLOUD
• NVIDIA
• NETAPP
• AMAZON-AWS
• AMAZON-TRI
• AMAZON-MOMENTA
• ECKERSON DATAOPS
• IBM

© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Reference Architecture
Autonomous Driving Data Lake
Build an MDF4/Rosbag-based data ingestion and processing pipeline for Autonomous Driving and
Advanced Driver Assistance Systems (ADAS).
10
9
8
7
6
5
4
3
2
Ingest data from autonomous fleet
with AWS Outposts for local data
processing.
Ingest vehicle telemetry data in real
time using AWS IoT Core and
Amazon Kinesis Data Firehose.
Remove and transform low quality
data.
Schedule the extract, transform, load
(ETL) jobs using Apache Airflow.
Enrich data with weather conditions
based on GPS location and
timestamp.
Extract metadata using ASAM
OpenSCENARIO and store data in
Amazon DynamoDB and Amazon
Elasticsearch Service.
Store data lineage in Amazon
Neptune and catalog data using AWS
Glue Data Catalog.
Process drive data and perform deep
signal validation.
Perform automated labeling using
Amazon SageMaker Ground Truth.
Provide a search function for
particular scenarios using AWS
AppSync.
1
1
2
3
4
5
6
7
8 9
10

Radar and video data processing in MDF4 format achieves the highest scalability by leveraging AWS Fargate for Amazon ECS.

Data Pipeline Complexity
Eckerson DataOps

Analytic Models Failure to Launch
Eckerson DataOps

Streaming Data Pipelines
Eckerson DataOps

Requirements for Autonomous Navigation
Eckerson DataOps

DataOps for Model and Pipeline
Development and Operationalization
Continuous Integration / Continuous
Development（CI/CD）
Eckerson DataOps

Components and Complexities of Analytics Ecosystems
Eckerson DataOps

IBM Enterprise Data pipeline Marketecture

IBM Autonomous driving development use cases

Contenu connexe

Tendances

Azure data platform overviewJames Serra

Choosing technologies for a big data solution in the cloudJames Serra

Introducing Databricks DeltaDatabricks

Azure data analytics platform - A reference architecture Rajesh Kumar

Moving to Databricks & DeltaDatabricks

BDA311 Introduction to AWS GlueAmazon Web Services

AWS Big Data PlatformAmazon Web Services

Introduction to Amazon RedshiftAmazon Web Services

Using MLOps to Bring ML to Production/The Promise of MLOpsWeaveworks

Slides: Knowledge Graphs vs. Property GraphsDATAVERSITY

Unified Big Data Processing with Apache Spark (QCON 2014)Databricks

Data Lakehouse, Data Mesh, and Data Fabric (r1)James Serra

Introduction SQL Analytics on Lakehouse ArchitectureDatabricks

Firebolt vs SnowflakeCarlos Fuentes

Building Modern Data Platform with Microsoft AzureDmitry Anoshin

Introduction to snowflakeSunil Gurav

Data Virtualization: An IntroductionDenodo

CAP Theorem - Theory, Implications and PracticesYoav Francis

Snowflake essentialsqureshihamid

Databricks for DummiesRodney Joyce

Tendances (20)

Azure data platform overview

Choosing technologies for a big data solution in the cloud

Introducing Databricks Delta

Azure data analytics platform - A reference architecture

Moving to Databricks & Delta

BDA311 Introduction to AWS Glue

AWS Big Data Platform

Introduction to Amazon Redshift

Using MLOps to Bring ML to Production/The Promise of MLOps

Slides: Knowledge Graphs vs. Property Graphs

Unified Big Data Processing with Apache Spark (QCON 2014)

Data Lakehouse, Data Mesh, and Data Fabric (r1)

Introduction SQL Analytics on Lakehouse Architecture

Firebolt vs Snowflake

Building Modern Data Platform with Microsoft Azure

Introduction to snowflake

Data Virtualization: An Introduction

CAP Theorem - Theory, Implications and Practices

Snowflake essentials

Databricks for Dummies

Similaire à Data pipeline and data lake for autonomous driving

GPSWKS401_Designing a Cloud Enterprise Data WarehouseAmazon Web Services

Success has Many Query Engines- Tel Aviv Summit 2018Amazon Web Services

How Disney+ uses fast data ubiquity to improve the customer experience Martin Zapletal

Building Data Lakes and Analytics on AWSAmazon Web Services

AMF305_Autonomous Driving Algorithm Development on Amazon AIAmazon Web Services

Construindo data lakes e analytics com AWSAmazon Web Services LATAM

Accelerate Productivity by Computing at the Edge - AWS Online Tech TalksAmazon Web Services

Build Data Lakes & Analytics on AWS: Patterns & Best PracticesAmazon Web Services

Build Data Lakes and Analytics on AWS: Patterns & Best PracticesAmazon Web Services

Managed Relational Databases - Amazon RDSAmazon Web Services

Hybrid as a Stepping Stone: It’s Not All or Nothing for Your Cloud Transforma...Amazon Web Services

Building Data Lakes for Analytics on AWSAmazon Web Services

Build Data Lakes and Analytics on AWS: Patterns & Best Practices - BDA305 - A...Amazon Web Services

Building Data Lakes and Analytics on AWS; Patterns and Best Practices - BDA30...Amazon Web Services

Building Data Lakes and Analytics on AWSAmazon Web Services

What's new with Amazon Redshift - ADB203 - New York AWS SummitAmazon Web Services

AWS March 2016 Webinar Series - Building Big Data Solutions with Amazon EMR a...Amazon Web Services

Build Data Lakes & Analytics on AWS: Patterns & Best Practices - BDA305 - Ana...Amazon Web Services

Building a Modern Data Platform in the CloudAmazon Web Services

Similaire à Data pipeline and data lake for autonomous driving (20)

GPSWKS401_Designing a Cloud Enterprise Data Warehouse

Success has Many Query Engines- Tel Aviv Summit 2018

How Disney+ uses fast data ubiquity to improve the customer experience

Building Data Lakes and Analytics on AWS

AMF305_Autonomous Driving Algorithm Development on Amazon AI

Construindo data lakes e analytics com AWS

Accelerate Productivity by Computing at the Edge - AWS Online Tech Talks

Build Data Lakes & Analytics on AWS: Patterns & Best Practices

Build Data Lakes and Analytics on AWS: Patterns & Best Practices

Managed Relational Databases - Amazon RDS

Hybrid as a Stepping Stone: It’s Not All or Nothing for Your Cloud Transforma...

Building Data Lakes for Analytics on AWS

Build Data Lakes and Analytics on AWS: Patterns & Best Practices - BDA305 - A...

Building Data Lakes and Analytics on AWS; Patterns and Best Practices - BDA30...

Building Data Lakes and Analytics on AWS

What's new with Amazon Redshift - ADB203 - New York AWS Summit

AWS March 2016 Webinar Series - Building Big Data Solutions with Amazon EMR a...

Build Data Lakes & Analytics on AWS: Patterns & Best Practices - BDA305 - Ana...

Building a Modern Data Platform in the Cloud

Plus de Yu Huang

Application of Foundation Model for Autonomous DrivingYu Huang

The New Perception Framework in Autonomous Driving: An Introduction of BEV N...Yu Huang

Data Closed Loop in Simulation Test of Autonomous DrivingYu Huang

Techniques and Challenges in Autonomous DrivingYu Huang

BEV Joint Detection and SegmentationYu Huang

BEV Object Detection and PredictionYu Huang

Fisheye based Perception for Autonomous Driving VIYu Huang

Fisheye/Omnidirectional View in Autonomous Driving VYu Huang

Fisheye/Omnidirectional View in Autonomous Driving IVYu Huang

Prediction，Planninng & Control at BaiduYu Huang

Cruise AI under the HoodYu Huang

LiDAR in the Adverse Weather: Dust, Snow, Rain and Fog (2)Yu Huang

Scenario-Based Development & Testing for Autonomous DrivingYu Huang

How to Build a Data Closed-loop Platform for Autonomous Driving?Yu Huang

Annotation tools for ADAS & Autonomous DrivingYu Huang

Simulation for autonomous driving at uber atgYu Huang

Multi sensor calibration by deep learningYu Huang

Prediction and planning for self driving at waymoYu Huang

Jointly mapping, localization, perception, prediction and planningYu Huang

Open Source codes of trajectory prediction & behavior planningYu Huang

Plus de Yu Huang (20)

Application of Foundation Model for Autonomous Driving

The New Perception Framework in Autonomous Driving: An Introduction of BEV N...

Data Closed Loop in Simulation Test of Autonomous Driving

Techniques and Challenges in Autonomous Driving

BEV Joint Detection and Segmentation

BEV Object Detection and Prediction

Fisheye based Perception for Autonomous Driving VI

Fisheye/Omnidirectional View in Autonomous Driving V

Fisheye/Omnidirectional View in Autonomous Driving IV

Prediction，Planninng & Control at Baidu

Cruise AI under the Hood

LiDAR in the Adverse Weather: Dust, Snow, Rain and Fog (2)

Scenario-Based Development & Testing for Autonomous Driving

How to Build a Data Closed-loop Platform for Autonomous Driving?

Annotation tools for ADAS & Autonomous Driving

Simulation for autonomous driving at uber atg

Multi sensor calibration by deep learning

Prediction and planning for self driving at waymo

Jointly mapping, localization, perception, prediction and planning

Open Source codes of trajectory prediction & behavior planning

Dernier

Introduction to Multiple Access Protocol.pptxupamatechverse

Sheet Pile Wall Design and Construction: A Practical Guide for Civil Engineer...Dr.Costas Sachpazis

BSides Seattle 2024 - Stopping Ethan Hunt From Taking Your Data.pptxfenichawla

Extrusion Processes and Their Limitations120cr0395

High Profile Call Girls Nagpur Meera Call 7001035870 Meet With Nagpur EscortsCall Girls in Nagpur High Profile

Russian Call Girls in Nagpur Grishma Call 7001035870 Meet With Nagpur EscortsCall Girls in Nagpur High Profile

(PRIYA) Rajgurunagar Call Girls Just Call 7001035870 [ Cash on Delivery ] Pun...ranjana rawat

UNIT-III FMM. DIMENSIONAL ANALYSISrknatarajan

The Most Attractive Pune Call Girls Budhwar Peth 8250192130 Will You Miss Thi...ranjana rawat

AKTU Computer Networks notes --- Unit 3.pdfankushspencer015

Call for Papers - African Journal of Biological Sciences, E-ISSN: 2663-2187, ...Christo Ananth

MANUFACTURING PROCESS-II UNIT-1 THEORY OF METAL CUTTINGSIVASHANKAR N

Coefficient of Thermal Expansion and their Importance.pptxAsutosh Ranjan

College Call Girls Nashik Nehal 7001305949 Independent Escort Service NashikCall Girls in Nagpur High Profile

(SHREYA) Chakan Call Girls Just Call 7001035870 [ Cash on Delivery ] Pune Esc...ranjana rawat

MANUFACTURING PROCESS-II UNIT-5 NC MACHINE TOOLSSIVASHANKAR N

Booking open Available Pune Call Girls Koregaon Park 6297143586 Call Hot Ind...Call Girls in Nagpur High Profile

Call Girls Pimpri Chinchwad Call Me 7737669865 Budget Friendly No Advance Boo...roncy bisnoi

(ANVI) Koregaon Park Call Girls Just Call 7001035870 [ Cash on Delivery ] Pun...ranjana rawat

Processing & Properties of Floor and Wall Tiles.pptxpranjaldaimarysona

Dernier (20)

Introduction to Multiple Access Protocol.pptx

Sheet Pile Wall Design and Construction: A Practical Guide for Civil Engineer...

BSides Seattle 2024 - Stopping Ethan Hunt From Taking Your Data.pptx

Extrusion Processes and Their Limitations

High Profile Call Girls Nagpur Meera Call 7001035870 Meet With Nagpur Escorts

Russian Call Girls in Nagpur Grishma Call 7001035870 Meet With Nagpur Escorts

(PRIYA) Rajgurunagar Call Girls Just Call 7001035870 [ Cash on Delivery ] Pun...

UNIT-III FMM. DIMENSIONAL ANALYSIS

The Most Attractive Pune Call Girls Budhwar Peth 8250192130 Will You Miss Thi...

AKTU Computer Networks notes --- Unit 3.pdf

Call for Papers - African Journal of Biological Sciences, E-ISSN: 2663-2187, ...

MANUFACTURING PROCESS-II UNIT-1 THEORY OF METAL CUTTING

Coefficient of Thermal Expansion and their Importance.pptx

College Call Girls Nashik Nehal 7001305949 Independent Escort Service Nashik

(SHREYA) Chakan Call Girls Just Call 7001035870 [ Cash on Delivery ] Pune Esc...

MANUFACTURING PROCESS-II UNIT-5 NC MACHINE TOOLS

Booking open Available Pune Call Girls Koregaon Park 6297143586 Call Hot Ind...

Call Girls Pimpri Chinchwad Call Me 7737669865 Budget Friendly No Advance Boo...

(ANVI) Koregaon Park Call Girls Just Call 7001035870 [ Cash on Delivery ] Pun...

Processing & Properties of Floor and Wall Tiles.pptx

Data pipeline and data lake for autonomous driving

1. DATA PIPELINE AND DATA LAKE IN AUTONOMOUS DRIVING YU HUANG SUNNYVALE, CALIFORNIA YU.HUANG07@GMAIL.COM

2. OUTLINE • TESLA • GOOGLE WAYMO • PLUSAI • ALIBABA CLOUD • NVIDIA • NETAPP • AMAZON-AWS • AMAZON-TRI • AMAZON-MOMENTA • ECKERSON DATAOPS • IBM

3. Tesla

7. Nvidia

8. Nvidia

9. Nvidia

10. Nvidia

11. Nvidia

12. Nvidia

13. NetApp

14. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Reference Architecture Autonomous Driving Data Lake Build an MDF4/Rosbag-based data ingestion and processing pipeline for Autonomous Driving and Advanced Driver Assistance Systems (ADAS). 10 9 8 7 6 5 4 3 2 Ingest data from autonomous fleet with AWS Outposts for local data processing. Ingest vehicle telemetry data in real time using AWS IoT Core and Amazon Kinesis Data Firehose. Remove and transform low quality data. Schedule the extract, transform, load (ETL) jobs using Apache Airflow. Enrich data with weather conditions based on GPS location and timestamp. Extract metadata using ASAM OpenSCENARIO and store data in Amazon DynamoDB and Amazon Elasticsearch Service. Store data lineage in Amazon Neptune and catalog data using AWS Glue Data Catalog. Process drive data and perform deep signal validation. Perform automated labeling using Amazon SageMaker Ground Truth. Provide a search function for particular scenarios using AWS AppSync. 1 1 2 3 4 5 6 7 8 9 10

15. Radar and video data processing in MDF4 format achieves the highest scalability by leveraging AWS Fargate for Amazon ECS.

16.

17.

18. Autonomous driving at AWS

19. TRI

20. Momenta

21. Next Generation Data Pipelines

22. Data Pipeline Complexity Eckerson DataOps

23. Analytic Models Failure to Launch Eckerson DataOps

24. Streaming Data Pipelines Eckerson DataOps

25. Requirements for Autonomous Navigation Eckerson DataOps

26. DataOps for Model and Pipeline Development and Operationalization Continuous Integration / Continuous Development（CI/CD） Eckerson DataOps