AWS Summit Seoul 2023 | Amazon Redshift Serverless를 활용한 LG 이노텍의 데이터 분석 플랫폼 혁신 과정
Signaler
Partager
•1 j'aime•154 vues
AWS Summit Seoul 2023 | Amazon Redshift Serverless를 활용한 LG 이노텍의 데이터 분석 플랫폼 혁신 과정
•1 j'aime•154 vues
Signaler
Partager
Télécharger pour lire hors ligne
LG 이노텍은 세계 시장을 선도하는 글로벌 소재·부품기업으로, Amazon Redshift 을 데이터 분석 플랫폼의 핵심 서비스로 활용하고 있습니다. 이 세션에서는 비즈니스 사용자의 데이터 활용성 증대와 리포팅 성능 개선을 위한 Redshift Serverless 도입 과정과 이를 통한 비용 최적화 여정과 함께, 데이터 메쉬(Data Mesh) 아키텍처 기반 데이터 분석 플랫폼의 밑그림을 엿볼 수 있는 기회를 제공합니다.
Recommandé
Contenu connexe
Tendances
Tendances(20)
Similaire à AWS Summit Seoul 2023 | Amazon Redshift Serverless를 활용한 LG 이노텍의 데이터 분석 플랫폼 혁신 과정
Similaire à AWS Summit Seoul 2023 | Amazon Redshift Serverless를 활용한 LG 이노텍의 데이터 분석 플랫폼 혁신 과정(20)
![[Keynote] 슬기로운 AWS 데이터베이스 선택하기 - 발표자: 강민석, Korea Database SA Manager, WWSO, A...](https://cdn.slidesharecdn.com/ss_thumbnails/d3s01aws-230704014400-3eeae447-thumbnail.jpg?width=768&fit=bounds)
![[보험사를 위한 AWS Data Analytics Day] 5_KB금융그룹과 계열사의 AWS 기ᄇ...](https://cdn.slidesharecdn.com/ss_thumbnails/awsdataanalyticsday5kbawsanalyticsjourneybyeong-eokkang-230608094829-c0103aff-thumbnail.jpg?width=768&fit=bounds)
Plus de Amazon Web Services Korea
Plus de Amazon Web Services Korea(17)
AWS Summit Seoul 2023 | Amazon Redshift Serverless를 활용한 LG 이노텍의 데이터 분석 플랫폼 혁신 과정
- 1. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. S E O U L | M A Y 3 , 2 0 2 3
- 2. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Redshift Serverless LG LG LG CNS AWS
- 3. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. • • Amazon Redshift Serverless • LG • Agenda
- 4. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Kinesis Data Analytics Amazon EMR AWS Glue / Amazon S3 Amazon Redshift Amazon OpenSearch Amazon Kinesis Data Firehose Amazon MSK Serverless Kinesis Data Steams AWS Data Exchange AWS DMS AWS Glue 3 Amazon QuickSight Amazon Athena Amazon Redshift
- 5. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. • • • •
- 6. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Redshift Serverless
- 7. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates.All rights reserved. 구분 서버리스 프로비저닝 리소스할당 RPU 기반 클러스터 기반 운영 및 관리 클러스터 관리 필요 없음(Min/Max) 클러스터 관리 필요 워크로드 관리(WLM) 자동 규모 확장 수동/자동 선택 Amazon Redshift Spectrum 스캔 데이터양에 관계 없음 RPU 시간 당 $0.438 (서울리전) 스캔 데이터양에 따라 다름 1 TB 스캔 당 $5 가격 정책 간단 RPU 시간 + 스토리지(RMS) 복잡 컴퓨팅 노드 + 스토리지(RMS) + 동시성 확장 + Spectrum * RPU : Redshift Processing Unit * RMS : Redshift Managed Storage
- 8. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Redshift Serverless 아키텍처 디자인 패턴
- 9. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon S3 AWS Glue Amazon QuickSight Amazon RedshiftProvisioned ( ) Amazon RedshiftServerless 데이터사이언스팀(소비자) Amazon SageMaker 신규 아키텍처 데이터 공유 #1:
- 10. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Glue Amazon Redshift Provisioned Amazon Redshift Provisioned 데이터 공유 데이터웨어하우스(생산자) 8 node * ra3.4xlarge 리포팅(소비자) 리포팅 팀 데이터 공유 신규 아키텍처 기존 아키텍처 Amazon Redshift Provisioned 리포팅 팀 야간 배치 야간 배치 AWS Glue 리포팅(소비자) #2: Amazon Redshift Serverless 데이터웨어하우스(생산자) 64 RPU
- 11. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Kinesis Data Streams Amazon Kinesis Data Firehose Amazon Redshift Provisioned Amazon Kinesis Data Streams Amazon Kinesis Data Firehose JDBC / ODBC JDBC / ODBC Amazon QuickSight Amazon QuickSight 신규 아키텍처 기존 아키텍처 데이터웨어하우스( + ) 12 node * ra3.4xlarge #3: Amazon Redshift Provisioned 데이터웨어하우스( ) 12 node * ra3.4xlarge Amazon Redshift Serverless 리포팅(소비자) 128 RPU 데이터 공유
- 12. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 모든 종류의 데이터 Files 스트림 데이터 데이터베이스 외부 데이터 Auto-Copy 실시간 수집 Zero-ETL Amazon Data Exchange 통합 데이터베이스 Federated Query AWS Glue Amazon EMR Amazon SageMaker 모든 분석 사용자 JDBC/ODBC Data API Apache Spark 통합 Redshift ML BI 도구 Amazon S3 Serverless endpoint Provisioned cluster Serverless endpoint Provisioned cluster 데이터 거버넌스 , , ,
- 13. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. LG
- 14. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 모바일 기기에 적용되는 초슬림 고화소 카메라 모듈을 비롯해 3D 센싱 모듈 분야의 글로벌 시장을 선도하고, 차량용 카메라, XR(확장현실) 등 사업 분야도 시장을 주도하고 있습니다. 첨단 스마트폰 IT/Server 용 반도체에 적용되는 Package- Substrate, 고해상도 디스플레이에 사용되는 Tape- Substrate를 생산 중이며, FC-BGA 신규 사업 추진을 통해 미래 글로벌 기판 시장의 혁신을 주도하고 있습니다. 디스플레이용 초슬림 PSU와 편리하고 안전한 IoT 환경 구축을 위한 고감도 통신부품 및 센서 분야에서 글로벌 시장 리더쉽을 확보하고 있습니다. 통신 모듈, 자율주행용 센서, 파워 모듈 등 미래 자동차 트렌드에 맞는 토탈 솔루션을 제공합니다. 주행 안정성, 운전자 편의 제공 및 성능 향상을 위한 기술로 글로벌 시장을 확대하고 있습니다 . Camera Module 3D Sensing Module Actuator Package Substrate FC-BGA Tape Substrate Display Mask Power Solution Smart Connectivity Solution Connectivity Solution Autonomous Driving Solution Electrification Solution Lighting Solution 광학 솔루션 기판 소재 전장 부품 전자 부품 LG 이노텍 사업 영역
- 15. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. POC / Data Lake / Warehouse Application SAS ( / , ) MLOps( ) MSTR ( , ) • 2023 ( ) • Feedback Service • 2022 • , , • / MLOps • 2021 • • • (ERP, SCM) • 2020 • (MES) / (MSTR, SAS) • 2019 • AWS Sensor / / Log ( ) / ( ) , ( ) MES ERP SCM Feedback Service AI / DX LG
- 16. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. LG Data Platform / On-Premise ML/DL / AWS Glue Amazon Athena (View ) AWS IOT Greengrass ( ) Amazon S3 ( ) Amazon MSK ( ) Amazon EC2 ( ) Amazon ElastiCache ( ) AWS Glue ( / / ) AWS Glue Sensor Amazon S3 ( ) ERP/MES Amazon Redshift ( ) Amazon Redshift ( / ) Amazon Redshift ( / ) Amazon S3 ( ) Amazon S3 (Map Type ) Amazon S3 ( ) Amazon S3 ( /IOT ) Informatica Informatica Informatica
- 17. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Batch Shift
- 18. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 0 10 20 30 40 50 60 32TB (DS2.8xlarge 2 Node) / / size(TB) #1.
- 19. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Infra Amazon Redshift Amazon Redshift Source Target Call Procedure Max 9,000 Sec ± α / #1.
- 20. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Batch Amazon Redshift Hourly Batch/ 1h 2h 3h 4h Shift Hourly ( ) #2. Hourly
- 21. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. OLAP Serverless ETL-Batch #3. Provisioned vs. Serverless
- 22. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Soft Landing Data Share
- 23. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. POC & Task PoC
- 24. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Cloud Amazon Redshift (RA3.4xlarge 5 Node) Provisioned Cluster Consumer Amazon Redshift Serverless 64 RPU Workgroup – MSTR Workgroup – SaS Workgroup – MLOps Producer Amazon Redshift Serverless 64 RPU Amazon Redshift Serverless 64 RPU / ML/DL / Amazon Redshift Serverless ( )
- 25. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 26. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Redshift DS2 • • (ElasticResize) RA3 Amazon Redshift RA3 • • RedshiftManagedStorage(RMS) • HDD • / #1.
- 27. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. • • (WLM) • RA3 • • • WLM Amazon Redshift Amazon Redshift Producer Consumer Amazon Redshift RA3.4xlarge 5 Node RA3.4xlarge 10 Node RA3.16xlarge 2 Node #2. Scale-up vs. Data Share
- 28. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. • • • • • • Amazon Redshift Provisioned Amazon Redshift Serverless Producer Consumer #3. Amazon Redshift Provisioned Amazon Redshift Provisioned Producer Consumer
- 29. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. • • Faster Classic Resize ? • 1 • , • • #1.
- 30. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. • / AUTO • ETL / OLAP • • • • / • Support Case , object id • #2.
- 31. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. • • Redshift Processing Unit(RPU) IP • Cold Start • , • , • #3.
- 32. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Cloud Amazon Redshift Consumer Amazon S3 AWS Lake Formation AWS Glue Amazon AppStream 2.0 Amazon Redshift Producer Amazon EC2 ETL / ML/DL / 2023
- 33. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS DATA MESH • / • • • Data As a Product • • LG
- 34. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Cloud Amazon Redshift Consumer Amazon S3 AWS Lake Formation Data Share Governance Amazon Aurora AWS Glue Auto-copy Zero-ETL Producer Consumer Data Steward AWS Glue Amazon Kinesis Data Streams Amazon Kinesis Data Firehose Amazon AppStream 2.0 Amazon Redshift Producer Amazon DataZone Amazon S3 / ML/DL /
- 35. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. o , . o . o .
- 36. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. 감사합니다 © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.