Contenu connexe Similaire à 아름답고 유연한 데이터 파이프라인 구축을 위한 Amazon Managed Workflow for Apache Airflow - 유다니엘 AWS 솔루션즈 아키텍트 :: AWS Summit Seoul 2021 (20) Plus de Amazon Web Services Korea (20) 아름답고 유연한 데이터 파이프라인 구축을 위한 Amazon Managed Workflow for Apache Airflow - 유다니엘 AWS 솔루션즈 아키텍트 :: AWS Summit Seoul 20211. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
K O R E A | M A Y 1 1 - 1 2 , 2 0 2 1
2. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
아름답고 유연한 데이터 파이프라인 구축을
위한 Amazon Managed Workflows for
Apache Airflow
유다니엘
솔루션즈 아키텍트
AWS
3. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data Pipeline Problem
Apache Airflow
Amazon Managed Workflows for Apache Airflow (MWAA)
Demo
Airflow Integrations
MWAA vs Step Function
Agenda
4. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
복잡한 데이터 파이프라인
5. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
오케스트레이션
6. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Apache Airflow
7. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Apache Airflow
8. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
사용 사례
ETL AI/ML DevOps
9. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
주요 구성 요소
스케줄러 워커 메타 데이터베이스
웹서버
10. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
셀프 구축 및 운영의 어려움
셋업 보안 유지관리
업그레이드
스케일링
11. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Managed Workflows
for Apache Airflow (MWAA)
12. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA가 제공하는 혜택 (1)
셋업
13. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA가 제공하는 혜택 (2)
스케일링
14. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA가 제공하는 혜택 (3)
보안
15. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA가 제공하는 혜택 (4)
업그레이드
16. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA가 제공하는 혜택 (5)
유지관리
17. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 아키텍처
Airflow Scheduler
Airflow Worker(s) Airflow Web Server
Meta Database
Private
Network
Public
Network
Service VPC
Customer VPC
18. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 사용 방법
MWAA 환경 생성 DAGs 및 Plugin을
S3에 복사
Airflow UI에 접속
19. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성
20. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성
21. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성
22. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성
23. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성
24. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성
25. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성
26. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성
27. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성
28. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성
29. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성
30. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성
31. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성
32. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MWAA 환경 생성
33. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo
34. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
데이터 파이프라인
DW workflow
Glue ETL
(데이터 수집)
ODS workflow
Athena
(테이블 생성)
Athena
(데이터 변환)
EMR Spark
(데이터 변환)
Glue Catalog
S3
Copy
RDS
S3
Redshift
35. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
데이터 파이프라인
DW workflow
Glue ETL
(데이터 수집)
ODS workflow
Athena
(테이블 생성)
Athena
(데이터 변환)
EMR Spark
(데이터 변환)
Glue Catalog
S3
Copy
RDS
S3
Redshift
36. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Apache Airflow 기본 개념
Operator
• Task 수행 방법 및 Task 수행을 위한 도구 보유
• e.g. Python Operator, Bash Operator, Postgres Operator, Email Operator
Task
• Operator가 수행하는 작업
• e.g.
- Python Operator로 일부 API를 호출하여 데이터를 수집하는 작업 구성
- MySQL Operator로 데이터베이스에 데이터를 쓰기 위한 작업 구성
DAG
• Task 스케줄링, Task 수행 순서, Task 실패 시 재시도 등을 정의하는 지침
• e.g.
- Task A가 성공적으로 완료되면 Task B를 실행
- Task 하나가 실패하면 전체 프로세스를 중지하고 알림 발송
37. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Apache Airflow 기본 개념
Connection
• 외부 시스템에 대한 연결 혹은 자격 증명 정보를 안전하게 저장하는 방법
• e.g.
- Postgres Connection = Postgres DB에 대한 연결 정보
- AWS Connection = AWS Access key
Variables
• 임의의 정보를 Key-Value 형식으로 저장하고 Task에서 사용 가능한 변수
• e.g.
- S3 bucket name
- EMR cluster ID
XComs
• Task 간의 메시지를 교환하기 위한 메커니즘
38. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
워크플로우
01.
RDS
데이터 수집
02.
ODS
테이블 생성
03.
DW 구성
04.
Redshift
데이터 로딩
05.
알림 발송
39. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
워크플로우
01.
RDS
데이터 수집
S3
Operator
Glue
Operator
AWS Connection
Variables
40. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
워크플로우
01.
RDS
데이터 수집
02.
ODS
테이블 생성
S3
Operator
Glue
Operator
AWS Connection
Variables
Athena
Operator
AWS Connection
Variables
41. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
워크플로우
01.
RDS
데이터 수집
02.
ODS
테이블 생성
03.
DW 구성
S3
Operator
Glue
Operator
AWS Connection
Variables
Athena
Operator
AWS Connection
Variables
S3
Operator
EMR
Operator
AWS Connection
Variables
Athena
Operator
42. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
워크플로우
01.
RDS
데이터 수집
02.
ODS
테이블 생성
03.
DW 구성
04.
Redshift
데이터 로딩
S3
Operator
Glue
Operator
AWS Connection
Variables
Athena
Operator
AWS Connection
Variables
Postgres
Operator
S3ToRedshift
Operator
AWS Connection
Redshift Connection
Variables
S3
Operator
EMR
Operator
AWS Connection
Variables
Athena
Operator
43. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
워크플로우
01.
RDS
데이터 수집
02.
ODS
테이블 생성
03.
DW 구성
04.
Redshift
데이터 로딩
05.
알림 발송
Email
Operator
Slack
Operator
S3
Operator
Glue
Operator
AWS Connection
Variables
Athena
Operator
AWS Connection
Variables
Postgres
Operator
S3ToRedshift
Operator
AWS Connection
Redshift Connection
Variables
S3
Operator
EMR
Operator
AWS Connection
Variables
Athena
Operator
44. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
45. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Airflow 시스템 통합
46. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS 워크플로우 서비스
Amazon MWAA AWS Step Functions
47. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
유용한 자료
Apache Airflow Documentation
https://airflow.apache.org
AWS MWAA Documentation
https://docs.aws.amazon.com/mwaa/index.html
AWS MWAA Blog
https://aws.amazon.com/blogs/big-data/category/application-
integration/amazon-managed-workflows-for-apache-airflow-amazon-mwaa/
48. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
여러분의 소중한 피드백을 기다립니다.
강연 종료 후, 강연 평가에 참여해 주세요!
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
49. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
감사합니다
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.