Copy from SpeakerDeck https://speakerdeck.com/yuzutas0

1. 3
2019-02-13
Data Pipeline Casual Talk #DPCT @
presented by @yuzutas0
https://www.pexels.com/photo/grayscale-photo-of-metal-building-185039/

2. WEB #DPCT
KKD
( / )
Tableau

3. 1.
2. 3
3.
4.
5.

4. @yuzutas0

7.
※

8.
DB id id id etc… ≠ ?
x : y :SUM( ) ≠ ?
is ( )

9.
※ 1 PdM & TL

11. 1.
2. 3
3.
4.
5.

12. 3

14. Data Lake
1

15. Data Lake
ML

16. Data Lake
S3 GCS
DB Dump

18. Data Warehouse

19. Data Warehouse

20. Data Warehouse
Hadoop hdfs Redshift BigQuery

22. Data Mart

23. Data Mart

24. Data Mart
Re:dash Tableau BI
Excel
Excel
※ Excel
※ / BI
※

26. 1.
2. 3
3.
4.
5.

27. 3
3

28. Q.
A.
PC Excel

29. Excel
※
raw

30. 3
1

31. Airflow
1 Air flow
” ”
Data
Excel
Ops

32.
T Data
Ops

35. Data Ops
※ n=1

36. ……
→ Hadoop
→ Fluentd
……
→ Excel
→

37. Data Ops
3

38. 3
or

39. 1.
2. 3
3.
4.
5.

41. 3
3
10 Excel
1

42. ※

43.
※

46.
/

47.
BigQuery
BigQuery 3
BigQuery - Google Cloud Platform
Source
Warehouse
Mart
IF

48. myapp__source__db
2
BEM CSS
source
warehouse
mart
DB

49. WITH 1 1
WITH
SQL

50. SQL on
SQL +
+ SQL → grep
:Cloud Composer (GCP Full Managed Airflow)

51. 1. Tableau
2. SQL on

52. 3
No
→
→
Yes
→
→ BigQuery + SQL + Composer

53. 1.
2. 3
3.
4.
5.

54.
→
→
Q.
A.

55.
3
SRE
not

56. Tableau
Tableau
Tableau Server CSV →
S3
- “Big ball of mud”

57. SRE
BigQuery + SQL

59. AsIs
S3 → Hadoop → Tableau
1

60. ToBe …?
Dev
Ops

62. 10 SQL
https://www.sbcr.jp/products/4797376272.html
https://www.shoeisha.co.jp/book/detail/9784798145143
Neal Ford ―
https://www.oreilly.co.jp/books/9784873118567/
2018 version by @fetarodc
https://www.slideshare.net/tetsutarowatanabe/2018version-115735455

63. DataOps p165-p168
268 4

64. presented by @yuzutas0
https://www.pexels.com/photo/grayscale-photo-of-metal-building-185039/