Contenu connexe Similaire à 20170210 sapporotechbar7 (20) Plus de Ryuji Tamagawa (20) 20170210 sapporotechbar72. ▸ facebook : Ryuji Tamagawa
▸ Twitter : tamagawa_ryuji
▸ FB
techbar
▸ FB
▸ Twitter
9. 1 / 5 : PyData
PyData.org
10. 1 / 5 : PyData
PyData
Anaconda Python
Blaze NumPy and pandas interface to Big Data'. dask
Bokeh
Canopy Python
IPython
matplotlib PyData
nose
numba JIT
NumPy PyData
Scipy PyData
Statsmodels
SymPy
pandas NumPy SciPy
scikit-image
scikit-learn PyData
12. 2 / 5 : pandas
pandas
▸ NumPy SciPy
▸ DataFrame
▸
13. 2 / 5 : pandas
pandas
Wes McKinney
14. 2 / 5 : pandas
DataFrame
16. 2 / 5 : pandas
▸
Python
▸
▸ PyData pandas
18. 3 /5 : Jupyter Notebook
IPython Notebook
▸ Jupyter Notebook
▸ Julia Python R
▸ JupyterCon
21. 3 /5 : Jupyter Notebook
pandas / matplotlib
22. 3 /5 : Jupyter Notebook
Interactive Widget
23. 3 /5 : Jupyter Notebook
▸ Learning Jupyter
25. 4 / 5 : Apache Spark
Hadoop
▸ MapReduce Spark
▸ 2010 Hadoop = MapReduce + HDFS
▸ Hadoop
OS
HDFS
Hive e.t.c.
HBaseMapReduce
YARN
Impala
e.t.c in-
memory SQL
engine
Spark
Spark Streaming, MLlib,
GraphX, Spark SQL)
Hadoop
HDFS S3
YARN Mesos
/
26. 4 / 5 : Apache Spark
Apache Spark PyData pandas
Apache Spark pandas
JVM Python
× dask
I/O
Scala Java Python R
JVM
Python
27. 4 / 5 : Apache Spark
Spark
▸
▸
▸ 1 PC
Hadoop / MapReduce
28. 4 / 5 : Apache Spark
DataFrame
29. 4 / 5 : Apache Spark
▸
▸ SSD
▸ Spark Parquet
▸ Performance comparison of different file formats
and storage engines in the Hadoop ecosystem
▸ Parquet Python
30. 4 / 5 : Apache Spark
Apache Spark
▸
▸ Parquet
▸
▸
34. 5/5 : 2017 and the future
PyData
▸
▸ Spark - pandas
▸ pandas → Spark …
35. 5/5 : 2017 and the future
Wes blog
▸ pandas Apache Arrow
▸ Blog
▸ PyData Blog
Wes OK
▸ 2017 : pandas, Arrow, Feather, Parquet, Spark, Ibis
http://qiita.com/tamagawa-ryuji/items/deb3f63ed4c7c8065e81
36. 5/5 : 2017 and the future
High speed Apache Parquet for Python
▸ Parquet
▸ Spark
▸ Python
▸ Fastparquet
▸ pyarrow
37. 5/5 : 2017 and the future
: apache arrow
▸ apache arrow
▸ PyData / OSS
▸ /