Financial Planning and Analysis teams often rely on spreadsheets for building data products that provide senior management with analysis and information that is crucial in decision-making. But spreadsheets do not scale and when it comes to expanding models FP&A analysts quickly hit a glass ceiling. The code that FP&A analysts write is in the form of Spreadsheet formulas. In this talk I will show how Spreadsheet formulas and data can be automatically processed at scale inside a Spark cluster by the driver and worker nodes. Essentially this means running a Spreadsheet at scale inside your Spark cluster. I will show how Spreadsheets and their calculated outputs can be transformed into Data Frames for further processing with Spark. We will also discuss next steps in FP&A data pipelines including AutoML and use of such pipelines for Data Science. The broader research topic is in the field of Model-Driven Data Product Design & Development which should be of interest to Spark Summit attendees who are looking for use cases and new opportunities to leverage existing corporate assets like Spreadsheets to automatically build working software that adds tremendous value at scale.

1. #SAISEco2
FP&A with Spreadsheets and
Spark
Oscar Castañeda-Villagrán
Universidad del Valle de Guatemala

2. #SAISEco2
About
2
• Researcher at Universidad del Valle de Guatemala.
• Research Interests:
• Program Transformation,
• Model-driven Data Product Design & Development

3. #SAISEco2
About
3
• Researcher at Universidad del Valle de Guatemala.
• Research Interests:
• Program Transformation,
• Model-driven Data Product Design & Development

4. #SAISEco2
About
4
• Researcher at Universidad del Valle de Guatemala.
• Research Interests:
• Program Transformation,
• Model-driven Data Product Design & Development
‣ Design and Development of

5. #SAISEco2
About
5
• Researcher at Universidad del Valle de Guatemala.
• Research Interests:
• Program Transformation,
• Model-driven Data Product Design & Development
‣ Design and Development of
‣ Large-scale Data Products

6. #SAISEco2
About
6
• Researcher at Universidad del Valle de Guatemala.
• Research Interests:
• Program Transformation,
• Model-driven Data Product Design & Development
‣ Design and Development of
‣ Large-scale Data Products
‣ with Spreadsheets as Models

7. 7
Prototyping …
http://bit.ly/2e5GmyY
#SAISEco2

8. 8
http://bit.ly/2e5GmyY
Prototyping Spark programs with …
http://bit.ly/2edYfMs
#SAISEco2

9. 9
Spreadsheets!
http://bit.ly/2e5GmyY http://bit.ly/2edYfMshttp://bit.ly/2e0TZA8
#SAISEco2

10. 10#SAISEco2

11. 11
Agenda
• Problem Statement and Motivation
• Architecture
• Program Transformation
• Pipeline
• Code-to-Code Transformation
• Code Generation
• Abstract Tree
• Parse Tree
• Spreadsheets as a DSL
• Generating Code
• Demo
• Q&A
#SAISEco2

12. 12
Disclaimer(s)
• Ongoing research …
• FP&A is one use case, but
Spreadsheets are much broader!
#SAISEco2

13. 13
Disclaimer(s)
• Ongoing research …
• FP&A is one use case, but
Spreadsheets are much broader!
#SAISEco2
• E.g. People have even
modeled Turing machines
with Spreadsheets! [1]

14. 14
Problem Statement
Prototype FP&A programs using Spreadsheet formulas
and automatically translate to Scala / Spark.
#SAISEco2

15. 15
Problem Statement
Prototype Any program using Spreadsheet formulas
and automatically translate to Scala / Spark.
#SAISEco2

16. 16#SAISEco2
Motivation
• At Spark Summit Europe 2016 I presented the Sparksheet
code generator for Spreadsheet formulas.
• Initially Sparksheet supported only 5 Spreadsheet
formulas, now it supports 150+ Spreadsheet formulas!
• Motivation is finding use cases.

17. 17
Motivation
Automatically translate
Spreadsheet datasets* to
Spark data pipelines on
Scala/Spark
Data!
http://bit.ly/2expmoF
*Spreadsheet dataset = Structured data + Spreadsheet formulas
#SAISEco2

18. 18
Architecture
Spreadsheet Formulas
Columnar Data
Take ES
snapshot
Restore ES
snapshot
Black Box
Data!
Refine
Code
http://bit.ly/2em6RUK
http://bit.ly/2e5H1jL
#SAISEco2

19. 19
Architecture
Spreadsheet Formulas
Columnar Data
Take ES
snapshot
Restore ES
snapshot
Black Box
Data!
Refine
Code
#SAISEco2

20. 20
Architecture
Columnar Data
Take ES
snapshot
Restore ES
snapshot
Data!
Refine
Code
#SAISEco2
Spreadsheet Formulas

21. 21
Architecture
Columnar Data
Take ES
snapshot
Restore ES
snapshot
Data!
Refine
Code
http://bit.ly/2em6RUK
http://bit.ly/2e5H1jL
#SAISEco2
Spreadsheet Formulas

22. 22
Program Transformation
“A program transformation is any
operation that takes a computer program
and generates another program.”
https://en.wikipedia.org/wiki/Program_transformation
#SAISEco2

23. 23
Program Transformation Pipeline
http://bit.ly/2e0TZA8
http://bit.ly/2efaib4
http://bit.ly/2di0cFq
#SAISEco2

24. 24
Demo #1
http://bit.ly/2e0TZA8
#SAISEco2
1. Show Spreadsheet model
2. Show Complex Spreadsheet Formula

25. 25
Program Transformation
Source program in Excel formula language.
#SAISEco2

26. 26
Demo #2
#SAISEco2

27. 27
Demo #2
#SAISEco2

28. 28
Demo #2
#SAISEco2
Sparksheet
Complex
Spreadsheet Formula

29. 29#SAISEco2

30. 30
Code-to-Code Transformation
#SAISEco2

31. 31
Code-to-Code Transformation
“The input to the code generator typically
consists of a parse tree or an
abstract syntax tree.”
https://en.wikipedia.org/wiki/Code_generation_(compiler)http://bit.ly/2dH0ybF
#SAISEco2

32. 32
Generating Code
“An elegant way to generate code from an AST
is to write a class for each non-terminal node in
the tree, and then each node in the tree simply
generates the piece of code that it is
responsible for.”
http://www.codeproject.com/Articles/26975/Writing-Your-First-Domain-Specific-Language-Part
http://bit.ly/2dH0ybF
#SAISEco2

33. 33
Generating Code
A practical way to generate
code is to take a Parse Tree and
write a pretty printer for the target
language.
http://bit.ly/2em73DM
#SAISEco2

34. 34
Generating Code (Example)
#SAISEco2
SUM(A,C)

35. 35#SAISEco2
SUM(A,C)
Generating Code (Example)

36. 36#SAISEco2
SUM(A,C)
Generating Code (Example)

37. 37
Demo #3
#SAISEco2

38. 38
Demo #3
#SAISEco2

39. 39
Demo #3
#SAISEco2

40. 40
Demo #3
#SAISEco2

41. 41
Demo #3
#SAISEco2

42. 42
Demo #3
#SAISEco2

43. 43
Demo #3
#SAISEco2

44. 44
Spreadsheets as a DSL
#SAISEco2
• Spreadsheet is a powerful data modeling tool.
• Start simple and evolve into a complex ML
pipeline.
• Spreadsheets are suitable to many
domains (FP&A is one such domain).

45. 45
What have we seen?
• Spreadsheet applications as Prototypes for Spark programs
• Program Transformation
• How to model as Pipeline
• Why considered Code-to-Code Transformation
• How to Generate Code
• AST (elegant)
• Parse Tree (practical)
• Spreadsheets as a DSL
• Generating Code
• Next Steps
#SAISEco2

46. 46#SAISDS8
Next Steps
• Use cases!
• Modeling Machine Learning in a Spreadsheet
• Prototype D|’s and ML|’s in a Spreadsheet
#SAISEco2

47. 47
References
• A Grammar for Spreadsheet Formulas Evaluated on Two Large
Datasets – Efthimia Aivaloglou, David Hoepelman & Felienne
Hermans, Proceedings of SCAM ’15
• http://www.felienne.com/archives/2974
• Pictures in presentation from Boards of Canada video “roygbiv”
https://youtu.be/yT0gRc2c2wQ
#SAISEco2

48. 48
Q&A
#SAISEco2

49. THANK YOU.
Email: ofcastaneda@uvg.edu.gt
Twitter: @oscar_castaneda
#SAISEco2