Over the last decade, eXtreme Programming practices like User Stories, Evolutionary Design, Test-Driven Development (TDD), Behavior Driven Developer (BDD), Refactoring, Continuous Integration and Automation have fundamentally changed software development processes and inherently how engineers work. Having experienced various benefits from XP practices on our J2EE stack, our team started to apply these practices to extract, transform, and load (ETL) and Data Analytics side of our product. Unfortunately, there is very little guidance available in this context, esp. for the SAS Platform. Right from finding the unit testing framework to structuring the code to designing our modules and setting up a Continuous Integration builds, our team had to figure out everything, the hard way. Join us to understand the challenges we faced during this process and how we resolved these challenges. Details: http://confengine.com/agile-pune-2014/proposal/535/extreme-programming-for-etl-and-data-analytics Conference: http://pune.agileindia.org/

1. eXtremeProgramming for ETL
Agile Pune 2014
_________________________
Prafulla Girgaonkar [@pgirgaonkar]
Naresh Jain [@nashjain]
Illustration: Umesh Kale

2. 25YEARS
6000+CLIENTS
Nr 1M ROOMS
94COUNTRIES
95%+
ago we started helping clients manage their revenue
later we’re still finding the revenue opportunity in every situation
around the world are priced every single day using IDeaS
on 6 continents have hotels that now run on IDeaS Revenue Management Systems
More than 95 percent of all our clients are still working with us today.
About IDeaS

3. ETL
DECISION
IP
PMS/CRS
Competitor
WebRate
Social Media
Periodically or
Real Time
Full
or
Delta
Messages or
File
*
*
*
*
ETL & IDEAS

4. Concept
Exploration
Structuring
Requirements
Discovery
Earlier ETL Practices

5. Concept
DataExploration
“Structurization”
ETLDevelopment
Certification
GA
Typical Development Cycle on an ETL Project

6. Vendor
IDEAS
Development
Earlier ETL Practices

7. Challenges and Flaws
Source : www.dogri.org/

8. Challenges and Flaws

9. Challenges and Flaws

10. Challenges and Flaws

11. Challenges and Flaws

12. Challenges and Flaws
Time
Discovery
Development

13. Challenges and Flaws
Source : http://roselawgroupreporter.com/

14. Challenges and Flaws

15. Requirement Conformity
Requirement Misunderstanding
Late Data Change
Delays
API Performance
Priorities
Challenges and Flaws -Vendor

16. Naresh Jain
•nareshjain.com
•@nashjain

17. PrafullaS Girgaonkar
Product Development for
Hospitality |Revenue Optimization |Finance
Prafulla.Girgaonkar@ideas.com
@pgirgaonkar

18. Journey
Hilstar
Opera
Pegasus

19. Old Approach

20. Existing Approach -Iterative & Incremental Approach

21. Initial XP Practices -Hilstar

22. DBAQADEVPMUX
Initial XP Practices –Whole Team
Source: http://www.selfishprogramming.com/ BOKM

23. Iteration #1
Iteration #2
Iteration #3
Initial XP Practices –Small Releases

24. Initial XP Practices -Continuous Integration

25. Initial XP Practices –User Story

26. Evolution of the Practices… OPERA
Source: http://www.freedomchurchunited.org/

27. Evolution of the Practices -Pairing

28. Evolution of the Practices –Planning/Grooming
Source: http://www.h4-entertainment.com/

29. Evolution of the Practices –Test Driven Development

30. Evolution of the Practices –Refactoring

31. Evolution of the Practices –Coding Standards

32. Evolution of the Practices –Collective Ownership

33. Evolution of the Practices –Simple Design

34. •The customer is always available
•Code theunit testfirst
•Leaveoptimizationfor later
Evolution of the Practices –Optimization at the End

35. Write a TEST before FIXING the BUG
Evolution of the Practices –Test Driven Bug Fixing

36. Further…Evolution of the Practices… Pegasus

37. Data Management 4GL
MACROS
PROCS
FUNCTIONS
Learn SAS 4gl @
http://support.sas.com/software/ products/ondemand- academics/
Further…Evolution of the Practices –SAS 4GL

38. Further…Evolution of the Practices –Earlier Code

39. No Unit test
No Build Integration
Language Unawareness
Discover the Tool
SASUNIT
Further…Evolution of the Practices –Testability

40. Further…Evolution of the Practices –Code Later

41. Part of build Process
Fail Fast
Further…Evolution of the Practices –Build Process Correction

42. Integrated With Jenkins
Fail Fast
Fail Reports
Further…Evolution of the Practices –Jenkins for SASUNIT

43. Engineering Practices are always at the core.
Tools May not be there.
There is WILL, there is WAY
CRUX

44. Benefits
Source: http://flickr.com

45. Source: http://docustar.com/blog/quality-check-is-99-9-good-enough/
Benefits –

46. Source: http://www.ftjco.com/wp-content/uploads/2012/10/diamonds.jpg
Benefits

47. Benefits

48. Benefits

49. Benefits
Relative Time
Hilstar
Opera
Pegasus

50. UNIT Test …
UNIT Test …
UNIT Test …
Source: www.topnews.in
Benefits

51. Source: http://en.wikipedia.org/wiki/Cello
Benefits

52. Change Management
Source: http://www.animalnational.com/monarch-butterfly-migration/

53. Change Management –Usual Reactions
Source: http://ericgerlach.com/introphilosophy8/
DEVs | QAs | POs | BOs

54. Change Management –Looking for Rituals
Source: http://www.theguardian.com/books/2014/apr/27/reading-print-books-ereaders-technology
DEVs | QAs | POs | BOs

55. Courtesy: Jeff Patton
Change Management –Smaller Release/Stories were Shaker
DEVs | QAs | POs | BOs

56. Change Management –Incremental Development
DEVs | QAs | POs | BOs

57. Change Management –Story Map
DEVs | QAs | POs | BOs

58. Change Management –Spikes were new
Source: http://www.kitchology.com/
DEVs | QAs

59. Change Management –CI & Failing Builds
DEVs | QAs

60. Change Management –Pairing
Source: https://www.airpair.com/mean-stack/posts/2014-10-job-post-mean-stack-developer
DEVs | QAs

61. Change Management –Team Ownership
DEVs | QAs
Source: https://dilbert.com

62. Change Management –Testers part of Design Discussion
Source: http://www.mercatornet.com/articles/view/health_economics_101_contraceptives_pay_for_themselves_usually_babies_dont
DEVs | QAs

63. Change Management –Defect are discussed; not entered
QAs
Source: http://letstalkaboutdiscussion.ca/

64. Change Management –Test Automation
QAs
Source: http://www.mobileqazone.com/profiles/blogs/software-testing-by-human-or-machine-what-should-shouldn-t-be

65. Road Ahead
Source: http://www.gettyimages.in/

66. Engineering Practice -Refactoring / Managing Code Smells
Source:www.123rf.com

67. •Engineering Practice -Simple Design & Solving critical challenges by taking them out of context
Road Ahead -Engineering Practice –Simple Design
Source: http://flickr.com/

68. •Engineering Practice -Simple Design & Solving critical challenges by taking them out of context
Road Ahead -Engineering Practice –Sandbox
Source: http://ifum.com/

69. Road Ahead -Engineering Practice -Evolutionary Design Extended
Source: http://nordicedge.com/

70. Road Ahead -Engineering Practice –Programming skills and polyglot

71. Road Ahead -Real Time Integration

72. WEI JAD
Road Ahead -Joint Application development with Vendor
Source: http://www.p-r-i.org/nadcap/nadcap-meetings/

73. Take Away
•Engineering Practices are universal
–determination and perseverance
•XP Practices lead to T-Shaped individual or Generalizing Specialist
•Product Discovery & Story Mapping vital for Alignment.

74. Prafulla Girgaonkar [@pgirgaonkar] Naresh Jain [@nashjain] Illustration: Umesh KaleSession Ends.. Efforts Continue…