Slides of the talk on the paper "Beyond Surveys: Analyzing Software Development Artifacts to Assess Teaching Efforts" by Christoph Matthies, Ralf Teusner and Guenter Hesse given at the Frontiers in Education 2018 conference in San Jose, CA, USA in October 2018. Preprints of the paper are available on arXiv (https://arxiv.org/abs/1807.02400)

1. Hasso Plattner Institute
University of Potsdam, Germany
christoph.matthies@hpi.de
@chrisma0
Beyond Surveys:
Analyzing Software Development Artifacts
to Assess Teaching Efforts
Christoph Matthies, Ralf Teusner, Guenter Hesse
’18, San Jose, CA, October 2018

2. “
Background
Course Focus
You will learn how to manage a long-running software
project with a large number of developers. [1]
2[1] https://hpi.de/plattner/teaching/archive/winter-term-201718/softwaretechnik-ii.html
”
An undergraduate software engineering capstone course

3. “
Background
Course Focus
You will learn how to manage a long-running software
project with a large number of developers. [1]
3[1] https://hpi.de/plattner/teaching/archive/winter-term-201718/softwaretechnik-ii.html
■ All participants (in teams) jointly develop a software
■ Self-organizing teams
■ Collaboration > technical skills
”
An undergraduate software engineering capstone course

4. “
Background
Course Focus
You will learn how to manage a long-running software
project with a large number of developers. [1]
4[1] https://hpi.de/plattner/teaching/archive/winter-term-201718/softwaretechnik-ii.html
■ All participants (in teams) jointly develop a software
■ Self-organizing teams
■ Collaboration > technical skills
■ Project work & intro exercises & lectures & tutors
■ Learn and apply agile methods
■ Real-world scenario, real development tools
”
An undergraduate software engineering capstone course

5. Challenges & Ideas
■ Course employed Scrum
■ Structured, prescriptive
■ Good for starting [2]
■ Kanban gaining popularity in industry
5
Evaluate and adapt the course over time
[2] V. Mahnic, “From Scrum to Kanban: Introducing Lean
Principles to a Software Engineering Capstone Course”

6. Challenges & Ideas
■ Course employed Scrum
■ Structured, prescriptive
■ Good for starting [2]
■ Kanban gaining popularity in industry
■ Idea: Update course and project!
■ Employ Scrum Sprints first, then switch to Kanban
■ Kanban for “finishing touches”
6
Evaluate and adapt the employed development methodology
[2] V. Mahnic, “From Scrum to Kanban: Introducing Lean
Principles to a Software Engineering Capstone Course”

7. Challenges & Ideas
7
Scrum: the usual agile development process
Scrum key ideas
■ Sprints: timeboxed
iterations
■ Planning and estimation
■ Review and
retrospectives
■ Prescriptive process

8. Challenges & Ideas
8
Kanban: the new kid on the block
Kanban key ideas
■ Visualize work items
on Kanban board
■ “Pull” them through
the process
■ Limit work-in-progress
■ Handle bottlenecks

9. Research Question
9
Goals of the research and background
How can we gauge (the effect of curriculum)
changes in student behavior during project work?

10. Research Question
■ Usual approach: end-of-term surveys
10
Goals of the research and background
How can we gauge (the effect of curriculum)
changes in student behavior during project work?

11. ■ Performed regularly after end of course (before grades)
■ Allows student feedback on the course
■ Standardized questions, overwhelmingly positive responses
→ Hard to gauge changes in curriculum design
End-of-term Surveys
11
General satisfaction indicators of the last 5 years

12. Research Question
■ Usual approach: end-of-term surveys
12
Goals of the research and background
How can we gauge (the effect of curriculum)
changes in student behavior during project work?

13. Research Question
■ Usual approach: end-of-term surveys
■ Programming project provides unique opportunity
■ Developers regularly produce software artifacts
■ GitHub: Version control system, issue tracker
13
Goals of the research and background
How can we gauge (the effect of curriculum)
changes in student behavior during project work?

14. ■ Collected data from the last 5 course instalments
■ Last 3 course iterations introduced Kanban
■ The 2 before these used only Scrum
Development Artifact Collection
14
Comparing development artifacts over course instalments

15. ■ Collected data from the last 5 course instalments
■ Last 3 course iterations introduced Kanban
■ The 2 before these used only Scrum
Crawled GitHub APIs and extracted artifacts
Development Artifact Collection
15
Comparing development artifacts over course instalments
commits
tickets / user stories

16. Development Artifact Analysis
Kanban usage had some noticeable effects
16
Gaining insights into team behaviors

17. Development Artifact Analysis
Kanban usage had some noticeable effects
■ Higher mean amount of non-comment events
■ Assigning labels (status, priority) & developers (responsibility)
→ More interaction with issues
17
Gaining insights into team behaviors

18. Development Artifact Analysis
Kanban usage had some noticeable effects
■ Higher mean amount of non-comment events
■ Assigning labels (status, priority) & developers (responsibility)
→ More interaction with issues
■ Commits towards end of Sprint higher in Kanban
■ Scrum (planning) vs Kanban (dynamic)
→ Better ability to adapt to changes
18
Gaining insights into team behaviors

19. Development Artifact Analysis
Key development artifacts measures did not change significantly
■ Mean amount of commits & touched files
■ Mean line changes per commit
■ Mean amount of unique issues referenced
■ Mean issues closed, mean comments
19
Gaining insights into team behaviors

20. Development Artifact Analysis
Hypotheses regarding changes in artifacts were violated
20
Gaining insights into team behaviors

21. Development Artifact Analysis
Hypotheses regarding changes in artifacts were violated
■ Similar percentage of issues opened and closed by same person
■ No dedicated Product Owner role
→ Expected higher engagement of entire team
21
Gaining insights into team behaviors

22. Development Artifact Analysis
Hypotheses regarding changes in artifacts were violated
■ Similar percentage of issues opened and closed by same person
■ No dedicated Product Owner role
→ Expected higher engagement of entire team
■ No change in user story length
■ No need to estimate, focus on throughput
→ Expected smaller user stories
22
Gaining insights into team behaviors

23. Kanban Survey
Survey in 2017/18 course instalment (N=18, 5 point Likert scale)
23
Asking the questions that actually matter

24. Kanban Survey
Survey in 2017/18 course instalment (N=18, 5 point Likert scale)
■ “Was the Kanban sprint more useful and
productive than another Scrum sprint?”
■ Yes!, mean 4.08
24
Asking the questions that actually matter

25. Kanban Survey
Survey in 2017/18 course instalment (N=18, 5 point Likert scale)
■ “Was the Kanban sprint more useful and
productive than another Scrum sprint?”
■ Yes!, mean 4.08
■ “Did you adapt your workflow?”
■ Yes., 3.83
25
Asking the questions that actually matter

26. Kanban Survey
Survey in 2017/18 course instalment (N=18, 5 point Likert scale)
■ “Was the Kanban sprint more useful and
productive than another Scrum sprint?”
■ Yes!, mean 4.08
■ “Did you adapt your workflow?”
■ Yes., 3.83
■ “Biggest (dis)advantages of Kanban?” (free text)
■ Advantages: Efficiency & Autonomy
■ Drawbacks: Only work on small stories,
uneven task distribution
26
Asking the questions that actually matter

27. Kanban Survey
Survey in 2017/18 course instalment (N=18, 5 point Likert scale)
■ “How did user stories change from using Scrum to Kanban?”
■ More bug-oriented (11 mentions)
■ Shorter (11 mentions)
■ With more detailed requirements (8 mentions)
27
Asking the questions that actually matter

28. Kanban Survey
Survey in 2017/18 course instalment (N=18, 5 point Likert scale)
■ “How did user stories change from using Scrum to Kanban?”
■ More bug-oriented (11 mentions)
■ Shorter (11 mentions)
■ With more detailed requirements (8 mentions)
■ “Would you recommend using Kanban
to next year’s participants?”
■ YES!, mean 4.33
28
Asking the questions that actually matter

29. Summary & Conclusion
■ Kanban introduction was liked by students, but w/ mixed success
29
Take-away messages
christoph.matthies@hpi.de @chrisma0

30. Summary & Conclusion
■ Kanban introduction was liked by students, but w/ mixed success
■ Development artifacts represent another dimension of analysis
■ Beyond the perceptions of students
■ Based on data naturally produced, high “response rate”
30
Take-away messages
christoph.matthies@hpi.de @chrisma0

31. Summary & Conclusion
■ Kanban introduction was liked by students, but w/ mixed success
■ Development artifacts represent another dimension of analysis
■ Beyond the perceptions of students
■ Based on data naturally produced, high “response rate”
■ Analysis allowed finding those areas where expectations are…
■ Confirmed
■ Violated! (even more interesting)
31
Take-away messages
christoph.matthies@hpi.de @chrisma0

32. Summary & Conclusion
■ Kanban introduction was liked by students, but w/ mixed success
■ Development artifacts represent another dimension of analysis
■ Beyond the perceptions of students
■ Based on data naturally produced, high “response rate”
■ Analysis allowed finding those areas where expectations are…
■ Confirmed
■ Violated! (even more interesting)
→ Opportunity for conversation and improvement
32
Take-away messages
christoph.matthies@hpi.de @chrisma0

33. Image Credits
33
In order of appearance
■ Archaeologist by Gan Khoon Lay from the Noun Project (CC BY 3.0 US)
■ Mortar Board by Mike Chum from the Noun Project (CC BY 3.0 US)
■ Target by Arthur Shlain from the Noun Project (CC BY 3.0 US)
■ Process by Laymik from the Noun Project (CC BY 3.0 US)
■ Questions by Gregor Cresnar from the Noun Project (CC BY 3.0 US)
■ Data collection by H Alberto Gongora from the Noun Project (CC BY 3.0 US)
■ Search Code by icon 54 from the Noun Project (CC BY 3.0 US)
■ Clipboard by David from the Noun Project (CC BY 3.0 US)
■ Idea by Gilbert Bages from the Noun Project (CC BY 3.0 US)