Talk given at Oredev 2015 (http://oredev.org/)

1. Analysing GitHub commits with R
@BasiaFusinska
barbarafusinska.com
barbara.fusinska@gmail.com
https://github.com/BasiaFusinska/RTalk

2. About me
Programmer
Math enthusiast
Sweet tooth
@BasiaFusinska

3. Goals

4. Agenda
• Data analysis
• Capturing GitHub data
• Using R in Data Analysis
– R basics
– Data exploration & processing
– I/O operations
• Azure ML:
– Datasets
– Experiments

5. Data analysis process
Raw Data
Processed
Data
Data Analysis
& Visualization
Exploratory
Data Analysis
Data
Capture

6. What do we want to find out?

7. Language statistics

8. GitHut Visualisation
http://githut.info/

9. Data Capture

10. GitHub Archive
https://www.githubarchive.org/

12. Event Data
Data Source
(Events/Logs/Files)
Store & reorganize
Query

13. Google BigQuery
https://cloud.google.com/bigquery/what-is-bigquery

14. GitHub API
https://developer.github.com/v3/

15. Why R?
• Ross Ihaka & Robert Gentleman
• Name:
– First letter of names
– Play on the name of S
– S-PLUS – commercial alternative
• Open source
• Nr 1 for statistical computing

16. Development environment

17. R Environment
• R project
– console environment
– http://www.r-project.org/
• IDE
– Any editor
– RStudio
http://www.rstudio.com/products/rstudio/download/

18. RStudio
Editor
Console
Environment
variables
Plots
Files
Help
Packages

19. R Basics

21. Filtering

23. Goal: Language distribution

25. Distribution of active repositories per
language

26. What is an active repository?

27. Source of truth

28. GitHub Archive CreateEvent

29. GitHub Archive PushEvent

30. GitHub Archive PullRequestEvent

32. Task: Reading Pull Requests
1. Read the file line by line and
extract only pull request events
2. Extract id and language
information
3. Count and visualise language
distribution
Data: 1h GitHub Archive Events
from 01-01-2015, 3 PM

33. Reading Events

34. Read Pull Requests

35. Unique data

36. Language information

37. Language information output

39. Missing information

40. Omitting information

41. Plotting

42. Now everything is sorted…

43. Goal: Analyze ACTUAL active
repositories

44. Missing data

45. Language information
• Active repositories – Create, Push and
PullRequest events
• Missing language information:
– Google BigQuery
– GitHub API
• Process various data sources

46. Google BigQuery

47. Different sources of data
• GitHub Archive:
– id,
– url in a form:
https://api.github.com/repos/:name
– (rare cases) language
• Google BigQuery:
– no id,
– url in a form:
https://github.com/:name
– language

48. Task: Reading Active Repositories
1. Read the file line by line and extract only
create, push and pull request events
2. Extract id and url information
3. Read Google BigQuery data from saved file
4. Combine repositories data and Google data
base on the same url and fill in missing
language information
5. Count and visualise language distribution

49. Read GitHub Archive

50. Read Google Data

51. Repository data

52. Various data sources

53. Combining data

54. Sorted data

55. Active repositories per languages

56. Goal: Using GitHub API

57. Task: Retrieve language info for
repository
• GET
/repos/:owner/:repo/languages
• Owner: BasiaFusinska
• Repo: RWorkshop

58. GitHub API from R

59. Task: Calling GitHub Search
• GET
/search/repositories
• Querying: q parameter
• Paging: page parameter

60. GitHub API Search

61. Digression

62. Goal: Measure productivity

63. Push events across a week

64. Task: Gather & Save Week Data
1. Read files line by line
and count push events
for every day
2. Fill in the retrieved
data into data frame
3. Save data in the csv
file

65. Reading multiple files

66. Save to file

67. Push Events (CSV File)

68. Exercise: Analyse the week
1. Read activity data from file.
2. Define new column as part of
the day.
3. Calculate mean value for
number of pushes for every
part of the day.
4. Compare and visualize data
for Monday, Wednesday and
Friday.

69. Read week data

70. Analyse week

71. Analyse week output

72. Plot week data

73. Push Events during the week

74. Thank you
barbara.fusinska@gmail.com
@BasiaFusinska
barbarafusinska.com
https://github.com/BasiaFusinska/RTalk