Calabash is an open source, cross-platform tool for automating acceptance testing of mobile apps. It allows writing tests in a behavior-driven development style using Gherkin syntax. The tests can query UI elements, interact with apps by tapping and typing, and assert outcomes across Android and iOS apps. Calabash supports resetting app data between tests, waiting for elements to load, and integrating with continuous integration for automated testing.

1. Acceptance Testing
with Calabash
@roland9
roland@intercom.io
Xcake Dublin, 9th Dec 2014

2. What is Calabash?

3. What is Calabash?
• Automated acceptance testing of mobile apps

4. What is Calabash?
• Automated acceptance testing of mobile apps
• Cross-platform (Android & iOS)

5. What is Calabash?
• Automated acceptance testing of mobile apps
• Cross-platform (Android & iOS)
• Open source - Company called Xamarin
backing it

6. What is Calabash?
• Automated acceptance testing of mobile apps
• Cross-platform (Android & iOS)
• Open source - Company called Xamarin
backing it
• https://github.com/calabash/calabash-ios

7. What is Calabash?
• Automated acceptance testing of mobile apps
• Cross-platform (Android & iOS)
• Open source - Company called Xamarin
backing it
• https://github.com/calabash/calabash-ios
• https://github.com/calabash/calabash-android

8. Others
• Frank, KIF, Zucchini, UIAutomation, iCuke, …

9. Why would I do it?

10. Why would I do it?

11. Why would I do it?

12. Other Testing Methods
• Unit Tests
• Integration Tests
• Acceptance Tests

13. Unit Tests
• Tests smallest unit of functionality
public void TestPhoneValidator()
{
string goodPhone = "(123) 555-1212";
string badPhone = "555 12"
PhoneValidator validator = new PhoneValidator();
Assert.IsTrue(validator.IsValid(goodPhone));
Assert.IsFalse(validator.IsValid(badPhone));
}
• Mocks and Stubs to simulate behaviour of other
objects

14. Integration Tests
• Verify that combined modules function correctly

15. Acceptance Tests
• Performing tests on the full system (e.g. using
your web page via a web browser)
• Described in plain English

16. How does it work?
• http://blog.lesspainful.com/2012/03/07/Calabash-iOS/

17. Install

18. Install
• ruby -v -> recommended version 2.0+

19. Install
• ruby -v -> recommended version 2.0+
• gem install calabash-cucumber

20. Install
• ruby -v -> recommended version 2.0+
• gem install calabash-cucumber
• cd path-to-my-ios-project

21. Install
• ruby -v -> recommended version 2.0+
• gem install calabash-cucumber
• cd path-to-my-ios-project
• calabash-ios setup

22. Install
• ruby -v -> recommended version 2.0+
• gem install calabash-cucumber
• cd path-to-my-ios-project
• calabash-ios setup
• calabash-ios gen

23. Install
• ruby -v -> recommended version 2.0+
• gem install calabash-cucumber
• cd path-to-my-ios-project
• calabash-ios setup
• calabash-ios gen
• in Xcode, build -cal scheme

24. Install
• ruby -v -> recommended version 2.0+
• gem install calabash-cucumber
• cd path-to-my-ios-project
• calabash-ios setup
• calabash-ios gen
• in Xcode, build -cal scheme
• cucumber

25. If all goes well:
• LPSimpleExample[11298:13703] HTTPServer:
Started HTTP server on port 37265

26. Build Directory
• export APP_BUNDLE_PATH=/Users/rolandgr/
<…>/Build/Products/Debug-iphonesimulator/
<…>-cal.app

27. Gherkin
• https://github.com/calabash/calabash-ios/wiki/01-Getting-started-guide

28. • Given …
• When …
• Then …
• And …

29. Useful Stuff

30. Useful Stuff
• export DEVICE_TARGET="iPhone 5s (7.1 Simulator)"

31. Useful Stuff
• export DEVICE_TARGET="iPhone 5s (7.1 Simulator)"
• cucumber NO_STOP=1

32. Useful Stuff
• export DEVICE_TARGET="iPhone 5s (7.1 Simulator)"
• cucumber NO_STOP=1
• cucumber -t @current

33. Useful Stuff
• export DEVICE_TARGET="iPhone 5s (7.1 Simulator)"
• cucumber NO_STOP=1
• cucumber -t @current
• cucumber -t @~excludedtag

34. Useful Stuff
• export DEVICE_TARGET="iPhone 5s (7.1 Simulator)"
• cucumber NO_STOP=1
• cucumber -t @current
• cucumber -t @~excludedtag
• calabash-ios console

35. Useful Stuff
• export DEVICE_TARGET="iPhone 5s (7.1 Simulator)"
• cucumber NO_STOP=1
• cucumber -t @current
• cucumber -t @~excludedtag
• calabash-ios console
• > start_test_server_in_background

36. Query Syntax
• query(“button")
=> [
{"class"=>"UIRoundedRectButton", "frame"=>{"y"=>287,
"width"=>72, "x"=>100, "height"=>37}, "UIType"=>"UIControl",
"description"=>"<UIRoundedRectButton: 0x7d463d0; frame =
(100 287; 72 37); opaque = NO; autoresize = RM+BM; layer =
<CALayer: 0x7d46ae0>>"},
{"class"=>"UIRoundedRectButton", "frame"=>{"y"=>215,
"width"=>73, "x"=>109, "height"=>37}, "UIType"=>"UIControl",
"description"=>"<UIRoundedRectButton: 0x7d3a760; frame =
(109 215; 73 37); opaque = NO; autoresize = RM+BM; layer =
<CALayer: 0x7d3a8a0>>”}
]

37. • query("button index:0 label”)
"find the first button, and then inside of that find all
labels”.
• query("button index:0 label", :text)

38. • touch("tabBarButton index:1")

39. accessibilityLabel
accessibilityIdentifier

40. Custom Steps - Ruby API

41. Custom Steps - Ruby API
• https://github.com/calabash/calabash-ios/wiki/03.5-Calabash-iOS-
Ruby-API

42. Custom Steps - Ruby API
• https://github.com/calabash/calabash-ios/wiki/03.5-Calabash-iOS-
Ruby-API
• query, check for existence of elements, wait for elements to exist

43. Custom Steps - Ruby API
• https://github.com/calabash/calabash-ios/wiki/03.5-Calabash-iOS-
Ruby-API
• query, check for existence of elements, wait for elements to exist
• keyboard

44. Custom Steps - Ruby API
• https://github.com/calabash/calabash-ios/wiki/03.5-Calabash-iOS-
Ruby-API
• query, check for existence of elements, wait for elements to exist
• keyboard
• scroll

45. Custom Steps - Ruby API
• https://github.com/calabash/calabash-ios/wiki/03.5-Calabash-iOS-
Ruby-API
• query, check for existence of elements, wait for elements to exist
• keyboard
• scroll
• rotate

46. Custom Steps - Ruby API
• https://github.com/calabash/calabash-ios/wiki/03.5-Calabash-iOS-
Ruby-API
• query, check for existence of elements, wait for elements to exist
• keyboard
• scroll
• rotate
• backdoor -> call method in iOS app (implement in app delegate)

47. Custom Steps - Ruby API
• https://github.com/calabash/calabash-ios/wiki/03.5-Calabash-iOS-
Ruby-API
• query, check for existence of elements, wait for elements to exist
• keyboard
• scroll
• rotate
• backdoor -> call method in iOS app (implement in app delegate)
• macro -> call other step definitions

48. http://www.slideshare.net/mobiletestsummit/ios-automated-testing-with-calabash-tips-and-tricks?
qid=0afecec1-fe64-4168-bb69-a548bd01b832&v=default&b=&from_search=8

49. Tips and Tricks

50. Tips and Tricks
• Resetting app data between scenarios

51. Tips and Tricks
• Resetting app data between scenarios
• Avoid sleep - use wait instead:

52. Tips and Tricks
• Resetting app data between scenarios
• Avoid sleep - use wait instead:
And I wait to see "send button"

53. Tips and Tricks
• Resetting app data between scenarios
• Avoid sleep - use wait instead:
And I wait to see "send button"
• Wait before you interact with any element

54. Tips and Tricks
• Resetting app data between scenarios
• Avoid sleep - use wait instead:
And I wait to see "send button"
• Wait before you interact with any element
• Wait after every action to finish

55. Tips and Tricks
• Resetting app data between scenarios
• Avoid sleep - use wait instead:
And I wait to see "send button"
• Wait before you interact with any element
• Wait after every action to finish
• Escape quotes…

56. Tips and Tricks
• Resetting app data between scenarios
• Avoid sleep - use wait instead:
And I wait to see "send button"
• Wait before you interact with any element
• Wait after every action to finish
• Escape quotes…
• https://groups.google.com/forum/#!forum/calabash-ios

57. Optional: Compare Images
• gem install imagemagick-binaries
• check you have compare: $ which compare

58. Optional: Compare Images
• gem install imagemagick-binaries
• check you have compare: $ which compare

59. Continuous Integration

60. Code Coverage

61. Other Info
• UIKonf talk ‘Unit Testing on iOS’ by Tim
Brückmann https://speakerdeck.com/tibr/unit-testing-
on-ios
• http://calaba.sh
• https://github.com/calabash/calabash-ios
• http://developer.xamarin.com/guides/testcloud/
calabash/introduction-to-calabash/

62. Summary
• Automated Testing - investment that pays off
• Together with unit testing etc: make changes
with confidence (still, manual QA required)
• Cross-platform mature toolset
• Continuous integration / code coverage

64. Acceptance Testing
with Calabash
@roland9
roland@intercom.io
Xcake Dublin, 9th Dec 2014