1. The document describes the development of a remote usability testing system that allows observers to remotely view, record, and analyze test participants using mobile applications on their own devices from any location.
2. The goals of the system were to make mobile usability testing more natural and less intrusive for participants by eliminating the need for specialized software or traveling to a lab, while still allowing observers to see the mobile screen, user's interactions, and voice.
3. The approach involved using a camera to capture the mobile screen, transmitting the video and audio streams to observers, and implementing screen capturing for some devices. This would extend their existing usability lab capabilities to remote mobile testing.
23. Approach 1: Camera capturing mobile screen DIY Document Camera PC for transmission of streams to server Webcam Face Capturing Also ideal for testing Paper Prototypes!
Usability Criteria = requirements: Efficiency: How much time and how many steps are required for people to complete basic tasks? Effectiveness: How many mistakes did people make? Satisfaction: How does the person feel about the tasks completed? Is the person confident, stressed? Would the user recommend this system to a friend? Learnability, memorability, errors, flexibility, user in control, etc… .
Improving usability Usability definition (ISO 9241): a product is usable if it enables users to efficiently and effectively achieve their goals in a satisfactory manner Know why you are testing! Know what you are testing! Real Users Usability testing is all about seeing the design through the eyes of the test participants. As they work their way through the design, you get to see and hear what works well and where it becomes frustrating to accomplish their goals. If you've recruited the wrong participant, you'll not learn what you want. Real Tasks Designing the right tasks => don’t guide users in a certain direction by using certain terminology Observe and record Observe : To see and hear what works and what doesn’t Record : Memory and communication tool to stakeholders
Software installation needed: You need to have a PC ready for performing tests => you miss a lot of context You cannot recruit users while they are actually using the application, web site, etc…. . Not all major OS’s: I am a Mac user and need to install fusion or parallels in order to use the observation software Hard to test applications on other OS’s Remote testing: Network issues Have to create a local network for each usability test
Spread of technologies: Java: x% Flash: x% RTMP Streaming server: Broadcasts screen and audio/video Records streams RTMP = Real-Time Messaging and Presence Protocol Adobe Open source
Well we got some bad news today. It was a matter of time but it looks like Apple doesn’t like applications like ScreenSplitr. They now prohibits taking screenshots while in the background and crashes ScreenSplitr when it tries to do so. This isn’t a perfect solution: we couldn’t capture click events and the output from the iPhone was a bit choppy. Also, the video we captured didn’t show the users hands actually interacting with the phone. ScreenSplitr on the iPhone in combination with iDemo on the Mac Pros: ScreenSplittr transmits the iPhone signal via USB so there is less delay, resulting in smoother animations. Cons: Having the iPhone connected to the Mac with a USB cable is less natural for user testers. It also doesn’t show where the user touches the screen. Display Recorder Pros: Records where user touches the screen. Transfers live video right to a browser on a Mac, so no additional software is needed on the Mac side. It also records everything right on the iPhone leading to smooth animations in the final output. Cons: Requires iPhone 3GS. We weren’t able to test this out ourselves. We would need to take the recorded screencast from the iPhone and match it to the recording of the user.
Works with all mobile devices No need to install software on mobile Interaction with mobile is visible
Because we are using RTMP streams it was very easy to exchange the screen-capturing Java-Applet by an extra Flash camera
Problem: Android (but also iPhone) devices prevent you from capturing screens from a separate background service We didn’t want to root or jailbrake devices, because you cannot do that with participants’ mobiles! Solution: Connect mobile to PC, running ADB (debug) server Java application, running on the PC, acts as a client to the debug server requesting frame buffers from the mobile device
Sip = session initiation protocol audio / video streaming over the Internet VoIP RTMP = real-time messaging and presence protocol Audio / video streaming over the Internet