Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à TechTalk: Wind Tunnel, Personas, and Testing Real UX
Similaire à TechTalk: Wind Tunnel, Personas, and Testing Real UX (20)
Plus de Lizzy Guido (she/her)
Plus de Lizzy Guido (she/her) (15)
Dernier
Dernier (20)
TechTalk: Wind Tunnel, Personas, and Testing Real UX
- 1. Wind Tunnel, Personas and Testing Real UX Partners Tech Talk Webinar David Broerman Partner Enablement Manager Email: davidb@perfectomobile.com
- 2. Agenda • Intro to Perfecto • Who is Perfecto • Basic Architecture/Setup • Digital Challenges • Overview of the Perfecto CQ Lab and Wind Tunnel • Wind Tunnel and Personas • What is a Persona? Existing and Custom Personas • Adding Wind Tunnel capabilities to your test • Easily add a Persona right into your script • Reporting and Digital Zoom • Sample Test and Demo that includes Wind Tunnel • Eclipse/Java/Selenium or Appium © 2016, Perfecto Mobile Ltd. All Rights Reserved. 2
- 3. Intro to Perfecto Perfecto Offers: A Cloud Based Continuous Quality Lab enabling testing and monitoring of all digital platforms on real devices and real end user conditions, for enterprises adopting agile delivery processes.3
- 4. Intro to Perfecto Dominant leader & visionary Strong Ecosystem Perfecto @ a glance Working with Enterprises and Partners to deliver exceptional digital experiences. “Perfecto Mobile marks shift in mobile apps QA with Continuous Quality Lab” 4 Setting the bar for market success • Founded in late 2006, Business HQ in Boston • Over 300 employees globally • Fortune 500 customer base
- 5. Cloud Based Quality Platform For The Digital Era 5© 2017, Perfecto Mobile Ltd. All Rights Reserved. Open & Integrated into Dev/Test/Ops Any test tool Any CI server Any test framework Any IDE Any Location Code/Debug Functional Test Performance Test Production Insight One Lab for all Digital Platforms Reporting & Analytics Wind Tunnel Layer Customer Support Browsers Desktops Mobile Devices IoT
- 6. Architected for Digital Quality at High Velocity 4/10/2017 6© 2017, Perfecto Mobile Ltd. All Rights Reserved. Maximize Digital Test Coverage Maximize % of Automation 1. Cloud-first architecture total reliability, unlimited scale, secure, global and available 24/7 2. Platform connectivity layer™ (PCL) architecture • Making open source enterprise grade • Enables high test automation coverage 3. One Script, One Lab for all digital platforms 4. Embedded into your toolchain to streamline activities 5. WindTunnel™ Layer persona based user environment testing for true digital coverage 6. DigitalZoom™ Reporting provides cross platform visibility and fast drill down Quality Across Dev- QA-Ops Fast Feedback Loop FOUR PILLARS OF SUCCESS
- 7. Perfecto PCL™ - Architectural Workflow 4/10/2017 7© 2017, Perfecto Mobile Ltd. All Rights Reserved. Standard Open Source Platform Under Test Standard Appium Server/ Selenium Grid Standard OSS Test Code In Dev IDE’s Platform Under Test Standard OSS Test Code In Dev IDE’s Standar d OSS Standard Appium Server/ Selenium Grid Perfecto PCL™ Benefits Standard Open Source With Perfecto PCL™ Test across platforms Same day support for any platform using any test framework Enhanced test coverage • Full system level control • Handle interruptions: text msg, system msg • Test audio (e.g. voice commands) • Test image (e.g. check deposit) • Test environment conditions (Wind Tunnel™) Visual validation of screen layout Always On lab
- 9. Global Data Centers Network 94/10/2017
- 10. Mobile Experience Quality Challenge The Mobile Experience Quality Challenge 1) Building common user environment condition profiles into test suites 2) Assembling multiple solutions to execute tests across profiles 3) Seamlessly integrating user condition testing within Continuous integration pipelines 4) Analyzing app performance blind spots 5) How to set up a lab and these various conditions? © 2016, Perfecto Mobile Ltd. All Rights Reserved. 10
- 11. Digital Presents Several Challenges for Testing Teams 11 Issue Negative Consequence 1 Separate tools and labs for web and mobile teams. • Silo-ed test methods result in redundant investments • Differing tools, differing expertise 2 Timely access to required lab resources in a dynamic market. • Release schedules slowed down • Quality risks go unmitigated due to inadequate coverage 3 Lack of easy visibility to user experience across all browsers and devices • No strategic quality of UX management • Inefficient root cause analysis Issue Negative Consequence 1 Separate tools and labs for web and mobile teams. • Silo-ed test methods result in redundant investments • Differing tools, differing expertise 2 Timely access to required lab resources in a dynamic market. • Release schedules slowed down • Quality risks go unmitigated due to inadequate coverage Issue Negative Consequence 1 Separate tools and labs for web and mobile teams. • Silo-ed test methods result in redundant investments • Differing tools, differing expertise
- 12. Perfecto Continuous Quality Lab One Script, One Lab for all Digital Platforms © 2015, Perfecto Mobile Ltd. All Rights Reserved.
- 13. User Centric Quality Lab with End User Conditions Using a Mobile App 1 2 Network Quality Cellular, Wi-Fi 3 Varied Device Conditions 4 Multiple devices Specific profiles 5 Location 6 Resource Conflicts 7 Interacting with sensors Peak time - Millions of transactions 8 Simple Functional Testing Is Not Enough 13
- 14. Overview of Perfecto Wind Tunnel Wind Tunnel Profile Selector Usage Patterns 1. Login 2. Search 3. Select 4. Payment 5. Logout Unchanged Test Flows Example: User Journey on Banking/Financial App Existing Flows Analyze & Optimize • Lives in NY • iPhone 6S • Verizon Customer • Constant WhatsApp msg Georgia 12 3 4 PersonaPoints of Interest Wind Tunnel Testing Wind Tunnel Report © 2016, Perfecto Mobile Ltd. All Rights Reserved. 14
- 15. The Personas • The Perfecto Wind Tunnel personas define the test environment • A Persona is defined by a set of capabilities • There are 5 available pre-defined personas • Personas can be created and edited • Georgia • 47 year old high income, corporate road warrior living in Manhattan traveling to San Fran & London • Traits • Network: Constantly connected, frequent network switches across various WiFi and cellular networks • Brands of choice: Apple & Verizon • Device(s): iPhone 6S, Smartwatch & iPad Pro • App loyalty: Intense social user, WhatsApp, Mobile Banking and PO & Expense Report approvals Location Carrier & Brands Device coverage Background apps © 2016, Perfecto Mobile Ltd. All Rights Reserved. 15
- 16. 5 Pre-Defined Personas 16© 2016, Perfecto Mobile Ltd. All Rights Reserved.
- 17. Wind Tunnel – Languages & Frameworks 17© 2016, Perfecto Mobile Ltd. All Rights Reserved.
- 18. Using & Tailoring Personas • Set a persona using the windTunnelPersona capability • Tailor an existing persona by defining new capabilities capabilities.setCapability("windTunnelPersona", "Georgia"); capabilities.setCapability("windTunnelPersona", "Georgia"); //Add location either by an address or lat,long coordinates capabilities.setCapability("windTunnelLocation", "New York City,NY"); //Set orientation to landscape and portrait capabilities.setCapability("windTunnelOrientation", "portrait"); //list the apps running in the background capabilities.setCapability("windTunnelBackgroundRunningApps", "Candy Crush Saga"); © 2016, Perfecto Mobile Ltd. All Rights Reserved. 18
- 19. Persona JSON file • Example persona JSON file{ "settings":{ "windTunnelLocationAddress":"New York, NY", "windTunnelBackgroundRunningApps":"United,Twitter,Outlook,Marriott,BofA", "windTunnelVNetwork":"4G LTE Poor", "windTunnelOrientation":"landscape" }, "properties":{ "description":"Corporate Road Warrior, Age 47", "name":"Georgia", "image":"PUBLIC:Personas/Georgia.png"}, "device":{ "model":"iPhone-6S||iPad Mini 4" } } © 2016, Perfecto Mobile Ltd. All Rights Reserved. 19 capabilities.setCapability("windTunnelPersona", "Georgia"); Example of Reference Persona – remove Georgia
- 20. Points of Interest •The focus points that are part of your Wind Tunnel execution and reports •The app issues that are being tested in real user conditions Examples: • Purchase transaction that takes too long • CPU consumption of the running Facebook app © 2016, Perfecto Mobile Ltd. All Rights Reserved. 20
- 21. Adding Points of Interest to the test •Java Example • Following code adds a Point of Interest that will add an indication of a successful page load. • Will be selectable in the wind tunnel report ... // Wind Tunnel: Add success point of interest to the Wind Tunnel report WindTunnelUtils.pointOfInterest(driver, "Point Of Interest 1: Page load","Success"); © 2016, Perfecto Mobile Ltd. All Rights Reserved. 21
- 22. How to Create a Wind Tunnel Script • Determine IDE and language • Download Perfecto Plug In • Create a new Perfecto Lab Project • Add sample code to your project (include persona for wind tunnel) • Run the test • Review the run results/report © 2016, Perfecto Mobile Ltd. All Rights Reserved. 22
- 23. Reporting and Digital Zoom
- 24. Reporting - Usage and Flow Link Execution Capability Download PDF: PerfectoLabUtils.downloadReport(driver, "pdf", "C:testreport"); Link to STR - String reportURL = (String)(driver.getCapabilities().getCapability(WindTunnelUtils.SINGLE_TEST_REPORT_URL_CAPABILITY Reporting Tab Capability – Get Grid URL & create testName Reporting Tab Grid View/Dashboard Legacy Test Report Single Test Report 1 2 3 4 5
- 25. Single Test Report The Single Test Report timeline includes: • Points of interest • Wind Tunnel events • Checkpoints (validation) © 2016, Perfecto Mobile Ltd. All Rights Reserved. 25
- 26. DigitalZoom™ Reports - Visibility, Traceability, Fast Analysis 4/10/2017 26© 2017, Perfecto Mobile Ltd. All Rights Reserved. • One cloud BI Reporting system f From executive level to root cause analysis for engineers • Full Flexibility to view any granularity • One click drill down with full traceability between reporting levels • Out-of-the-box digital relevant filters + Build any custom filters • Collaboration capabilities for remote joint analysis • Digital View • Fast analysis • Fast feedback loop
- 27. Demo
- 28. Question/Answer & Wrap-Up Thank You for attending! David Broerman Partner Enablement Manager Email: davidb@perfectomobile.com
Notes de l'éditeur
- Welcome everyone and thanks for attending today’s session on Wind Tunnel, Personas and Real User Experience Testing My name is ….
- The Perfecto CQ (Continuous Quality) Lab platform is a cloud architected quality platform. One lab for all digital platforms, from browsers, desktops to mobile to IoT We have capabilities to mimic the real user environment with our Wind Tunnel Including Location, orientation, network conditions like 3g 4g, and everything you need to reproduce your users environment While providing Quality visibility and trending with our reporting and analytics for fast feedback loops Completely accessible/embeddable in your SDLC process and tools Accessible from coding to customer support To integrated in Any test tool, CI server, test framework, IDE and location
- How is our perfecto hosted cq lab different? There are six key capabilities that make our lab uniquely capable to satisfy all four pillars of success They are: Cloud is not hosting devices. It’s much more than that. We connect differently to platforms than anyone else. This removes instability and barriers to high automation. One lab for responsive web is an example, multiple screen sizes can be tested using the same script across both mobile and desktop browsers. Our solution is architected to allows us to detach our lab from the interface – allowing the lab to be embedded across tools We are the only vendor with the ability to perform persona based testing, defined as a parameter in you test scripts. We decouple the persona conditions form the test case, allowing the same test to be run across different personas (with different network, location, background apps, etc). It is unique in that no reporting tool on the market provides a comprehensive digital view with the flexibility to see digital platform in one view, and zoom in to the point of failure, by platform/and step in the flow.
- Our platform connectivity layer provides a many benefits over standard open source. Same day support was a big pain encountered in 2017 for those not using Perfecto. lack of support for latest iOS platform (iOS 10, iPhone7) - It took 4 months for Appium to support iOS 10 in a GA release. iOS10 adoptions was at 54% of the market within the first 6 months. With Perfecto, our customer saw 0 interruption. You can even test on beta platforms the day they are released. Open source has no beta platform support. This can be the difference between testing on new platforms a month or more before they are released to the public. The Perfecto hosted cloud has a built in implementation of Remote Web Driver. So you are able to use Selenium and Appium through Perfecto’s Server. So for example, instead of having to set up an Appium Server locally, and then tether devices (1 iOS device per mac or a couple of Android devices to a windows machine), you can just connect to Perfecto’s server instance through the cloud and interact with the devices like you had them in your hand. So the benefit there is you or your team not having to deal with a local server instance, plus not having to manage physical devices or plugging them into machines locally. A few other benefits with using Perfecto’s instance, is being able to interact with objects using visual, setting end user conditions (like GPS location, phone calls and sms messages), network virtualization (3G 4G), and capturing device vitals (CPU, Memory and Bytes in/Bytes out). The bottom line is that it’s the same Selenium/Appium that users are used to working with, plus the additional Perfecto functionalities that we bring to the table. This setup allows you and your teams to be more productive, test on real devices with those all important real end user conditions PCL (platform connectivity layer) provides control over how we interact with the device, without modifying the interface or the jailbreaking or rooting the device itself, while providing system level control to the device. It is this capability, the PCL, that enables complex flows to be automated, enabling you to get to 80- 90% automation.
- The first picture is inside one of Perfecto’s Secure Datacenters, with racks of servers and devices locked in cabinets. Notice in the second picture, that we have real devices connected, via a USB cable, to the Perfecto Handset Server (or HSS). The HSS, which is responsible for managing all device connectivity, gets its commands from the MobileCloud manager (MCM) – which is a server. The MCM is the component responsible for executing all of the scripts as well as storing within its repository and reports. Through this USB cable we get a live streaming from the device and are able to control the device (what we call system level control); including any device real user commands/gestures (such as tap, swipe, click etc.). We don’t jailbreak/root or take the devices apart. We purchase them directly from the manufacturer, or we can take devices from you or a customer and also put them in the cloud.
- Perfecto operates 14 global data centers, including North America, South America, Canada, Israel, Europe, India (Gujarat - Guu Ju Rat) , Australia, and China. Photos: Top Left & Right – WiFi paddles/repeaters enabling increased device density | Top Right – Another Picture inside one of our Datacenters Bottom Left, FCC approved cellular tower repeater | Bottom Right, GPS signal repeater Center: Global NOC located in Wakefield, MA
- Different users at different points of the day go through different experiences (user flows of the app) in different environments. For example, a travelling business person who is on and off planes and is exposed to poor reception; and a gamer will have a lot of CPU consuming apps running in the backgrounds. Devices are configured differently, in terms of permissions, access, the number of background apps running. For example, there may be apps using location. There is a lot happening outside the control of my app.
- Digital Presents Several Challenges for Testing Teams and Perfecto wants to be able to help you with those challenges:
- Perfecto can Help with our Cloud Based Continuous Quality Lab – with One script and one lab for all digital platforms
- Let’s dig into Quality. Mobile quality differs dramatically from web. Specifically it is the user conditions. Therefore Quality and Dev teams must look beyond testing just a function toward testing all functions across all user conditions. The objective is to set defining and controlling user conditions as a key required capability. This can be expressed as a day in the life of how you use your favorite app. Think about traveling from point a to b and various network, device, device resources consumed and potentially conflicting apps that are also running on your device. App performance is influenced by a variety of factors, such as degraded network conditions, frequent network transitions back and forth between cellular and WiFi, on board sensor or resource conflicts along with a non-stop stream interruptions – SMSs, notifications, system pop-ups and phone calls. These factors impact app performance. Mobile app quality programs must account for these factors during testing to consistently deliver high-quality end user experience.
- Can now take real end user experiences and bring into the Perfecto CQ lab. Test conditions, unplanned events, environment changes and expand the test coverage and predictability for what is released to production. You just can’t do these types of tests on emulators! You can also add Wind Tunnel to an existing script and that will help expand your test coverage to include real user experience. Wind Tunnel allows testing on what is going on outside the control of the application and not just what is inside the application. This includes the application server, the carrier network, and the device OS. For example, testing what happens when an incoming call event occurs during a transaction in a bank application. The solution Wind Tunnel offers the following abilities to extend coverage to real user conditions when testing user journeys or flows. Number 1 - Pre-defined Personas that simulate network, location and other properties. Number 2 - Points of interest that identify crucial application flows such as login, search, select, payment,logout Number 3 - Timer reporting ability to track timings and performance of the application. Example might be how long it took for a page to load or a process to happen, and what that actually looks like for a user (so the UX or user experience on the device glass). Number 4 - All these abilities are easy to add to the test and visible in a Report designed to help in understanding and drill down of issues.
- Defining the real user environment is a challenge. Perfecto has created the concept of Persona in which a type of user is defined. Each persona contains a series of properties that together simulate a real end user. By setting a persona all its pre-defined properties are set. It is possible to use the Perfecto provided profiles and also to create custom profiles. There are five different available personas ready to use (with unique capabilities). For example, Georgia, a 47 year old high income, corporate road warrior living in Manhattan, traveling to San Fran & London. Her traits include: Network: Constantly connected, frequent network switches across various WiFi and cellular networks Brand of choice: Apple & Verizon App loyalty: Intense social user, WhatsApp, Mobile Banking and PO & Expense Report approval. Georgia’s list of background apps allows testing your app, considering the resources consumed by the other apps constantly running on Georgia’s device, such as high CPU, network consumption, camera, etc. These traits are translated into capabilities, creating Georgia's persona that can be easily incorporated into your test.
- These are the five different available personas ready to use. See above image and detailed table for their unique capabilities. Existing tests can be executed across the set of personas. The personas can be added individually to a test, or parameterized and run utilizing common continuous integration servers, such as Jenkins or test execution managers such as TestNG.
- Perfecto agnostically supports a wide variety of languages and test frameworks
- To set a persona use the windTunnelPersona capability. Example - java To tailor an existing persona with specific values, define new capability values to override those that have already been defined in the persona.
- What is a persona? A persona is a set of capabilities stored in the format of a JSON file located in the Perfecto lab repository. The pre-defined persona JSON files are located in PUBLIC:Personas folder (for example: PUBLIC:Personas/Georgia.json). When you set values of elements common to a persona definition, you override the persona’s original corresponding value (e.g. setLocation or mobile:location:set override the persona location). Notice Georgia has more than one device, therefore, unless you have overridden that value, one of her devices is randomly selected for the test execution. If you do not need any specific persona attributes, yet you do wish to add the Wind Tunnel mechanism to your test, you can avoid adding a persona altogether, use other Wind Tunnel commands, and still generate the report. A reference persona with mostly empty fields is automatically added to your capabilities, with some outline image appearing in the Wind Tunnel report.
- The Wind Tunnel Point of interest (POI) allows mobile teams to pin point real user specific issues around performance, functionality and more. By using POI users can get a detailed dashboard with view into the actual error for a specific transaction. For example, a purchase transaction that takes too long (see the Wind Tunnel’s Timer functionality), or the CPU consumption of the running Facebook app (in the vitals’ CPU graph). This allows identifying app quality issues in specific environment under specific conditions before app delivery to production, assuring end-user satisfaction and eliminating redundant long debugging cycles.
- Points of interest are the points in the script that the user may want to log, in order to highlight in the Wind Tunnel report. For example, loading a new page. The user can also use the point of interest to indicate a logical failure such as incorrect data showing in the app. ** For details and code snippets see the Adding Wind Tunnel to your test post in the Wind Tunnel guide. https://community.perfectomobile.com/series/24692/posts/1048049
- Notes: next slide we’ll run the sample test.
- Here is some additional information on Perfecto Reporting, 1 –Execution of tests from your favorite IDE, language, testing framework 2 – Can see reports in the Reporting Tab – In Perfecto Cloud 3 - Legacy Test Report - Can be Exported out and saved in different formats (example html, pdf). Can also be coded for in script, so maybe you are using Selenium/Appium for your testing and want to save script as a pdf format. 4 - Single Test Report – Link in Legacy Report or can be coded for in script (URL) 5 - Grid View/Dashboard – New Reporting Area – can see tests in Dashboard format, filter and drill down Just a quick note, Perfecto doesn’t limit you to just these reports/dashboards. You could also use an Open Source reporting framework like Allure or a dashboard/business intelligence tool like Splunk.
- Wind Tunnel provides an interactive test report enabling real time analysis. Using a persona (from Perfecto’s wind tunnel) in your test will create a Single Test Report. This report is opened in a browser. Alternatively, you can open the Report from the link that appears on the report page in the Reporting tab of the CQ Lab interface. Understanding the results: When reading the Single Test Report, first take a look at the points of interest reported from the test and identify any failures (commands and timers). Take a look at the previous command, the app & OS vitals (graph), the device screenshots and video. (When selecting a specific point in the Single Test Report timeline, the device video is synced to that time.) Additional Notes if needed: Download the Perfecto report from your test, using the downloadReport method, and click on the link to open the Single Test report. You can also retrieve the report URL via: String reportURL = (String) (driver.getCapabilities().getCapability(WindTunnelUtils.WIND_TUNNEL_REPORT_URL_CAPABILITY));
- Frequency and volume of testing is going up. The result is a bottleneck being created in the form of test results that need to be reported on. Are you able to quickly view quality across digital platforms? At the test suite level? At the test step where failure occurred? DigitalZoom is our reporting solution. It is architected for a top to bottom view on digital quality. From a high level view across all digital platforms, to a platform specific view, to the specific test step and device where failure occurred. DigitalZoom provides: Full traceability Full digital platform coverage
- Let’s review a sample test. Find a test in Eclipse with Persona setup Use Eclipse Project Named LenovoPOCMaster – Run on IpHONE AppiumTest.java – run as java (right click, select run config, under Java Application, highlight Appium Test (5), click run Credentials in code: To run the sample remember to add your credentials and Lab URL String host = “myLab.perfectomobile.com"; capabilities.setCapability("user", “myUser"); capabilities.setCapability("password", “MyPassword"); goes to Lenovo.com - warranty process Also opens wind tunnel apps in background (such as B of A and Twitter), set location –NY and 4GLTE Poor signal (Georgia)
- Let’s take a few minutes to see if we have any questions, and answer them. Thanks everyone for attending today…Please let us know if we can help with any additional information on these topics. Have a good day, evening or night wherever you may be.