Evaluating and Testing Web APIs
•
7 j'aime•9,205 vues
This document discusses evaluating and testing web APIs. It covers several topics: - The importance of API quality for customers who rely on APIs for their applications and business needs. High-quality APIs must be available, consistent, transparent, secure, performant, reliable and usable. - Different approaches to testing APIs, including functional automation testing at different stages of the development process from continuous integration to production monitoring. - Key aspects of API quality that should be tested, such as expected results and behaviors, error handling, internationalization, security, performance, and usability. - The presentation aims to convey that thorough evaluation and testing of APIs is necessary to ensure they meet customers' needs and expectations.
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (20)
Similaire à Evaluating and Testing Web APIs
Similaire à Evaluating and Testing Web APIs (20)
Plus de SmartBear
Plus de SmartBear (20)
Dernier
Dernier (20)
Evaluating and Testing Web APIs
- 1. EVALUATING AND TESTING WEB APIS Ole Lensmar – SmartBear Software
- 2. the goal of this presentation…
- 3. Web APIs – what’s the fuss?
- 4. Aysnc / WebSockets REST / JSON SOAP / XML Technically speaking… but that’s missing the point…
- 5. 20 Years Ago ACME Corp (does monolithics)
- 6. 10 Years Ago ACME Corp (does SOA) Web app Corp
- 7. Corp Corp Now! ACME Corp (does APIs) Web app Corp App + API API API API Devi ce APP API Devi ce Devi ce Devi ce API API
- 8. API Quality – Who needs it?
- 9. API Hierarchy of Needs www.apiux.com
- 11. Who is your target persona?
- 13. Align with their API technology SOAP / REST / etc… XML / JSON / etc… QoS / Security
- 14. Help them understand your API vs
- 15. Provide API Metadata Validation Code Generation Coverage Understanding Simulation wsdl, swagger, wadl, hal, json schema, apiary.io, xml schema, ws-*, apiary, api-docs, iodocs, etc
- 16. Align your API with their domain Process / Workflow Nomenclature Related APIs
- 17. A 3:30:3 Litmus test for APIs 3 Minutes to understand what an API does 30 seconds to sign up 3 minutes to the first request (Ori Pekelman)
- 18. APIs need to work
- 19. Expected Results
- 25. Bad Error Messages… Change unexpectedly Don’t match their status code (HTTP) Don’t tell users what they did wrong Don’t tell users what they need to do right Show clients stuff that could be misused
- 26. APIs are global from day 1 Dates & Timezones Regional Formatting Localized Messages / Errors
- 28. API Functional Test Automation TestCase API 1 API 2
- 29. TestCase API 1 API 2 Request 1 Response validate API Functional Test Automation
- 30. TestCase API 1 API 2 Request 1 Response Transfer parameters Request 2 Response validate validate API Functional Test Automation
- 31. TestCase API 1 API 2 Request 1 Response Request 3 Response Transfer parameters Request 2 Response Transfer parameters validate validate validate API Functional Test Automation
- 32. Pre vs Post-Deployment Quality Continuous Integration (Agile) Automated Test Execution
- 33. Pre vs Post-Deployment Quality Continuous Integration (Agile) Continuous Deployment (DevOps) Automated Test Execution months/weeks/d ays/hours
- 34. Pre vs Post-Deployment Quality Continuous Integration (Agile) Continuous Deployment (DevOps) Automated Test Execution Production Monitoring months/weeks/d ays/hours Test asset re-use
- 36. Customers rely on your API
- 37. Your API is key to their success Is your API available? Is your API consistent? Is your API transparent?
- 38. Is your API Secure? Do you handle identity? Do you maintain integrity? Do you assess vulnerabilities?
- 39. API Performance Does the API perform consistently? Does the API recover? Does the API scale?
- 40. Usability Functionality Core elements of API Quality Reliability
- 41. so – let’s get back to that goal…
Notes de l'éditeur
- By the end of this session you will have turned into juvenileexperts on the key components of API Quality. In your organization, whenever you hear about API initiatives and how you should test them– you will be able to raise your hand and say “I know all about that – a chap at StarWEST with a funny accent – he spelled it all out for us” – and you will be able to rise and help your company or team produce the finest APIs yet to be seen. If not – I have failed – and you can let me know on the evaluation talon. Or you can mailme to tell of your endeavors – I want to know!But - to get us there in mere 40 minutes I will be in a constant hurry – nevertheless - please don’t hesitate to interrupt if you have any questions – after all – my fate as an API aficionado lies in your hands.
- First – do we all know what I mean by Web APIs? Do we have a common understanding? Because many of you might be thinking; what’s the big deal? An API is an interface for a system – testing them is no different than testing other code; we test that they “work”, we report bugs, we hug developers because they involve you early in the process, we make sure performance is ok - and we put them into production – how hard can it be? Well –that’s not the whole story – but to make my case I need to give you some context on the evolution of Web APIs – where they come from.
- One way to look at Web APIs is from a technology point of view – SOAP/XML -> REST/JSON -> Asycn (WebSockets, etc)And trust me when I say that I could talk a lot about these technologies - But that’s not really it – that’s not where the real action is when it comes to the fuss around APIs – to understand what the noise is all about we need to take a step back
- 20 years ago (monolithic) -> 10 years ago (SOA) -> now (WOA)
- 20 years ago (monolithic) -> 10 years ago (SOA) -> now (WOA)
- Suddenly – APIs are at the heart of a companies business – having evolved from being strictly internal where they used to be building block with a controlled environment, usage, dependencies and lifecycle are totally within the owning business – they now are the business, and as such have a totally different context; their usage is unpredictable – their lifecycle is controlled by others – they are subject to competition from other API providers – and their users are biased from day one on how an API should be. That’s a totally different challenge for QA than an API that is strictly internal – in a controlled environment. Of course – the functional and performance aspects still apply – but suddenly a bunch of other traits related to QoS and Usability are of primary concern for the API (and thus your business) to succeed.
- So, let’s try to drill into this a little bit… who are the actual stakeholders for API quality? Who really needs it – and what is important to them?
- Bruno Pedro at apiux.com created this API Hierarchy of Needs – trying to visualize the needs of an API for it to be successful. What’s really interesting to note here – and something the spurred some debate – is that usability is at the bottom of the pyramid. He argues that it doesn’t matter how well an API works or how reliable it is – if you can’t use it, it doesn’t matter.
- Keeping the API Hierarchy of needs in mind, lets start with the insight that although APIs are ultimately invoked by code – they are initially always consumed by people - Developers, Testers, the like! And for you to be able to do that in an efficient and satisfying way – the API has to be usable – which brings us to the perhaps most important prerequisite for a successful API : it’s usability – how easy is it to use the API?
- Know your users – is it Josh, the HTML5 Hscker? Is it John the j2EE Expert?
- You might have heard of the term UX before – User Experience, and similarly the term DX (Developer Experience) designates how easy it is for a Developer to consume and API – and this is where API Usability experts have to focus; on the developers that will initially be tasked with integrating the API. We all know that developers are a special bunch – and as such have special needs – lets explore some of them.
- Know your
- So let’s say we are going to use an API – how do we get started?
- Metadata is a somewhat heated subject in some API communities; with SOAP there has never been an issue – but with REST (which was a countermovement to SOAP in a certain sense) – there has been a lively debate. The main argument against metadata has been that it is hard to maintain – which isn’t always true. The advantages for metadata are on the other hand several:Code generationAutomated validationDetailed understanding of complex messagesFor testers: possibility to measure coverageWhat can metadata give that documentation doesn’t? Isn’t metadata a kind of documentation?
- The APIs you create are most often consumed in some kind of context – and if that context is well established it usually already follows certain rules/heuristics applicable to the task at hand. For example, let’s say your API is related to geospatial information – and that your users often also use the Google Maps APIS
- This is a test that attempts to measure API Usability: the 3:30:3 Test. Devised by OriPekelman – it might not be the most scientifically correct – but it does give an indication of what is important. When visting a companies API developer portal, it should take: 3 minutes to understand what the API does, 30 seconds to sign up, and 3 minutes to be able to make your first call to the API.
- Finally – and this is perhaps the one many layman testers think of first (not you of course) – the direct stakeholder of the API is the code that integrates with it – because ultimately, in production, APIs are invoked by code – not people. This is where functional testing, test automation, automated test execution all come into play. Let’s look at some hands on examples.
- Here you test for expected and unexpected results. Test not only the content of the results – but also the format. Going back to the discussion on usability and nomenclature – make sure that if you use a common nomenclature (for example for coordinates) use the right data-types as well. Make sure you do both positive and negative tests – error messages are equally important.Formatting, DatatypesValuesConsistency
- Here you test for expected and unexpected results. Test not only the content of the results – but also the format. Going back to the discussion on usability and nomenclature – make sure that if you use a common nomenclature (for example for coordinates) use the right data-types as well. Make sure you do both positive and negative tests – error messages are equally important.Formatting, DatatypesValuesConsistency
- Here you test for expected and unexpected results. Test not only the content of the results – but also the format. Going back to the discussion on usability and nomenclature – make sure that if you use a common nomenclature (for example for coordinates) use the right data-types as well. Make sure you do both positive and negative tests – error messages are equally important.Formatting, DatatypesValuesConsistency
- Here you test for expected and unexpected results. Test not only the content of the results – but also the format. Going back to the discussion on usability and nomenclature – make sure that if you use a common nomenclature (for example for coordinates) use the right data-types as well. Make sure you do both positive and negative tests – error messages are equally important.Formatting, DatatypesValuesConsistency
- Here you test for expected and unexpected results. Test not only the content of the results – but also the format. Going back to the discussion on usability and nomenclature – make sure that if you use a common nomenclature (for example for coordinates) use the right data-types as well. Make sure you do both positive and negative tests – error messages are equally important.Formatting, DatatypesValuesConsistency
- Here you test for expected and unexpected results. Test not only the content of the results – but also the format. Going back to the discussion on usability and nomenclature – make sure that if you use a common nomenclature (for example for coordinates) use the right data-types as well. Make sure you do both positive and negative tests – error messages are equally important.Formatting, DatatypesValuesConsistency
- This error message for example – can anyone see what’s wrong with this?
- One thing that I haven’t really emphasized is that APIs are global from day 1 – their consumers could be calling in from all over the world – and if your API makes use of dates, times and timezones that is a common area for problems. Make sure that the system handles this input correctly.
- You chain multiple requests together to test entire transactions – transferring content in a response to the following requests.
- You chain multiple requests together to test entire transactions – transferring content in a response to the following requests.
- You chain multiple requests together to test entire transactions – transferring content in a response to the following requests.
- You chain multiple requests together to test entire transactions – transferring content in a response to the following requests.
- If you’re in an agile shop doing continuous integration you might want to set up corresponding continuous testing processes – and if you are doing automated deployment – you should perhaps consider to reuse some of those pre-deployment testing assets as post-deployment functional monitors.Exploratory Testing!
- If you’re in an agile shop doing continuous integration you might want to set up corresponding continuous testing processes – and if you are doing automated deployment – you should perhaps consider to reuse some of those pre-deployment testing assets as post-deployment functional monitors.Exploratory Testing!
- If you’re in an agile shop doing continuous integration you might want to set up corresponding continuous testing processes – and if you are doing automated deployment – you should perhaps consider to reuse some of those pre-deployment testing assets as post-deployment functional monitors.Exploratory Testing!
- Finally - consider this: since and API is consumed by a machine there is no “wiggle-room” as there is for the previous two stakeholders – which could be expected to adapt to lack of quality in certain areas. Here it’s straight forward: either it works, and delivers the desired functionality – or it doesn’t and the customer gets strange error messages.
- The next Stakeholder is the business that is using your API. Think about it: their business (and perhaps entire existence!) depends on it – if your API doesn’t work as they need it to – both their reputation (and yours!) is at great risk! For them a different number of aspects come into play – all wrapped up in the somewhat boring term “Quality of Service”.Now – QoS isn’t new to Web APIs – it is equally important for internal APIs - but the big difference is that for Web APIs you can’t really foresee how and in which context your API will be consumed – which makes it even more important that you are one step ahead in regard to your APIs quality. I’ll give you three main areas:
- Reliability – can they rely on your service being available? Will you provide consistent response times? For a mobile app – do you give consistent results from all over the world? -> You can highlight these and make sure that Monitoring – from all over the world.
- Security – this is a biggie – we’ve all heard about security breaches at facebook, twitter, the Playstation Network. You customers want to know that any data they enter into your system that belongs to them – can never – and I mean never – come into wrong hands. -> Security Testing
- Performance / Scalability – this is the really hard part: what if your API is a building block in the latest startup craze? What if load increases from 100s request per minute to 1000 requests/sec over the timespan of a couple of weeks – or days? Are you prepared? Should you be? Do you want to miss out on that opportunity? If you can scale successfully with your customers – they will be loyal and love you forever. Incidentally – you can see your own business as an equally invested stakeholder – the same requirements apply. And added dimension such as competitors enter the discussion -> is your API better than the competition in these regards?
- OK – you’ve all been very brave and patient - lets recap this a bit: API Quality has three primary stakeholder and corresponding areas of concern:Users: User Experience -> required by the people initially consuming the APIBusiness: Quality of Service -> required by the business depending on the APICode: Consistent Functionality -> required by the code calling the API And although perhaps some of this is outside the scope of your role at your company – you can be that champion – you can be the one that brings these things to the attention of those building the API: I’m sure you remember that I made a promise - let’s revisit that and look at how I hope things to play out when you get back to your offices: