Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (14)
Similaire à Introduction to testing with MSTest, Visual Studio, and Team Foundation Server 2010
Similaire à Introduction to testing with MSTest, Visual Studio, and Team Foundation Server 2010 (20)
Dernier
Dernier (20)
Introduction to testing with MSTest, Visual Studio, and Team Foundation Server 2010
- 1. Getting to know MSTest With Visual Studio 2010 and Team Foundation Server 2010 Thomas Weller www.thomas-weller.de info@thomas-weller.de July 2010
- 2. Outline Authoringtests (withMSTest) Visual Studio 2010 Team Foundation Server 2010 Advancedtestingconcepts UI Automation
- 4. What makes a good (unit) test...
- 5. Test context
- 7. Attributes:
- 10. [TestProperty]
- 13. Containing class: [TestClass]
- 15. Throws a special exception to fail a test
- 17. Arrange
- 18. Act
- 20. try to isolate code under test
- 21. short – simple – fast (CI) – readable
- 22. Test documentation, usage example
- 23. nodependencies on other tests (random order!)
- 32. Test lists/metadata (*.vsmdi)
- 33. Tool Windows:
- 34. Test Viewwindow
- 35. Test List Editor window
- 37. Test Runs window
- 44. Creating a Bug work item
- 45. Creating the test
- 46. Creating a Test Case work item for the Bug
- 47. Associating test and work item
- 48. Check in
- 51. Writing the test
- 57. Mocking
- 58. „Moling“
- 59. Data-driventests: xml
- 63. in AssemblyInfo.cs:
- 65. Isolating the tested code from dependencies
- 66. Specify arguments and return values
- 67. Verify method calls
- 71. MS Moles: Replace anything with a delegate
- 78. The recorder
- 79. UI Spy
- 80. UI object model
- 81. Example (Concurrency)
- 83. .NET based, OS, by ThoughtWorks (CC.Net)
- 85. Example - Concurrency (I)
- 86. Example - Concurrency (II)
- 87. … ?
Notes de l'éditeur
- Gives an introduction to writing tests (along with some best practices) with the MSTest frameworkin the context of Visual Studio 2010and Team Foundation Server 2010and presents some advanced concepts and techniques that go beyond those very simple (but mostly - practically - useless) introductory examples.
- This presentation has the following parts:Gives a first overview of a test method in MSTest other Testing frameworks (e.g. Nunit, MbUnit, xUnit.Net) are similar, mostly only syntactical differencesDemonstrates how testing is a part of Visual Studio (2010)the various tool windows that VS providesand how to work with them.Shows a sample workflow in the broader context of the Team Foundation server environmentPresents some more ‚advanced‘ concepts of Unit Testing(but you can‘t do without them when it comes to real-life scenarios)Shows some additional toolsLast not least: automated UI TestingMost difficult and complicated part of TestingPresents a strategy for WPF along with the required tools
- What does a ‚test class‘ look like, how can I write such a thing?Some important rules to keep in mind when authoring tests.The TestContext class in the MSTest frameworkWhat is it?Why is it important?What do I need it for?The various Initialize/Cleanup attribute pairsWhat is it?Why is it important?What do I need it for?Some useful attributes
- The sample class that I use as a test target for my simple examples, along with a corresponding interface.Very simple, self-explaining (hopefully...).
- A test class is nothing but an ordinary C# class.A test method is nothing but an ordinary C# method. (Signature is public void,no arguments).Marked as test class/method only by attributes.
- A test of course tests something.. i.e. it (mostly) asserts some value (variable, field, property...)Assert class (static) – such a class is common to all testing frameworksMS ‚additionally‘ provides some specialized Assert classes for strings etc.Assert classes throw exceptions (keep in mind when working with try/catch inside your tests)
- A test should always have the so-called triple-A structure:Arrange: Prepare everything for your test (e.g. initialize values...). This is usually the most complicated part of a test.Act: Execute the action that you want to test.Assert: Check that the outcome actually is as you would expect.
- Some guidelines for writing tests (‚best practices‘):One test asserts only one thingeven if this may be trivialMostly (but not always) this means: One AssertA test tends to be useless if the tested class/method depends on other classes/methods or external resourcesSuch a test has multiple reasons to fail, so what does the result ‚Failed‘ mean in such a case?The outcome of the test may depend upon external systems (e.g. database/network)A ‚unit test‘ is by definition an isolated test (otherwise it‘s a ‚(integration) test‘, but not a ‚unit test‘...)Remember that all of our tests are run in our Continuous Integration build system on each check-inexcept you explicitely excluded them for some reasonTests have to be fast (you may end up with hundreds or even thousands of them)A good test is the best documentation that a programmer can ever wish forIntention-revealing usage exampleTest runners (not only MSTest) by default run the tests in random orderNo dependencies, no ordering (you could do that with MSTest, but it‘s a bad smell)Think of a test as a complete execution unit
- The TestContext class is a singleton, specific to the current test runprovided by the MsTest framework, accessible in all test classesthink of it as your access point to the runtime environmentProvides methods to output custom text messagesProvides access to external resources (e.g. files, db connections...)Slide shows a sample where a text message is written, along with its appearance in the Test Result Details window.
- Property is automatically set by framework during initialization.
- There are pairs of Initialize/Cleanup methods on three levelsUse them (as the name suggests) to perform common initializations and cleanupagain: ordinary methods, identified by attributesall of the methods are optional, so pick the ones you need
- Attribute is used to assert that an exception of a specific type was thrown.You can also check for specific text in the exception message, should that be necessary (not shown here).
- Used to attach some custom explanatory text to the test method.Shows up in the Properties tool window.
- Attach any name/value pairs to a test method.Shows up in the Properties tool window (additional rows).
- Attach any number of categories to a test method (string).Useful for easy filtering in Test lists and CI builds.
- Useful usage example: Define test categories to control when a test should run.
- This part of the presentation shows you how the MSTest testing framework is integrated in Visual Studio.First, there are two configuration files, you need to know about (both are xml-format):The test run configuration (machine-specific, you can have as many as required)Test lists (useful for various purposes)Second, I will show you the various tool windows that VS provides for working with tests, and how they can be used.Side note:Visual Studio also has a feature to automatically generate a test class along with all test methods for a given class. I won‘t show this here for two reasons:It‘s not in the spirit of Test-driven development, because it already assumes some code to be tested. It‘s something that comes from a ‚test-after‘ philosophy. Therefore you better don‘t get used to it and the workflow it implies.The generated code is quite sloppy, with lots of unnecessary blank lines and useless comments. As a result, it takes more time to clean up and reorganize the generated code than it takes to write the few necessary lines from scratch.This is not to say that it‘s completely useless. Rather it depends on the specific scenario – as always. Just check it out, it‘s self-explaining...
- Test run configurations are used to adapt the test run to a specific machine. The build machine may need settings that are slightly different from your development machine.Notes:Beware that there are also settings that should be the same in all test settings (e.g. Timeout values)Be sure to have the ‚Enable deployment‘ checkbox checked, if you work with any files.Ideally you will only ever need one settings file.
- Test lists allow to create custom lists of test methods, that can be run with one click from Visual Studiop. They can be useful in various ways, e.g.:When developing a feature, you can collect all the relevant tests in a list to always have them easily available.You can define different test lists for different purposes/scenarios (you can use it for achieving the same as with the TestCategory attribute).
- Open via Test Results window...
- This part presents a simple but typical example workflow that shows how all the previously shown integrates with the new TFS environment.For a bug, the steps typically are as follows:You create a bug work item in TFS.Then you create a test that reproduces this bug.Next, you create a Test Case work item for your test and link it to your test method.Last not least: On Check-In, you link your changes to the test case.There will also be a short sidestep to show the various builds on the Build server and how they are intended.
- Preliminary remark:Everything in the TFS issue tracking system is a so-called Work Item. There are different types like e.g. Bugs, requirements etc. The list of available WIs depends on the Project management strategy and settings.You can create a Bug (or any other WI) either from the web-based Project Portal or from VS‘ Team Explorer.
- Add a meaningful title and the necessary details.
- Write the test that reproduces this error.(The Bug will be considered as Fixed, when this test finally will succeed...)
- From the details window of the Bug WI, Test Cases tab: Create a new Test Case WI.(The Test Case will be associated with the Bug, as the visualization shows)
- Now link the test method to the Test Case WI. You can do this from both ends:Either by browsing all tests from the Associated Automation tab of the Test Case details window.Or by right clicking on the test method in the Test View tool window and then browsing for the Test Case WI.
- On Check-In, associate your code with the relating Test Case WI.This association will show up in the version control‘s history.These connections greatly enhance traceability within the development system.
- Our build system has three different builds defined:The Continuous Integration build is triggered by each check-in (it accumulates consecutive check-ins if necessary). Therefore, it should be reasonably fast and exclude all tests that take a very long time, don‘t need to run that frequently, or are fragile for some reason. This build compiles the project in Debug Mode.Additionally, there‘s a nightly build (As the name implies, it runs each night. So you always have a fresh build when you come to work in the morning). This gives room for longer running tests as well as for long-running tasks like e.g. compilation and publishing of xml documentation (Sandcastle), deep statical analysis (FxCop, NDepend), creation of deployment packages and the like. This build compiles the project both in Debug and Release Mode.Furthermore there is a manual build (guess what: only manually triggered) which is used to finally build the solution for rollout. This build compiles the project in Release Mode and gets its sources from the read-only ‚REL‘ version-control branch (rather than from the ‚DEV‘ branch, which is our main line).The TestCategory attribute is used to define when a test shall be included/excluded. See usage example for this attribute (or the UI test method at the end).Note to readers:This reflects the configuration for the then current company environment / development team. Although this configuration reflects recommendable best practices, it doesn‘t necessarily has to be this way.
- Now back to some concepts that are more closely related to writing tests. While the here bespoken may ce called ‚advanced concepts‘, they are actually quite essential to your daily testing work, and you will see that in most cases you can‘t do anything useful without them.First we will look at the means that MSTest provides to access private code.Then I will show you an alternative way to do the same – it is preferable in most situations.Next I‘ll introduce you to the concept of Isolation/Mocking and present you a tool that can be used for that. Hereupon the MS Moles framework is presented with an example. It‘s said to be a mocking framework as well, but it actually is a different technical breed and you use it for slightly different things.I will end this part with the concept of ‚data-driven testing‘ (where test methods are provided with test data from outside). A data-driven test is executed once for each set of arguments (‚data row‘) that is provided by the data source. I‘ll show you examples for the two most widely used variations:The test data are provided via an xml file.The test data are provided via a database.
- For accessing private members, MSTest provides the PrivateObject helper class.As you can see from the list, it‘s basically a convenience wrapper that hides all the raw reflection code that you‘d have to write otherwise.For static elements, there‘s the PrivateType class (analogue to PrivateObject).
- Example for PrivateObject.General remark:Some consider ‚testing private members‘ a bad practice. I would not go that far - the important thing is the logic under test, not its access modifier (this is especially true for TDD).
- PrivateType/PrivateObject use reflection -> slow.If possible (new code or easy/safe refactoring possible), use this strategy instead:change method under test to internalselectively grant access to the test assembly from your tested assemblyrecommended location: AssemblyInfo.csif the tested assembly is strongly signed, the test assembly has to be also and you need to provide the key
- Writing a meaningful test normally requires that you isolate the code under test from the rest of the application – if you don‘t do that, then your test may effectively ‚test‘ the entire application (or at least a big portion of it). In such a case, the test has noumerous reasons why it can fail. What exactly is the test really testing then? (This is one of the most common reasons for a test to be almost useless...)This is where the concept of ‚Mocking‘ comes into play.Things you can do with mocking:Create test dummies that can replace the dependencies to other classes which the tested code normally has.Specify return values from your test dummy – specific to certain argument values, if needed.Verify that a certain method was called – again you can check for the method arguments, if you need to.The framework which we will use here is Moq (pron. ‚Mock‘ or ‚MockYou‘)Open SourceVery simple, but not too simple - gives you everything you ever needVery easy to learnMakes heavy use of lambda-syntaxGood documentationThere are many other OS mocking frameworks (even a bit too much for my taste), and the number is still growing...
- Example for mocking with Moq: The code to test.
- Example for mocking with Moq: The test method (in this case verifies a method call).
- The traditional OS mocking frameworks generate transparent runtime proxies which override interface implementations and virtual methods with (initially) empty method stubs. While you should use it where possible (it‘s fast), there are some serious drawbacks (Especially when writing tests for an already existing codebase, where you have no influence on the architectural design):You need to follow the ‚Dependency Injection‘ design principle, otherwise you‘re stuck.You need interfaces and/or virtual methods everywhere.What about static classes (e.g. Helpers, Singletons)?What about .NET framework stuff?All this can‘t be done with traditional mocking. Here we need another framework to the rescue: MS Moles.While at first sight it serves the same purpose as Moq (namely Isolation), it does this with a totally different technology: First you need to create a special assembly (derived from the original assembly under test), which contains stubs (or hooks in C++-speak) for all the original classes members. These stubs then during runtime dynamically replace the class‘original entry points. This is possible because the test is run by a specialized test runner.All this is quite handy, useful and often required, but it comes with significant performance costs...
- Create the Moles assembly from your project references...
- The name of the Moles assembly is derived from the original name (‚.Moles‘ is added).There is also an xml file and a .Designer file.Add the dll and the xml file to your project, so that they are checked in. Forget about the .Designer file.Declare the type that you want to have stubs for in your test class‘ file (or in AssemblyInfo.cs, if appropriate) with the MoledType assembly-level attribute.Note: Step 1 is not necessary anymore in newer versions of MS Moles.
- Tests which use Moles have to declare a special test runner.Moled method name original method name + argument type name(s)
- Data-driven tests with xml file:Create an xml of the form Rows/Row/Argument name(s).
- You can then access this data via the TestContext object.Note that you have to declare the data file and it‘s location via additional test method attributes.Oh, and don‘t forget toadd the data file to the project and check it inenable deployment in the *.testsettings file
- You can inspect the results in the Test Result Details window...
- Data-driven tests via database are very similar. First create a table for the test data (and remember the column names).
- Then define the connection string to access the data in one of two ways:Either in the Test Properties tool window where you are assisted by the well-known Visual Studio dialogue for defining a connection stringOr directly in code via the DataSource attribute
- Finally, let me give you a short overview on how we can test against the UI. UI testing definitely is the most difficult and demanding hill on the testing landscape: It requires quite some resources on the build server (usually such tests should run in a separate virtual machine).It usually requires the programmer to write an additional object model on top of the code-to-test.UI tests are difficult to automate (sometimes it‘s even impossible).(This is not to say that UI tests are a bad thing. But it‘s definitely not the kind of stuff that you want to begin your testing career with.)In this section:First I‘ll present you another Open Source framework, namely White.Together with the included recorder application, which is an important tool in daily work.The other tool that you need (for inspecting an application‘s UI): UI Spy from MicrosoftLast I‘ll go quickly over a small part of the object model that lies underneath all the UI tests and finally I will show you a test method that uses it.(Afterwards you most likely will be ready for a cup of coffee...)Note to readers: The presentation originally was targeting a WPF desktop application.
- The most important facts about the White framework...
- The recorder is used to generate C# code from the user‘s UI actions...
- UI Spy is very similar to Spy++, but for .NET/WPF.Used to inspect a UI‘s control structure/properties.Free download, included with the Windows SDK 6.0 (mind the correct version).
- This is a small portion of the object model that was written for the UI tests.It is used to have a high-level facade on top of the application‘s UI-relating code.
- Test code, pt. 1...
- Test code, pt. 2:An actual test method.You can immediately see that things aren‘t that simple here anymore and that there lies a quite huge helper class library underneath this test.
- Questions?Concerns?Or only a desperate need for coffee?