Getting started with unit testing is not hard, the only problem is that most programs are more than a simple calculator with two parameters and a returns value that is easy to verify.
Writing unit tests for multi-threaded code is harder still.
Over the years I discovered useful patterns that helped me to test multi-threaded and asynchronous code and enabled the creation of deterministic, simple and robust unit tests.
Come learn how to test code that uses concurrency and parallelism – so that the excuses of not writing unit tests for such code would become as obsolete as a single core processors.
How to Troubleshoot Apps for the Modern Connected Worker
Unit testing patterns for concurrent code
1. Unit testing patterns for concurrent code
Dror Helper
drorh@oz-code.com | @dhelper | http://blog.drorhelper.com
Examples: https://github.com/dhelper/ConcurrentUnitTesting
2. About.ME
• Senior consultant @CodeValue
• Developing software (professionally) since 2002
• Mocking code since 2008
• Test Driven Developer
• Blogger: http://blog.drorhelper.com
3. We live in a concurrent world!
The free lunch is over!
• Multi-core CPUs are the new standard
• New(er) language constructs
• New(ish) languages
4. Meanwhile in the unit testing “world”
[Test]
public void AddTest()
{
var cut = new Calculator();
var result = cut.Add(2, 3);
Assert.AreEqual(5, result);
}
5. The dark art of concurrent code
Several actions at the same time
Hard to follow code path
Non deterministic execution
6. Good unit tests must be:
• Trustworthy
– Consistent results
– Only fail due to bug or requirement change
• Maintainable
– Robust
– Easy to refactor
– Simple to update
• Readable
7. Concurrency test “smells”
× Inconsistent results
× Untraceable fail
× Long running tests
× Test freeze
8. How can we test this method
public void Start()
{
_worker = new Thread(() => {
while (_isAlive) {
Thread.Sleep(1000);
var msg = _messageProvider.GetNextMessage();
//Do stuff
LastMessage = msg;
}
});
_worker.Start();
}
9. Testing start take #1
[TestMethod]
public void ArrivingMessagePublishedTest()
{
var fakeMessageProvider = A.Fake<IMessageProvider>();
A.CallTo(() => fakeMessageProvider.GetNextMessage()).Returns("Hello!");
var server = new Server(fakeMessageProvider);
server.Start();
Thread.Sleep(2000);
Assert.AreEqual("Hello!", server.LastMessage);
}
10. Test smell - “Sleep” in test
× Time based - fail/pass inconsistently
× Test runs for too long
× Hard to investigate failures
11. “In concurrent programming if
something can happen, then sooner
or later it will, probably at the most
inconvenient moment”
Paul Butcher – Seven concurrency models in seven weeks
12. Testing start take #2
[TestMethod]
public async Task ArrivingMessagePublishedTest()
{
var fakeMessageProvider = A.Fake<IMessageProvider>();
A.CallTo(() => fakeMessageProvider.GetNextMessage()).Returns("Hello!");
var server = new Server(fakeMessageProvider);
server.Start();
await Task.Delay(2000);
Assert.AreEqual("Hello!", server.LastMessage);
}
14. Pattern #1 – humble object pattern
We extract all the logic from the hard-to-test component into a component
that is testable via synchronous tests.
http://xunitpatterns.com/Humble%20Object.html
Perform Action
Code under Test
Start
Humble object
Async
Perform action
Assert Result
Production
Code
15. public void Start()
{
_worker = new Thread(() => {
while (_isAlive) {
Thread.Sleep(1000);
var msg = _messageProvider.GetNextMessage();
//Do stuff
LastMessage = msg;
}
});
_worker.Start();
}
16. public void Start()
{
_worker = new Thread(() => {
while (_isAlive) {
Thread.Sleep(1000);
_messageHandler.HandleNextMessage();
}
});
_worker.Start();
}
17. And finally – the test
[TestMethod]
public void ArrivingMessagePublishedTest()
{
var fakeMessageProvider = A.Fake<IMessageProvider>();
A.CallTo(() => fakeMessageProvider.GetNextMessage()).Returns("Hello!");
var messageHandler = new MessageHandler(fakeMessageProvider);
messageHandler.HandleNextMessage();
Assert.AreEqual("Hello!", messageHandler.LastMessage);
}
18. Concurrency as part of program flow
public class MessageManager
{
private IMesseageQueue _messeageQueue;
public void CreateMessage(string message)
{
// Here Be Code!
_messeageQueue.Enqueue(message);
}
}
public class MessageClient
{
private IMesseageQueue _messeageQueue;
public string LastMessage { get; set; }
private void OnMessage(object sender, EventArgs e)
{
// Here Be Code!
LastMessage = e.Message;
}
}
19. Test before – Test After
Start
Code under test
Async
Logic2
Assert results
Logic1
Start Logic1 Fake Assert Results
Start Fake Logic2 Assert Results
20. Testing Flow part 1
[TestMethod]
public void AddNewMessageProcessedMessageInQueue()
{
var messeageQueue = new AsyncMesseageQueue();
var manager = new MessageManager(messeageQueue);
manager.CreateNewMessage("a new message");
Assert.AreEqual(1, messeageQueue.Count);
}
21. Testing Flow part 2
[TestMethod]
public void QueueRaisedNewMessageEventClientProcessEvent()
{
var messeageQueue = new AsyncMesseageQueue();
var client = new MessageClient(messeageQueue);
client.OnMessage(null, new MessageEventArgs("A new message"));
Assert.AreEqual("A new message", client.LastMessage);
}
22. Avoid concurrency Patterns
The best possible solution
No concurrency == no problems
× Do not test some of the code
× Not applicable in every scenario
23. How can we test this class?
public class ClassWithTimer
{
private Timer _timer;
public ClassWithTimer(Timer timer)
{
_timer = timer;
_timer.Elapsed += OnTimerElapsed;
_timer.Start();
}
private void OnTimerElapsed(object sender, ElapsedEventArgs e)
{
SomethingImportantHappened = true;
}
public bool SomethingImportantHappened { get; private set; }
}
24. Not a good idea
[TestMethod]
public void ThisIsABadTest()
{
var timer = new Timer(1);
var cut = new ClassWithTimer(timer);
Thread.Sleep(100);
Assert.IsTrue(cut.SomethingImportantHappened);
}
25. Set timeout/interval to 1
Also seen with a very small number (or zero)
Usually done when need to wait for next tick/timeout
× Time based == fragile/inconsistent test
× Hard to investigate failures
× Usually comes with Thread.Sleep
26. Fake & Sync
Test
Code under Test
Fake
Logic
Assert Results
27. Using Typemock Isolator to fake Timer
[TestMethod, Isolated]
public void ThisIsAGoodTest()
{
var fakeTimer = Isolate.Fake.Instance<Timer>();
var cut = new ClassWithTimer(fakeTimer);
var fakeEventArgs = Isolate.Fake.Instance<ElapsedEventArgs>();
Isolate.Invoke.Event(
() => fakeTimer.Elapsed += null, this, fakeEventArgs);
Assert.IsTrue(cut.SomethingImportantHappened);
}
28. Unfortunately not everything can be faked
• Mocking tool limitation (example: inheritance based)
• Programming language attributes
• Special cases (example: MSCorlib)
Solution – wrap the unfakeable
× Problem – it requires code change
29. Wrapping Threadpool
public class ThreadPoolWrapper
{
public static void QueueUserWorkItem(WaitCallback callback)
{
ThreadPool.QueueUserWorkItem(callback);
}
}
30. Wrapping Threadpool
public class ClassWithWrappedThreadpool
{
public void RunInThread()
{
ThreadPool.QueueUserWorkItem(_ =>
{
SomethingImportantHappened = true;
});
}
public bool SomethingImportantHappened { get; private set; }
}
ThreadPoolWrapper.QueueUserWorkItem(_ =>
31. Testing with new ThreadpoolWrapper
[TestClass]
public class WorkingWithThreadpool
{
[TestMethod, Isolated]
public void UsingWrapperTest()
{
Isolate.WhenCalled(() => ThreadPoolWrapper.QueueUserWorkItem(null))
.DoInstead(ctx => ((WaitCallback)ctx.Parameters[0]).Invoke(null));
var cut = new ClassWithWrappedThreadpool();
cut.RunInThread();
Assert.IsTrue(cut.SomethingImportantHappened);
}
}
32. How can we test that an
asynchronous operation
never happens?
33. Test Code Under Test
Is Test
Async
Deterministic Assert Results
Run in sync
34. Another day – another class to test
public void Start()
{
_cancellationTokenSource = new CancellationTokenSource();
Task.Run(() =>
{
var message = _messageBus.GetNextMessage();
if(message == null)
return;
// Do work
if (OnNewMessage != null)
{
OnNewMessage(this, EventArgs.Empty);
}
}, _cancellationTokenSource.Token);
}
35. Switching code to “test mode”
• Dependency injection
• Preprocessor directives
• Pass delegate
• Other
36. Run in single thread patterns
• Fake & Sync
• Async code sync test
38. The Signaled pattern
Start
Run
Code under test
Signal
Fake object
Call
Wait Assert result
39. Using the Signaled pattern
public void DiffcultCalcAsync(int a, int b)
{
Task.Run(() =>
{
Result = a + b;
_otherClass.DoSomething(Result);
});
}
40. Using the Signaled pattern
[TestMethod]
public void TestUsingSignal()
{
var waitHandle = new ManualResetEventSlim(false);
var fakeOtherClass = A.Fake<IOtherClass>();
A.CallTo(() => fakeOtherClass.DoSomething(A<int>._)).Invokes(waitHandle.Set);
var cut = new ClassWithAsyncOperation(fakeOtherClass);
cut.DiffcultCalcAsync(2, 3);
var wasCalled = waitHandle.Wait(10000);
Assert.IsTrue(wasCalled, "OtherClass.DoSomething was never called");
Assert.AreEqual(5, cut.Result);
}
42. Busy assertion
[TestMethod]
public void DifficultCalculationTest()
{
var cut = new ClassWithAsyncOperation();
cut.RunAsync(2, 3);
AssertHelper.BusyAssert(() => cut.Result == 5, 50, 100, "fail!");
}
43. Synchronized test patterns
× Harder to investigate failures
× Cannot test that a call was not made
Test runs for too long but only when it fails
Use if other patterns are not applicable
44. Unit testing for concurrency issues
Because concurrency (and async) needs to be tested as well
45. Test for async
Start Method under test
Fake object
wait
Assert results
46. Test for Deadlock
Start
Execute
Code under
test
Thread
WWaaitit Assert result
Call Thread
47. [TestMethod, Timeout(5000)]
public void CheckForDeadlock()
{
var fakeDependency1 = A.Fake<IDependency>();
var fakeDependency2 = A.Fake<IDependency>();
var waitHandle = new ManualResetEvent(false);
var cut = new ClassWithDeadlock(fakeDependency1, fakeDependency2);
A.CallTo(() => fakeDependency1.Call()).Invokes(() => waitHandle.WaitOne());
A.CallTo(() => fakeDependency2.Call()).Invokes(() => waitHandle.WaitOne());
var t1 = RunInOtherThread(() => cut.Call1Then2());
var t2 = RunInOtherThread(() => cut.Call2Then1());
waitHandle.Set();
t1.Join();
t2.Join();
}
T1
T2
L1
L2
48. Avoid
Concurrency
Humble
object
Test before –
test after
Run in single
thread
Fake & Sync
Async in
production -
sync in test
Synchronize
test
The Signaled
pattern
Busy
assertion
Concurrency
tests
Test for
Deadlock
Test for non
blocking
Concurrent unit testing patterns
49. Conclusion
It is possible to test concurrent code
Avoid concurrency
Run in single thread
Synchronize test