Deadlock Victim

The Java Specialists' Newsletter - javaspecialists.eu / zenika.fr

Deadlock Victim
by Dr Heinz Kabutz && Olivier Croisier
The Java Specialists Newsletter && The Coder's Breakfast
heinz@javaspecialists.eu && olivier.croisier@zenika.com
twitter: @heinzkabutz && twitter: @oliviercroisier

1


You discover a race condition

2


Data is being corrupted

3


What ya gonna do?

4


Who ya gonna call?

5


Ghostbusters?

6


you do what we all do

7


you use synchronized

8


synchronized

9


synchronized can be bad
medicine

10


deadlock awaits you!

11


12


Deadlock Victim

13


Correctness ﬁx
can lead to
liveness fault

14


lock ordering deadlock

15


at least two locks

16


and at least two threads

17


locked in different orders

18


can cause a deadly embrace

19


requires global analysis to ﬁx

20


resource deadlocks

21


bounded queues

22


bounded thread pools

23


semaphores

24


class loaders

25


lost signals

26


The Drinking Philosophers

27


@ the Java Specialist Symposium
in Crete

28


Symposium == sym + poto
Literally - to drink together

29


our philosophers need two cups

30


they either want to drink or think

31


Table is ready, all philosophers are
thinking
1

5 2

4 3
32


philosophers 5 wants to drink,
takes right cup
1

5 2

4 3
33


Philosopher 5 is now drinking with
both cups
1

5 2

4 3
34


takes right cup
1

5 2

4 3
35


both cups
1

5 2

4 3
36


takes right cup
1

5 2

4 3
37


Philosopher 3 ﬁnished drinking,
returns right cup
1

5 2

4 3
38


both cups
1

5 2

4 3
39


Philosopher 3 returns Left cup
1

5 2

4 3
40


philosophers can get stuck

41


if they all pick up the right cup

42


A deadlock can easily happen with
this design
1

5 2

4 3
43


Philosopher 5 wants to drink, takes
right cup
1

5 2

4 3
44


right cup
1

5 2

4 3
45


right cup
1

5 2

4 3
46


right cup
1

5 2

4 3
47


right cup
1

5 2

4 3
48


Deadlock!
1

5 2

4 3
49


they will all stay thirsty
1

5 2

4 3
50


this is a deadly embrace
1

5 2

4 3
51


this can be detected automatically

52


search the graph of call stacks

53


and look for circular dependencies

54


ThreadMXBean does that for us

55


simple deadlocks are quickly found

56


but what do you DO with them?

57


what will the dog do with the car if
he ever catches it?

58


databases choose deadlock victim

59


but what does Java do?

60


it should probably throw an Error

61


but the threads simply hang up

62


it is best to avoid causing them!

63


global order of locks in program

64


public class LeftRightDeadlock {
private final Object left = new Object();
private final Object right = new Object();
public void leftRight() {
synchronized (left) {
synchronized (right) {
doSomething();
}
}
}
public void rightLeft() {
doSomethingElse();
}
}
}
}

65


Interleaving causes deadlock
lock left trying to lock right

A leftRight() BLOCKED

lock right trying to lock left

B rightLeft() BLOCKED

66


we deﬁne a ﬁxed global order

67


e.g. left before right

68


deadlock is solved!
public void rightLeft() {
doSomethingElse();
}
}
}
69


our drinking philosophers

70


each cup gets a unique number

71


lock the highest number cup ﬁrst

72


then lock the lower number cup

73


drink

74


unlock lower order cup

75


unlock higher order cup

76


think

77


Global Lock ordering
• We start with all the 1
philosophers thinking
1 2
5 2

5 3

4

4 3
78


Philosopher 5 takes cup 5
• Cup 5 has higher number 1

• Remember our rule!
1 2
5 2
5
3

4

4 3
79


• Must take the cup with 1
the higher number 2
ﬁrst
1
• In this case 5 2
cup 2 5
3

4

4 3
80


1
2

1
5 2
5 3

4

4 3
81


1
2

1
5 2
5 3

4 4 3
82


Philosopher 1 takes cup 1 -
Drinking
1
1 2

5 2
5 3

4 4 3
83


Philosopher 1 returns Cup 1
• Cups are returned in the 1
opposite order to what 2
they are acquired
1
5 2
5 3

4 4 3
84


Drinking
1
2
1

5 2
5 3

4 4 3
85


Philosopher 5 returns cup 1
1
2

1
5 2
5 3

4 4 3
86


1

1 2
5 2
5 3

4 4 3
87


Drinking
1

2
1
5 2
5 3

4 4 3
88


1

2
1
5 2
3
5

4 4 3
89


1

2
1
5 2
3

5

4 4 3
90


1

1 2
5 2
3

5

4 4 3
91


1

1 2
5 2

3

5

4 4 3
92


Drinking
1

1 2
5 2

5 3

4 4 3
93


Philosopher 3 Returns cup 3
1

1 2
5 2

3

5

4 4 3
94


1

1 2
5 2

3

5
4

4 3
95


Drinking
1

1 2
5 2

3

5

4 4
3
96


1

1 2
5 2

3

5
4

4 3
97


1

1 2
5 2

5 3

4

4 3
98


sorting locks by property is easy

99


e.g. account IBAN number

100


or employee number

101


or some other characteristic

102


maybe even
System.identityHashCode()

103


use "open calls"

104


Vector synchronizes writeObject()
private synchronized void writeObject(ObjectOutputStream s)
throws IOException {
s.defaultWriteObject();
}

105


synchronized writeObject() is
invoking s.defaultWriteObject()
}

106


this is not an "open call"
}

107


but can it cause problems?
}

108


yes it can!
}

109


serialize both vectors at same time
Vector v1 = new Vector();
Vector v2 = new Vector();
v1.add(v2);
v2.add(v1);

110


this could happen with nested data
structures

111


it happened to an IBM client

112

private void writeObject(ObjectOutputStream stream)
Vector<E> cloned = null;
// this specially fix is for a special dead-lock in customer
// program: two vectors refer each other may meet dead-lock in
// synchronized serialization. Refer CMVC-103316.1
synchronized (this) {
try {
cloned = (Vector<E>) super.clone();
cloned.elementData = elementData.clone();
} catch (CloneNotSupportedException e) {
// no deep clone, ignore the exception
}
}
cloned.writeObjectImpl(stream);
}

private void writeObjectImpl(ObjectOutputStream stream)
stream.defaultWriteObject();
}
113


Java 7 uses an "open call"
private void writeObject(ObjectOutputStream s)
final ObjectOutputStream.PutField fields = s.putFields();
final Object[] data;
synchronized (this) {
fields.put("capacityIncrement", capacityIncrement);
fields.put("elementCount", elementCount);
data = elementData.clone();
}
fields.put("elementData", data);
s.writeFields();
}
114


More synchronized writeObject()s
• File

• StringBuffer

• Throwable

• URL

• Hashtable

• and others

115


we have seen the deadly embrace

116


are there other deadlocks?

117


semaphore deadlocks

118


Bounded DB Connection Pools

• For example, say you have two DB connection pools

• Some tasks might require connections to both databases

• Thus thread A might hold semaphore for D1 and wait for D2,
whereas thread B might hold semaphore for D2 and be waiting for
D1

119


public class DatabasePool {
private final Semaphore connections;
public DatabasePool(int connections) {
this.connections = new Semaphore(connections);
}

public void connect() {
connections.acquireUninterruptibly();
System.out.println("DatabasePool.connect");
}

public void disconnect() {
System.out.println("DatabasePool.disconnect");
connections.release();
}
}
120


ThreadMXBean does not show
this as a deadlock!

121


JVM does not detect this deadlock

122


Avoiding and diagnosing deadlocks
Avoiding Liveness Hazards

123



• You cannot get a lock-ordering deadlock with a single lock

• Easiest way to avoid deadlocks

• But not always practical

124



• For multiple locks we need to deﬁne a lock order

• specify and document possible lock sequences

• Identify where multiple locks could be acquired

• Do a global analysis to ensure that lock ordering is consistent

• This can be extremely difﬁcult in large programs

125


Use open calls whenever possible

126


Unit Testing for lock ordering
deadlocks

• Code typically has to be called many times before a deadlock happens

• How many times do you need to call it to prove that there is no
deadlock?

• Nondeterministic unit tests are bad - they should either always pass
or always fail

127


Testing the Bank
• In the transferMoney() method, a deadlock occurs if after the ﬁrst
lock is granted, the ﬁrst thread is swapped out and another thread
requests the second lock
public class Bank {
public boolean transferMoney(Account from, Account to, Amount amount) {
synchronized (from) {
// if thread is swapped out here, we can end up with a deadlock
synchronized (to) {
return doActualTransfer(from, to, amount);
}
}
}
}
128


We can force the context switch

129


simply add a sleep after ﬁrst lock

130


Adding a sleep to cause deadlocks
public class Bank {
public boolean transferMoney(Account from, Account to,
Amount amount) {
synchronized (from) {
sleepAWhileForTesting();
synchronized (to) {
return doActualTransfer(from, to, amount);
}
}
}
protected void sleepAWhileForTesting() {}
}

131


In our unit test we use a SlowBank
public class SlowBank extends Bank {
private final long timeout;
private final TimeUnit unit;
public SlowBank(long timeout, TimeUnit unit) {
this.timeout = timeout;
this.unit = unit;
}
protected void sleepAWhileForTesting() {
try {
unit.sleep(timeout);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
132


Won't this cause problems in
production?

133


In production we only use Bank

134


HotSpot will optimize away empty
sleepAWhileForTesting()

135


it is thus a zero cost

136


as long as SlowBank is never used

137


How can you verify the lock-
ordering deadlock?

138


CTRL+Break, jstack, kill -3,
jconsole, etc.

139


but that won't be a good unit test

140


Better via Java's ThreadMXBean

141


ThreadMXBean

• Get instance with ManagementFactory.getThreadMXBean()

• ﬁndMonitorDeadlockedThreads()

• Includes only "monitor" locks, i.e. synchronized

• Only way to ﬁnd deadlocks in Java 5

142


ThreadMXBean

• Better is ﬁndDeadlockedThreads()

• Includes "monitor" and "owned" (Java 5) locks

• Preferred method to test for deadlocks

• But, does not ﬁnd deadlocks between semaphores

• See http://www.javaspecialists.eu/archive/Issue130.html

143


public class BankDeadlockTest {
private final static ThreadMXBean tmb =
ManagementFactory.getThreadMXBean();

private boolean isThreadDeadlocked(long tid) {
long[] ids = tmb.findDeadlockedThreads();
if (ids == null) return false;
for (long id : ids) {
if (id == tid) return true;
}
return false;
}

144


private void checkThatThreadTerminates(Thread thread)
throws InterruptedException {
for (int i = 0; i < 2000; i++) {
thread.join(50);
if (!thread.isAlive()) return;
if (isThreadDeadlocked(thread.getId())) {
fail("Deadlock detected!");
}
}
fail(thread + " did not terminate in time");
}

145

@Test
public void testForTransferDeadlock() throws InterruptedException {
final Account alpha = new Account(new Amount(1000));
final Account ubs = new Account(new Amount(1000000));
final Bank bank = new SlowBank(100, TimeUnit.MILLISECONDS);

Thread alphaToUbs = new Thread("alphaToUbs") {
public void run() {
bank.transferMoney(alpha, ubs, new Amount(100));
}
};
Thread ubsToAlpha = new Thread("ubsToAlpha") {
public void run() {
bank.transferMoney(ubs, alpha, new Amount(100));
}
};

alphaToUbs.start();
ubsToAlpha.start();

checkThatThreadTerminates(alphaToUbs);
}
}
146


Output with broken
transferMoney() method
• We see the deadlock within about 100 milliseconds
junit.framework.AssertionFailedError: Deadlock detected!
at BankDeadlockTest.checkThatThreadTerminates(BankDeadlockTest.java:20)
at BankDeadlockTest.testForTransferDeadlock(BankDeadlockTest.java:55)

147


Output with broken
transferMoney() method

• If we ﬁx the transferMoney() method, it also completes within about
100 milliseconds

• This is the time that we are sleeping for testing purposes

• Remember that the empty sleepAWhileForTesting() method will be
optimized away by HotSpot

148


Timed lock attempts

• Another technique for solving deadlocks is to use the timed tryLock()
method of explicit Java locks

149


However, if tryLock() fails it may be

150


a deadlock

151


or some thread taking a long time

152


or a thread in an inﬁnite loop

153


But ThreadMXBean shows it as
deadlocked

154


even if it is a temporary condition

155


The JVM has support for tryLock
with synchronized

156


but it is hidden away

157


So here it is

158


159


Told you it was hidden!

160


It is hidden for a good reason

161


if you need tryLock functionality

162


rather use explicit locks
(ReentrantLock)

163


tryLock with synchronized

• Technically possible in Sun JVM using

• Unsafe.getUnsafe().tryMonitorEnter(monitor)

• Unsafe.getUnsafe().monitorExit(monitor)

• http://www.javaspecialists.eu/archive/Issue194.html

164


Deadlock analysis with thread
dumps

165


Deadlock analysis with thread
dumps
• We can also cause a thread dump in many ways:

• Ctrl+Break on Windows or Ctrl- on Unix

• Invoking "kill -3" on the process id

• Calling jstack on the process id

• Only shows deadlocks since Java 6

• Intrinsic locks show more information of where they were acquired

166


Can a deadlock be resolved?

167


NO with synchronized

168


YES with ReentrantLock

169


synchronized() goes into
BLOCKED state

170


ReentrantLock.lock() goes into
WAITING state

171


We can write a DealockArbitrator

172


This stops one of the threads with
a DeadlockVictimError

173


Our special Error for deadlocks
public class DeadlockVictimError extends Error {
private final Thread victim;
public DeadlockVictimError(Thread victim) {
super("Deadlock victim: " + victim);
this.victim = victim;

}
public Thread getVictim() { return victim; }
}

174

public class DeadlockArbitrator {
private static final ThreadMXBean tmb =
ManagementFactory.getThreadMXBean();

public boolean tryResolveDeadlock(
int attempts, long timeout, TimeUnit unit)
throws InterruptedException {
for (int i = 0; i < attempts; i++) {
long[] ids = tmb.findDeadlockedThreads();
if (ids == null) return true;
Thread t = findThread(ids[i % ids.length]);
if (t == null)
throw new IllegalStateException("Could not find thread");
t.stop(new DeadlockVictimError(t));
unit.sleep(timeout);
}
return false;
}
175


ﬁnding threads by id is tricky
public boolean tryResolveDeadlock() throws InterruptedException {
return tryResolveDeadlock(3, 1, TimeUnit.SECONDS);
}

private Thread findThread(long id) {
for (Thread thread : Thread.getAllStackTraces().keySet()) {
if (thread.getId() == id) return thread;
}
return null;
}
}

176


Applicability of DeadlockArbitrator

• Only use in extreme circumstances

• Code that is outside your control and that deadlocks

• Where you cannot prevent the deadlock

• Remember, it only works with ReentrantLock

177


Deadlock Victim Questions?
by Dr Heinz Kabutz && Olivier Croisier
The Java Specialists Newsletter && The Coder's Breakfast
heinz@javaspecialists.eu && olivier.croisier@zenika.com
twitter: @heinzkabutz && twitter: @oliviercroisier

178

Deadlock Victim

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (19)

More from Zenika

More from Zenika (20)

Recently uploaded

Recently uploaded (20)

Deadlock Victim