Spyware-ridden software development. Romain Robbes, Michele Lanza.ESUG 2006, Prague

1. Romain Robbes
Michele Lanza
Facoltà
di scienze
informatiche
Università
della
Svizzera
italiana
ESUG
2006
Spyware-ridden
software
development

2. Context
What is SpyWare?
Results & demo
Further possibilities
Romain Robbes Spyware-ridden software development
SpyWare is a research
prototype built for my Ph.D.
2
SpyWare
75%
25%
?
?

3. Romain Robbes Spyware-ridden software development
More than 3/4 of the cost of
software is maintenance*
3
75%
25%
initial effort
maintenance
“A program that is
used in a real-world
environment must
change, or become
progressively less
useful in that
environment. ”
– Lehman’s laws
* from: http://www.cs.jyu.fi/~koskinen/smcosts.htm
75%
25%

4. Romain Robbes Spyware-ridden software development
The lifecycle of software
4
Requirements
Gathering
Understanding &
Reengineering
Design &
Implementation
75%
25%

5. Romain Robbes Spyware-ridden software development
Software development is
incremental
5
Version 4Version 1 Version 2 Version 3
75%
25%

6. Romain Robbes Spyware-ridden software development
Software development is
incremental
5
Version 4Version 1 Version 2 Version 3
checkout
commit
debugadd feature
75%
25%

7. Romain Robbes Spyware-ridden software development
Software development is
incremental
5
Version 4Version 1 Version 2 Version 3
checkout
commit
debugadd feature
add method
add class
add method
75%
25%

8. Romain Robbes Spyware-ridden software development
Languages and methodologies
target this problem
6
The holy trinity?
75%
25%

9. Romain Robbes Spyware-ridden software development
The change rate of the
software increases
7
It is easier to lose reference points
Evolvability and understandability are at odds
XP states that the software is in maintenance
100% of the time ...
Are conventional reengineering tools adapted
to agile development?
75%
25%

10. Romain Robbes Spyware-ridden software development
Software evolution analysis
helps reengineering
8
Bar
Foo
- asdf: int
+ Foo(c: int)
+ bar(): void
Quux
ConcreteA
- asdf: int
+ bar(): void
ConcreteB
Bar
- stuff
+ factory(): Bar
ConcreteA
- asdf: int
+ bar(): void
+ baz(): int
ConcreteB
AbstractBar
- stuff
- other
+ factory():
AbstractBar
FooHistory holds
useful information
Versioning
system
Metrics
&
trends
75%
25%

11. Romain Robbes Spyware-ridden software development
Versioning systems lose
information
9
75%
25%

12. Romain Robbes Spyware-ridden software development
> cvs update
Versioning systems lose
information
9
75%
25%

13. Romain Robbes Spyware-ridden software development
> cvs update
> vim Foo.cc
Versioning systems lose
information
9
75%
25%

14. Romain Robbes Spyware-ridden software development
> cvs update
> vim Foo.cc
(some work done...)
Versioning systems lose
information
9
75%
25%

15. Romain Robbes Spyware-ridden software development
> cvs update
> vim Foo.cc
(some work done...)
> cvs commit
Versioning systems lose
information
9
75%
25%

16. Romain Robbes Spyware-ridden software development
> cvs update
> vim Foo.cc
(some work done...)
> cvs commit ?
Versioning systems lose
information
?9
75%
25%

17. class Foo {
private int x;
private int y;
public getX() { return x; }
public setX(newX) { x = newX; }
public getY() { return y; }
public setY(newY) { y = newY; }
public baz() {
blah.blah(blah);
z = getX() + getY();
return bar(z);
}
public quux() {
return getY() + 4;
}
public asdf() {
return getX() * 8 + getY();
}
private bar(z) {
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
}
f = new Foo();
f.baz();
print f.getX() + f.getY();
Romain Robbes Spyware-ridden software development
Our paranoid programmer
saves every 5 minutes
class Foo {
public int x;
public int y;
public doFoo() {
blah.blah(blah);
z = x + y;
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
public quux() {
return y + 4;
}
public asdf() {
return x * 8 + y;
}
}
f = new Foo();
f.doFoo();
print f.x + f.y;
10
75%
25%

18. class Foo {
private int x;
private int y;
public getX() { return x; }
public setX(newX) { x = newX; }
public getY() { return y; }
public setY(newY) { y = newY; }
public baz() {
blah.blah(blah);
z = getX() + getY();
return bar(z);
}
public quux() {
return getY() + 4;
}
public asdf() {
return getX() * 8 + getY();
}
private bar(z) {
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
}
f = new Foo();
f.baz();
print f.getX() + f.getY();
Romain Robbes Spyware-ridden software development
Our paranoid programmer
saves every 5 minutes
class Foo {
public int x;
public int y;
public doFoo() {
blah.blah(blah);
z = x + y;
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
public quux() {
return y + 4;
}
public asdf() {
return x * 8 + y;
}
}
f = new Foo();
f.doFoo();
print f.x + f.y;
1.Extract method
10
75%
25%

19. class Foo {
private int x;
private int y;
public getX() { return x; }
public setX(newX) { x = newX; }
public getY() { return y; }
public setY(newY) { y = newY; }
public baz() {
blah.blah(blah);
z = getX() + getY();
return bar(z);
}
public quux() {
return getY() + 4;
}
public asdf() {
return getX() * 8 + getY();
}
private bar(z) {
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
}
f = new Foo();
f.baz();
print f.getX() + f.getY();
Romain Robbes Spyware-ridden software development
Our paranoid programmer
saves every 5 minutes
class Foo {
public int x;
public int y;
public doFoo() {
blah.blah(blah);
z = x + y;
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
public quux() {
return y + 4;
}
public asdf() {
return x * 8 + y;
}
}
f = new Foo();
f.doFoo();
print f.x + f.y;
1.Extract method
10
75%
25%

20. class Foo {
private int x;
private int y;
public getX() { return x; }
public setX(newX) { x = newX; }
public getY() { return y; }
public setY(newY) { y = newY; }
public baz() {
blah.blah(blah);
z = getX() + getY();
return bar(z);
}
public quux() {
return getY() + 4;
}
public asdf() {
return getX() * 8 + getY();
}
private bar(z) {
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
}
f = new Foo();
f.baz();
print f.getX() + f.getY();
Romain Robbes Spyware-ridden software development
Our paranoid programmer
saves every 5 minutes
class Foo {
public int x;
public int y;
public doFoo() {
blah.blah(blah);
z = x + y;
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
public quux() {
return y + 4;
}
public asdf() {
return x * 8 + y;
}
}
f = new Foo();
f.doFoo();
print f.x + f.y;
1.Extract method
2.Rename method
10
75%
25%

21. class Foo {
private int x;
private int y;
public getX() { return x; }
public setX(newX) { x = newX; }
public getY() { return y; }
public setY(newY) { y = newY; }
public baz() {
blah.blah(blah);
z = getX() + getY();
return bar(z);
}
public quux() {
return getY() + 4;
}
public asdf() {
return getX() * 8 + getY();
}
private bar(z) {
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
}
f = new Foo();
f.baz();
print f.getX() + f.getY();
Romain Robbes Spyware-ridden software development
Our paranoid programmer
saves every 5 minutes
class Foo {
public int x;
public int y;
public doFoo() {
blah.blah(blah);
z = x + y;
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
public quux() {
return y + 4;
}
public asdf() {
return x * 8 + y;
}
}
f = new Foo();
f.doFoo();
print f.x + f.y;
1.Extract method
2.Rename method
10
75%
25%

22. class Foo {
private int x;
private int y;
public getX() { return x; }
public setX(newX) { x = newX; }
public getY() { return y; }
public setY(newY) { y = newY; }
public baz() {
blah.blah(blah);
z = getX() + getY();
return bar(z);
}
public quux() {
return getY() + 4;
}
public asdf() {
return getX() * 8 + getY();
}
private bar(z) {
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
}
f = new Foo();
f.baz();
print f.getX() + f.getY();
Romain Robbes Spyware-ridden software development
Our paranoid programmer
saves every 5 minutes
class Foo {
public int x;
public int y;
public doFoo() {
blah.blah(blah);
z = x + y;
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
public quux() {
return y + 4;
}
public asdf() {
return x * 8 + y;
}
}
f = new Foo();
f.doFoo();
print f.x + f.y;
1.Extract method
2.Rename method
3.Create accessors
10
75%
25%

23. class Foo {
private int x;
private int y;
public getX() { return x; }
public setX(newX) { x = newX; }
public getY() { return y; }
public setY(newY) { y = newY; }
public baz() {
blah.blah(blah);
z = getX() + getY();
return bar(z);
}
public quux() {
return getY() + 4;
}
public asdf() {
return getX() * 8 + getY();
}
private bar(z) {
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
}
f = new Foo();
f.baz();
print f.getX() + f.getY();
Romain Robbes Spyware-ridden software development
Our paranoid programmer
saves every 5 minutes
class Foo {
public int x;
public int y;
public doFoo() {
blah.blah(blah);
z = x + y;
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
public quux() {
return y + 4;
}
public asdf() {
return x * 8 + y;
}
}
f = new Foo();
f.doFoo();
print f.x + f.y;
1.Extract method
2.Rename method
3.Create accessors
10
75%
25%

24. class Foo {
private int x;
private int y;
public getX() { return x; }
public setX(newX) { x = newX; }
public getY() { return y; }
public setY(newY) { y = newY; }
public baz() {
blah.blah(blah);
z = getX() + getY();
return bar(z);
}
public quux() {
return getY() + 4;
}
public asdf() {
return getX() * 8 + getY();
}
private bar(z) {
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
}
f = new Foo();
f.baz();
print f.getX() + f.getY();
Romain Robbes Spyware-ridden software development
Our paranoid programmer
saves every 5 minutes
class Foo {
public int x;
public int y;
public doFoo() {
blah.blah(blah);
z = x + y;
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
public quux() {
return y + 4;
}
public asdf() {
return x * 8 + y;
}
}
f = new Foo();
f.doFoo();
print f.x + f.y;
1.Extract method
2.Rename method
3.Create accessors
+18 / -11CVS:
10
75%
25%

25. class Foo {
private int x;
private int y;
public getX() { return x; }
public setX(newX) { x = newX; }
public getY() { return y; }
public setY(newY) { y = newY; }
public baz() {
blah.blah(blah);
z = getX() + getY();
return bar(z);
}
public quux() {
return getY() + 4;
}
public asdf() {
return getX() * 8 + getY();
}
private bar(z) {
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
}
f = new Foo();
f.baz();
print f.getX() + f.getY();
Romain Robbes Spyware-ridden software development
Our paranoid programmer
saves every 5 minutes
class Foo {
public int x;
public int y;
public doFoo() {
blah.blah(blah);
z = x + y;
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
public quux() {
return y + 4;
}
public asdf() {
return x * 8 + y;
}
}
f = new Foo();
f.doFoo();
print f.x + f.y;
1.Extract method
2.Rename method
3.Create accessors
+18 / -11CVS:
10
75%
25%

26. Romain Robbes Spyware-ridden software development
Recapitulation
11
Software evolution is hard but
necessary
Agile practices make it change even
faster
Versioning systems are inappropriate
sources of information for SE tools
75%
25%
75%
25%
Versioning
system

27. Romain Robbes Spyware-ridden software development
The principles behind SpyWare
12
1.Model changes, not versions
2.Use the IDE, not the code repository
3.Provide tools in the IDE
?

28. Romain Robbes Spyware-ridden software development
Modelling and capturing change
information
Rather than storing versions, we record
the semantic actions of the programmer
13
ConcreteA
- asdf: int
+ bar(): void
ConcreteB
Bar
- stuff
+ factory(): Bar
ConcreteA
- asdf: int
+ bar(): void
+ baz(): int
ConcreteB
AbstractBar
- stuff
- other
+ factory():
AbstractBar
Foo
?

29. Romain Robbes Spyware-ridden software development
Modelling and capturing change
information
Rather than storing versions, we record
the semantic actions of the programmer
13
ConcreteA
- asdf: int
+ bar(): void
ConcreteB
Bar
- stuff
+ factory(): Bar
ConcreteA
- asdf: int
+ bar(): void
+ baz(): int
ConcreteB
AbstractBar
- stuff
- other
+ factory():
AbstractBar
Foo
add class
add method
rename class
add attribute
change method
change method
?

30. Romain Robbes Spyware-ridden software development
The general architecture of
SpyWare
14
IDE
SpyWare
plugin
Change
repository
Tools
ConcreteA
- asdf: int
+ bar(): void
+ baz(): int
ConcreteB
AbstractBar
- stuff
- other
+ factory():
AbstractBar
Foo
Bar
Foo
- asdf: int
+ Foo(c: int)
+ bar(): void
Quux
ConcreteA
- asdf: int
+ bar(): void
+ baz(): int
ConcreteB
AbstractBar
- stuff
- other
+ factory():
AbstractBar
Foo
Bar
Foo
- asdf: int
+ Foo(c: int)
+ bar(): void
Quux
:
?

31. Romain Robbes Spyware-ridden software development
How does SpyWare work?
15
?

32. Romain Robbes Spyware-ridden software development
How does SpyWare work?
15
myNewMethod: foo
?

33. Romain Robbes Spyware-ridden software development
How does SpyWare work?
15
myNewMethod: foo
self bar: foo + 1.
?

34. Romain Robbes Spyware-ridden software development
How does SpyWare work?
15
myNewMethod: foo
self bar: foo + 1.
self baz: foo - 1.
?

35. Romain Robbes Spyware-ridden software development
How does SpyWare work?
15
myNewMethod: foo
self bar: foo + 1.
self baz: foo - 1.
Incremental
compilation
Method
compiled!
?

36. Romain Robbes Spyware-ridden software development
How does SpyWare work?
15
myNewMethod: foo
self bar: foo + 1.
self baz: foo - 1.
?

37. Romain Robbes Spyware-ridden software development
The change builder infers
change information
16
?

38. Romain Robbes Spyware-ridden software development
The change builder infers
change information
16
Method
compiled!
?

39. Romain Robbes Spyware-ridden software development
The change builder infers
change information
16
Is it a new one?
Yes!
Who? when?
Romain Robbes
@ 21h32
?

40. Romain Robbes Spyware-ridden software development
The change builder infers
change information
16
create method
add to class
create variable
add to method
OK...
?

41. Romain Robbes Spyware-ridden software development
The change builder infers
change information
16
?

42. Romain Robbes Spyware-ridden software development
Tools exploit the changes
17
Model maintainer
Bar
quux
baz:
?

43. Romain Robbes Spyware-ridden software development
Tools exploit the changes
17
create method
add to class
create variable
add to method
Model maintainer
Bar
quux
baz:
?

44. Romain Robbes Spyware-ridden software development
Tools exploit the changes
17
create method
add to class
create variable
add to method
Model maintainer
Bar
fooquux
baz: myNewMethod:
?

45. Romain Robbes Spyware-ridden software development
SpyWare case studies:
18
SpyWare
×9 ×1

46. Romain Robbes Spyware-ridden software development
Short SpyWare demo
19
SpyWare

47. Romain Robbes Spyware-ridden software development
Future possibilities
20
Several tools can be implemented on top of
this mechanism: here are a few.
?

48. Romain Robbes Spyware-ridden software development
Fine-grained software
evolution analysis
21
class Foo {
private int x;
private int y;
public getX() { return x; }
public setX(newX) { x = newX; }
public getY() { return y; }
public setY(newY) { y = newY; }
public baz() {
blah.blah(blah);
z = getX() + getY();
return bar();
}
public quux() {
return getY() + 4;
}
public asdf() {
return getX() * 8 + getY();
}
private bar(z) {
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
}
f = new Foo();
f.baz();
print f.getX() + f.getY();
class Foo {
public int x;
public int y;
public doFoo() {
blah.blah(blah);
z = x + y;
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
public quux() {
return y + 4;
}
public asdf() {
return x * 8 + y;
}
}
f = new Foo();
f.doFoo();
print f.x + f.y;
class Foo {
private int x;
private int y;
public getX() { return x; }
public setX(newX) { x = newX; }
public getY() { return y; }
public setY(newY) { y = newY; }
public baz() {
blah.blah(blah);
z = getX() + getY();
return bar();
}
public quux() {
return getY() + 4;
}
public asdf() {
return getX() * 8 + getY();
}
private bar(z) {
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
}
f = new Foo();
f.baz();
print f.getX() + f.getY();
class Foo {
public int x;
public int y;
public doFoo() {
blah.blah(blah);
z = x + y;
blu = blu * 2;
t = blurg(z);
bli[t] = blu;
return t;
}
public quux() {
return y + 4;
}
public asdf() {
return x * 8 + y;
}
}
f = new Foo();
f.doFoo();
print f.x + f.y;
Versioning
System
IDE
?
?

49. Romain Robbes Spyware-ridden software development
Query and understand recent
changes
22
ConcreteA
- asdf: int
+ bar(): void
+ baz(): int
ConcreteB
AbstractBar
- stuff
- other
+ factory():
AbstractBar
Foo
Bar
Foo
- asdf: int
+ Foo(c: int)
+ bar(): void
Quux
?

50. Romain Robbes Spyware-ridden software development
Query and understand recent
changes
22
ConcreteA
- asdf: int
+ bar(): void
+ baz(): int
ConcreteB
AbstractBar
- stuff
- other
+ factory():
AbstractBar
Foo
Bar
Foo
- asdf: int
+ Foo(c: int)
+ bar(): void
Quux
?

51. Romain Robbes Spyware-ridden software development
Query and understand recent
changes
22
ConcreteA
- asdf: int
+ bar(): void
+ baz(): int
ConcreteB
AbstractBar
- stuff
- other
+ factory():
AbstractBar
Foo
Bar
Foo
- asdf: int
+ Foo(c: int)
+ bar(): void
Quux
?

52. ConcreteA
- asdf: int
+ bar(): void
+ baz(): int
ConcreteB
AbstractBar
- stuff
- other
+ factory():
AbstractBar
Foo
Bar
Foo
- asdf: int
+ Foo(c: int)
+ bar(): void
Quux
Romain Robbes Spyware-ridden software development
Find precise causes of bugs
23
?

53. ConcreteA
- asdf: int
+ bar(): void
+ baz(): int
ConcreteB
AbstractBar
- stuff
- other
+ factory():
AbstractBar
Foo
Bar
Foo
- asdf: int
+ Foo(c: int)
+ bar(): void
Quux
Romain Robbes Spyware-ridden software development
Find precise causes of bugs
23
?

54. ConcreteA
- asdf: int
+ bar(): void
+ baz(): int
ConcreteB
AbstractBar
- stuff
- other
+ factory():
AbstractBar
Foo
Bar
Foo
- asdf: int
+ Foo(c: int)
+ bar(): void
Quux
Romain Robbes Spyware-ridden software development
Find precise causes of bugs
23
?

55. ConcreteA
- asdf: int
+ bar(): void
+ baz(): int
ConcreteB
AbstractBar
- stuff
- other
+ factory():
AbstractBar
Foo
Bar
Foo
- asdf: int
+ Foo(c: int)
+ bar(): void
Quux
Romain Robbes Spyware-ridden software development
Find precise causes of bugs
23
:-)
?

56. ConcreteA
- asdf: int
+ bar(): void
+ baz(): int
ConcreteB
AbstractBar
- stuff
- other
+ factory():
AbstractBar
Foo
Bar
Foo
- asdf: int
+ Foo(c: int)
+ bar(): void
Quux
Romain Robbes Spyware-ridden software development
Find precise causes of bugs
23
:-)
?

57. ConcreteA
- asdf: int
+ bar(): void
+ baz(): int
ConcreteB
AbstractBar
- stuff
- other
+ factory():
AbstractBar
Foo
Bar
Foo
- asdf: int
+ Foo(c: int)
+ bar(): void
Quux
Romain Robbes Spyware-ridden software development
Find precise causes of bugs
23
:-)
?

58. Romain Robbes Spyware-ridden software development
Find merge conflicts faster
24
?

59. Romain Robbes Spyware-ridden software development
Find merge conflicts faster
24
!
!
?

60. Romain Robbes Spyware-ridden software development
Conclusions
25

61. Romain Robbes Spyware-ridden software development
SpyWare introduces a model of
software changes
26
+ no information lost
+ accuracy
+ tool support
ConcreteA
- asdf: int
+ bar(): void
ConcreteB
Bar
- stuff
+ factory(): Bar
ConcreteA
- asdf: int
+ bar(): void
+ baz(): int
ConcreteB
AbstractBar
- stuff
- other
+ factory():
AbstractBar
Foo
- performance?
- space?
- validation?

62. FOR SPYWARE NOW
http://romain.robb.es/spyware

64. Entity
- id
History
- changes
Change
- id
- timestamp
- author
EntityState
- entity
- parent
- children
- properties
1
1
1 *
1 *
1
*
Romain Robbes Spyware-ridden software development
Our model emphasizes changes
over entities
29

65. Romain Robbes Spyware-ridden software development
Spyware versus the change log
30
Ad-hoc format (a bunch of do-its)
Not aware of refactorings
Tied to one image
Data loss because of purges
Exporting with change sets only keeps
the last version of a method