Helvetia. Lukas Renggli, ESUG 2010, Barcelona, Spain

1. Mastering Embedded Languages
Lukas Renggli
Software Composition Group
University of Bern, Switzerland
!" #"$%&

2. Smalltalk is a highly
dynamic language
PhotobySiasvanSchalkwyk,sxc.hu

3. Smalltalk syntax
is locked down
PhotobySimeonEichmann,sxc.hu

4. Host-language compiler
can be patched
PhotobyCamillaReenberg,sxc.hu

5. Patching wrecks your
development tools
PhotobyArieKok,sxc.hu

7. Mastering Embedded Languages
!" #"$%&

8. Marco Zanoli, cc-by-sa, 2.5, www.wikipedia.de, 2006

9. Marco Zanoli, cc-by-sa, 2.5, www.wikipedia.de, 2006

10. Marco Zanoli, cc-by-sa, 2.5, www.wikipedia.de, 2006

11. §
§
§
Marco Zanoli, cc-by-sa, 2.5, www.wikipedia.de, 2006

14. Requirements

15. Adopt, Extend, Overload

16. Adopt, Extend, Overload
change semantics

17. Adopt, Extend, Overload
change semantics
introduce new syntax

18. Adopt, Extend, Overload
change semantics
introduce new syntax
change syntax & semantics

19. Multiple Context
Dependent Languages

20. Homogeneous Data
and Code Abstraction

21. Homogeneous
Tool Support
§
§
§

22. Context Specific Languages
with Homogeneous Tool Integration
!" #"$%&

23. Package Name
x = 1
y = 1 (2, 1)
(2, 2)(1, 2)
x = 2
y = 2

24. aBuilder row grow.
aBuilder row fill.
aBuilder column grow.
aBuilder column fill.
aBuilder x: 1 y: 1 add: (LabelShape new
text: [ :each | each name ];
borderColor: #black;
borderWidth: 1;
yourself).
aBuilder x: 1 y: 2 w: 2 h: 1 add: (RectangleShape new
borderColor: #black;
borderWidth: 1;
width: 200;
height: 100;
yourself)
Package Name
x = 1
y = 1 (2, 1)
(2, 2)(1, 2)
x = 2
y = 2

25. row = grow.
row = fill.
column = grow.
column = fill.
(1 , 1) = label
text: [ :each | each name ];
borderColor: #black;
borderWidth: 1.
(1 , 2) - (2 , 1) = rectangle
borderColor: #black;
borderWidth: 1;
width: 200;
height: 100.
Package Name
x = 1
y = 1 (2, 1)
(2, 2)(1, 2)
x = 2
y = 2

26. A pidgin is a simplified form of the
host language. It introduces new
vocabulary and semantic meaning.

27. shape {
cols: #grow, #fill;
rows: #grow, #fill;
}
label {
position: 1 , 1;
text: [ :each | each name ];
borderColor: #black;
borderWidth: 1;
}
rectangle {
position: 1 , 2;
colspan: 2;
borderColor: #black;
borderWidth: 1;
width: 200;
height: 100;
}
Package Name
x = 1
y = 1 (2, 1)
(2, 2)(1, 2)
x = 2
y = 2

28. A creole a is a new language
formed from the contact of
multiple languages.

29. The He vetia Model!" #"$%&

30. Traditional Smalltalk Compiler
Smalltalk
Parser
Semantic
Analysis
Bytecode
Generation
Executable
Code
Source
Code

31. Traditional Smalltalk Compiler
Smalltalk
Parser
Semantic
Analysis
Bytecode
Generation
Executable
Code
Source
Code
<parse> <transform> <attribute> <bytecode>
Rules

32. Traditional Smalltalk Compiler
Smalltalk
Parser
Semantic
Analysis
Bytecode
Generation
Executable
Code
Source
Code
<parse> <transform> <attribute> <bytecode>
Rules
<complete> <highlight> ......

33. Traditional Smalltalk Compiler
Smalltalk
Parser
Semantic
Analysis
Bytecode
Generation
Executable
Code
Source
Code
<parse> <transform> <attribute> <bytecode>
Rules
Pidgin path
Creole path
<complete> <highlight> ......

34. Homogenous
Embedding

35. Homogeneous Tools

36. IV + VII = XI
,>++++++[<-­‐-­‐-­‐-­‐-­‐-­‐-­‐-­‐>-­‐],[<+>-­‐]<.

37. Host Language

38. Host Language
Renggli et al.ECOOP 2010

39. Language Boxes
Modular Language Changes

40. Editor
Compiler
Debugger
Syntax
Language 1
Language 2

41. Editor
Compiler
Debugger
Syntax
Language 1
Language 2

42. Editor
Compiler
Debugger
Syntax
Language 1
Language 2

43. Editor
Compiler
Debugger
Syntax
Language 1
Language 2

44. Marco Zanoli, cc-by-sa, 2.5, www.wikipedia.de, 2006

45. /w+@w+.w+/

46. Language
Change
Language
Concern
Language
Scope
Language
Box
*
*
*
*

47. Language Change
High-level Description

48. Language Change
Add Regex
as an additional choice
to Primary .

49. Language Change
/Regex: /^/

50. Language Change
StringPrimary:
Number

51. Regex
Language Change
StringPrimary:
Number

52. Language Concern
Production Action

53. Language Concern
Compiler
aToken asRegex lift

54. Highlighting
aToken -> Color orange
Language Concern

55. Other Concerns
Language Concern
Context Menus
Navigation
Search
Code Expansion
Code Completion
Error Correction
Custom Inspector

56. Language Scope
Active?

57. ‣ System
‣ Packages
‣ Classes
‣ Methods
Language Scope

58. Multiple Language
Extensions

59. Host Language
Grammar

60. Host Language
Grammar
Lan
B
Language
Box 1
Language
Box 2

61. Language
Box 3
Language
Box 1
Language
Box 2
Lan
B
Host Language
Grammar

62. Language
Box 3
ge
2
Language
Box 4
Host Language
Grammar

63. Host Language
Grammar
Custom

64. ‣ Fine-grained language changes
‣ Fine-grained language scoping
‣ Composable and reusable model
‣ Tool integration (Editor, Debugger)

65. Language Boxes
Host Language

66. Language Boxes
Host Language
Renggli et al.SLE 2009

67. Language Boxes
Host Language
Dynamic
Grammars

68. [Adapted from The New Oxford American Dictionary on Grammar]
grammar
a set of rules governing what
strings are valid or allowable
in a [formal] language.

69. dynamic grammar
a high-level grammar that
executes at runtime behaviors
that other grammars perform
during compilation, see
dynamic languages.
[Adapted from Wikipedia on Dynamic Programming Languages]

70. Dynamic Grammars
Dynamic Languages
‣ Late-bound behavior
‣ First-class representation
‣ On-the-fly transformation
‣ Introspection and reflection

71. Why would
we want that?

72. Language
Change
Language
Concern
Language
Scope
Language
Box
*
*
*
*

73. Language
Change
=
Host Language
Grammarδ

74. Language
Change
=
Grammar
Transformation

75. scanIdentifier
self step.
((currentCharacter between: $A and: $Z) or:
[ currentCharacter between: $a and: $z ]) ifTrue: [
[ self recordMatch: #IDENTIFIER.
self step.
(currentCharacter between: $0 and: $9)
or: [ (currentCharacter between: $A and: $Z) or:
[ currentCharacter between: $a and: $z ] ] ]
whileTrue.
^ self reportLastMatch ]

76. scanIdentifier
self step.
((currentCharacter between: $A and: $Z) or:
[ currentCharacter between: $a and: $z ]) ifTrue: [
[ self recordMatch: #IDENTIFIER.
self step.
(currentCharacter between: $0 and: $9)
or: [ (currentCharacter between: $A and: $Z) or:
[ currentCharacter between: $a and: $z ] ] ]
whileTrue.
^ self reportLastMatch ]
✖

77. #( #[1 0 9 0 25 0 13 0 34 0 17 0 40 0 21 0 41]
#[1 0 9 0 25 0 13 0 34 0 93 0 76 0 157 0 112]
#[1 2 38 0 21 2 38 0 25 2 38 0 26 0 13 0 34]
#[0 1 154 0 16 0 21 0 25 0 26 0 34 0 40 0 41]
#[0 1 210 0 76 0 81]
#[0 1 214 0 76 0 81]
#[1 0 173 0 76 0 177 0 81]
#[0 1 134 0 16 0 21 0 25 0 26 0 34 0 40 0 41]
#[1 1 46 0 21 1 46 0 25 1 46 0 26 1 69]
#[1 1 54 0 21 1 54 0 25 1 54 0 26 1 54 0 34]
#[0 2 102 0 21 0 25 0 26 0 34 0 40 0 41 0 76]
#[0 2 50 0 21 0 25 0 26 0 76 0 79]
#[1 1 13 0 76 2 85 0 124 1 21 0 125]
#[1 2 89 0 17 2 30 0 21 2 30 0 82]
#[1 2 93 0 21 2 97 0 82] )

78. #( #[1 0 9 0 25 0 13 0 34 0 17 0 40 0 21 0 41]
#[1 0 9 0 25 0 13 0 34 0 93 0 76 0 157 0 112]
#[1 2 38 0 21 2 38 0 25 2 38 0 26 0 13 0 34]
#[0 1 154 0 16 0 21 0 25 0 26 0 34 0 40 0 41]
#[0 1 210 0 76 0 81]
#[0 1 214 0 76 0 81]
#[1 0 173 0 76 0 177 0 81]
#[0 1 134 0 16 0 21 0 25 0 26 0 34 0 40 0 41]
#[1 1 46 0 21 1 46 0 25 1 46 0 26 1 69]
#[1 1 54 0 21 1 54 0 25 1 54 0 26 1 54 0 34]
#[0 2 102 0 21 0 25 0 26 0 34 0 40 0 41 0 76]
#[0 2 50 0 21 0 25 0 26 0 76 0 79]
#[1 1 13 0 76 2 85 0 124 1 21 0 125]
#[1 2 89 0 17 2 30 0 21 2 30 0 82]
#[1 2 93 0 21 2 97 0 82] )
✖

80. a..z a..z
0..9
ID ::= letter { letter | digit } ;

81. letter
letter digit
sequence
choice
star
id := #letter , (#letter / #digit) star

82. letter
letter digit
sequence
choice
star
id := #letter asParser ,
(#letter asParser / #digit asParser) star

83. letter
letter digit
sequence
choice
star
ID ::= letter { letter | digit } ;
!" #"$%&

84. Scannerless Parser

85. Parsing Expression
Grammar

86. Packrat Parser

87. Parser Combinator

89. Grammar
Transformation

90. Graph
Rewriting

91. letter
letter digit
sequence
choice
star
#letter → #letter / $_

92. letter
digit
sequence
choice
starchoice
_
letter
choice
_
#letter → #letter / $_

93. Optimizations

94. letter
digit
sequence
choice
starchoice
_
letter
choice
_
(`a / `b) / `c → `a / `b / `c

95. letter
digit
sequence
choice
starchoice
_
letter _
(`a / `b) / `c → `a / `b / `c

96. letter
digit
sequence
choice
starchoice
_
letter _
remove duplicates

97. letter digit
sequence
choice
starchoice
_
remove duplicates

98. letter digit
sequence
choice
starchoice
_
remove duplicates
in general
a graph

99. Conflicts?

100. SELECT * FROM users

101. SELECT * FROM users
| r |
r :=
^ User fromRow: r
.

102. expr sql|

103. SELECT * FROM users<SQL: >

104. expr sql/
ordered

105. No Conflicts

106. No Ambiguities

107. expr sql/
surprise

108. / exprsql
surprise

109. expr sql|
!expr sql / !sql expr

110. expr sql$
!expr sql / !sql expr / ui

111. Language Boxes
Host Language
Dynamic
Grammars

112. Language Boxes
Host Language
Dynamic
Grammars
Renggli et al.DYLA 2010

113. Language Boxes
Host Language
Dynamic
Grammars
Embedded Languages

114. apply
hasChanged
hasConflict
Change
object
*
changes
Process
0..1
currentTransaction
do: aBlock
retry: aBlock
checkpoint
abort: anObject
escapeContext
Transaction
previousCopy
workingCopy
ObjectChange
applyBlock
conflictTestBlock
CustomChange
*
Transactional Memory

115. Domain-Specific
Program Checking

116. Meta-Programming
Facilities
``(`,(aString) asRegex)
The LISP stuff,
for a cool language

117. Language Boxes
Host Language
Dynamic
Grammars
Language Extensions
Renggli et al.
CLSS 2009 Renggli et al.
IWST 2009 Nierstrasz et al.
LNCS 2009 Renggli et al.
TOOLS 2010

118. scg.unibe.ch/research/helvetia