Parsing Contexts for PetitParser IWST 2014 at ESUG, Cambridge

1. Parsing Contexts
for PetitParser
Jan Kurš
kurs@iam.unibe.ch
Software Composition Group

2. 2
print “Hello!”
if (outOf(milk)):
print “We are out of milk!”
order(milk)
if (outOf(eggs)):
…
if keyword
block of commands
eggs identifier
Indentation Example

3. 3
print “Hello!”
if (outOf(milk)):
print “We are out of milk!”
order(milk)
if (outOf(eggs)):
…
if keyword
block of commands
eggs identifier
Indentation Example

4. 4
print “Hello!”
if (outOf(milk)):
print “We are out of milk!”
order(milk)
if (outOf(eggs)):
…
if keyword
block of commands
eggs identifier
Indentation Example
ifKeyword := 'if' asParser.

5. 5
print “Hello!”
if (outOf(milk)):
print “We are out of milk!”
order(milk)
if (outOf(eggs)):
…
if keyword
block of commands
eggs identifier
Indentation Example

6. 6
print “Hello!”
if (outOf(milk)):
print “We are out of milk!”
order(milk)
if (outOf(eggs)):
…
if keyword
block of commands
eggs identifier
Indentation Example
identifier := #letter, (#letter / #digit) star.

7. 7
print “Hello!”
if (outOf(milk)):
print “We are out of milk!”
order(milk)
if (outOf(eggs)):
…
if keyword
block of commands
eggs identifier
Indentation Example

8. 8
print “Hello!”
if (outOf(milk)):
print “We are out of milk!”
order(milk)
if (outOf(eggs)):
…
if keyword
block of commands
eggs identifier
Indentation Example
block := indent, command plus, dedent.

9. 9
print “Hello!”
if (outOf(milk))
print “We are out of milk!”
order(milk)
if (outOf(eggs))
…
if keyword
block of commands
eggs identifier
Indentation Example
block := indent, command plus, dedent.

10. 10
print “Hello!”
if (outOf(milk))
print “We are out of milk!”
order(milk)
if (outOf(eggs))
…
if keyword
block of commands
eggs identifier
Indentation Example
block := indent, command plus, dedent.
indent := ???
dedent := ???

11. Simple Complex 11
Regular
Languages
Context-Free
Languages
Context-Sensitive
State of the Art
Languages

12. 12
Regular
Expression
Simple Complex
Context-Free
Languages
Regular
Languages
State of the Art
Context-Sensitive
Languages

13. 13
Turing
Machine
Regular
Expression
Simple Complex
Context-Free
Languages
Regular
Languages
State of the Art
Context-Sensitive
Languages

14. 14
Turing
Machine
Regular
Expression
Simple Complex
Context-Free
Languages
Regular
Languages
State of the Art
Context-Sensitive
Languages

15. 15
Turing
Machine
Regular
Expression
Simple Complex
Context-Free
Languages
Regular
Languages
State of the Art
Context-Sensitive
Languages

16. 16
Turing
Machine
Regular
Expression
Simple Complex
Context-Free
Languages
Regular
Languages
State of the Art
Context-Sensitive
Languages

17. 17
Turing
Machine
Regular
Expression
Simple Complex
Context-Free
Languages
Regular
Languages
State of the Art
Context-Sensitive
Languages

18. 18
Sensitive
Turing
Machine
Context-Free Context-Regular Regular
Expression
Simple Complex Goal

19. 19
Goal
Sensitive
Turing
Machine
Context-PPaarrssiinngg
CCoonntteexxttss
Free Context-Regular Regular
Expression
Simple Complex

20. 20
Sensitive
Turing
Machine
Context-PPaarrssiinngg
CCoonntteexxttss
Free Context-Regular Regular
Expression
Simple Complex Goal

21. 21
PetitParser uses Parser Combinators

22. 22
PetitParser uses Parser Combinators
#letter, (#letter / #digit) star

23. 23
PetitParser uses Parser Combinators
#letter, (#letter / #digit) star

24. 24
PetitParser uses Parser Combinators
#letter, (#letter / #digit) star
,,
#letter
star
//
#lette r #digit

25. 25
String
String

26. 26
String
String

27. 27
String
String
“Hello”
##lleetttteerr
“ello”

28. 28
print “Hello!”
if (outOf(milk))
print “We are out of milk!”
order(milk)
if (outOf(eggs))
…
if keyword
block of commands
eggs identifier
Indentation Example
block := indent, command plus, dedent.

29. 29
print “Hello!”
if (outOf(milk))
print “We are out of milk!”
order(milk)
if (outOf(eggs))
…
if keyword
block of commands
eggs identifier
Indentation Example
block := indent, command plus, dedent.
indent := ???
dedent := ???

30. 30
print “Hello!”
if (outOf(milk))
print “We are out of milk!”
order(milk)
if (outOf(eggs))
…
if keyword
block of commands
eggs identifier
Indentation Example
block := indent, command plus, dedent.
▼

31. 31
print “Hello!”
if (outOf(milk))
print “We are out of milk!”
order(milk)
if (outOf(eggs))
…
if keyword
block of commands
eggs identifier
Indentation Example
block := indent, command plus, dedent.
indent := ???
dedent := ???
▼

32. 32
String
String

33. 33
String
String

34. 34
String
String
String
&
Indentation Stack
String
&
Indentation Stack

35. 35
String
String
PPaarrssiinngg
CCoonntteexxttss
PPaarrssiinngg
CCoonntteexxttss

36. 36
String
String
PPaarrssiinngg
CCoonntteexxttss
PPaarrssiinngg
CCoonntteexxttss
String
Other Data

37. 37
print “Hello!”
if (outOf(milk))
print “We are out of milk!”
order(milk)
if (outOf(eggs))
…
if keyword
block of commands
eggs identifier
Indentation Example
▼
[:context |
| column indentation stack |
(context isBeginOfLine) ifFalse: [
^ Failure
].
context consumeLeadingWhitespace.
" Save the current column "
column := context stream column.
stack := (context propertyAt: #indent).
indentation := stack top.
(column > indentation) ifTrue: [
stack push: column.
^ #indent
].
^ Failure
]
block := , command plus,
[:context |
| column referenceIndentation stack |
(context isBeginOfLine) ifFalse: [
^ Failure
].
context consumeLeadingWhitespace.
column := context stream column.
" Restore previous column from the "
" stack and compare with current "
stack := (context propertyAt: #indent).
stack pop.
referenceIndentation := stack top.
(column == referenceIndentation) ifTrue: [
^ #dedent
].
^ Failure
]

38. 38
Sensitive
Turing
Machine
Context-PPaarrssiinngg
CCoonntteexxttss
Free Context-Regular Regular
Expression
Simple Complex

39. 39
Sensitive
Turing
Machine
Context-Free Context-Regular Regular
Expression
Simple Complex PPaarrssiinngg
CCoonntteexxttss

40. 40
Grammar
Implementors

41. 41
Grammar
Implementors
Framework
Implementors

42. 42
Grammar
Implementors
Framework
Implementors
N ← (N ∪ T)*

43. 43
Grammar
Implementors
Framework
Implementors
N ← (N ∪ T)*
indent, command plus, dedent

44. 44
Grammar
Implementors
Framework
Implementors
N ← (N ∪ T)*
indent, command plus, dedent
, command plus,
[:context |
| column indentation stack |
(context isBeginOfLine) ifFalse: [
^ Failure
].
context consumeLeadingWhitespace.
" Save the current column "
column := context stream column.
stack := (context propertyAt: #indent).
indentation := stack top.
(column > indentation) ifTrue: [
stack push: column.
^ #indent
].
^ Failure
]
[:context |
| column referenceIndentation stack |
(context isBeginOfLine) ifFalse: [
^ Failure
].
context consumeLeadingWhitespace.
column := context stream column.
" Restore previous column from the "
" stack and compare with current "
stack := (context propertyAt: #indent).
stack pop.
referenceIndentation := stack top.
(column == referenceIndentation) ifTrue: [
^ #dedent
].
^ Failure
]

45. 45
Sensitive
Turing
Machine
PPaarrssiinngg
CCoonntteexxttss
Context-PPaarrssiinngg
CCoonntteexxttss
Free Context-Regular Regular
Expression
Simple Complex

46. 46
Sensitive
Turing
Machine
Context-PPaarrssiinngg
CCoonntteexxttss
Free Context-Regular Regular
Expression
Simple Complex

47. 47
Sensitive
Turing
Machine
Context-PPaarrssiinngg
CCoonntteexxttss
Free Context-Regular Regular
Expression
Simple Complex

48. 48
indentation

49. 49
indentation

50. 50
indentation
tolerant parsing

51. 51
indentation
tolerant parsing

52. 52
indentation
tolerant parsing
typedef

53. 53
indentation
tolerant parsing
typedef

54. 54
indentation
tolerant parsing
typedef
recursion

55. 55
indentation
tolerant parsing
typedef
recursion

56. 56
indentation
typedef
recursion
PPaarrssiinngg
CCoonntteexxttss
tolerant parsing

57. http://smalltalkhub.com/#!/~JanKurs/PetitParser
57

58. 58
Turing
Machine
Regular
Expression
PPaarrssiinngg
CCoonntteexxttss

59. 59
String
String
Parsing
Context
Parsing
Context
Turing
Machine
Regular
Expression
PPaarrssiinngg
CCoonntteexxttss

60. 60
String
String
Parsing
Context
Parsing
Context
N ← (N ∪ T)*
Framework
Implementors
Grammar
Implementor
Turing
Machine
Regular
Expression
PPaarrssiinngg
CCoonntteexxttss

61. 61
String
String
Parsing
Context
Parsing
Context
indentation
tolerant parsing
typedef
recursion
PPaarrssiinngg
CCoonntteexxttss
N ← (N ∪ T)*
Framework
Implementors
Grammar
Implementor
Turing
Machine
Regular
Expression
PPaarrssiinngg
CCoonntteexxttss