The Art Of Parsing @ Devoxx France 2014

@dbolkensteyn @_godin_#parsing
The Art of Parsing
Evgeny Mandrikov @_godin_
Dinesh Bolkensteyn @dbolkensteyn
http://sonarsource.com

@dbolkensteyn @_godin_#parsing 2/56
The Art of Parsing
// TODO: don't forget to add huge disclaimer that all opinions hereinbelow
are our own and not our employer (they wish they had them)
Evgeny Mandrikov
@_godin_
Dinesh Bolkensteyn
@dbolkensteyn

I want to create a parser
«Done»!
Use Yacc, JavaCC,
ANTLR, SSLR, …
or hand-written ?

What is the plan?
Why
• javac and GCC are hand-written
• do we use parser-generators ?
Together we will implement parser for
• arithmetic expressions
• common constructions from Java
• C++ ;)

Java formal grammar
JLS8
JLS7

Answer is
42

Pill of theory
NUM ➙ 42
Nonterminal
Productions
Terminals
(tokens)

Grammar for numbers
NUM ➙ NUM DIGIT
| DIGIT
DIGIT ➙ 0|1|2|3|4|5|6|7|8|9
4, 8, 15, 16, 23, 42,…
Alternatives

Arithmetic expressions
4 – 3 – 2 = ?

expr ➙ expr – expr
| NUM
4 – 3 – 2 = ?

expr
4 3
2
expr
| NUM
(4 – 3)– 2 =-1

4
3 2
expr
expr
| NUM
(4 – 3)– 2 =-1
4 –(3 – 2)= 3
expr
4 3
2
expr

expr ➙ NUM – expr
| NUM
| NUM
(4 – 3)– 2 =-1
4 –(3 – 2)= 3
expr
4 3
2
expr 4
3 2
expr
expr

expr ➙ NUM – expr
| NUM
| NUM
expr ➙ expr – NUM
| NUM
(4 – 3)– 2 =-1
4 –(3 – 2)= 3
4
3 2
expr
expr
expr
4 3
2
expr

Show me the code
int expr() {
int res = expr();
if (token == '–')
return res – num();
return num();
}
int expr() {
int res = expr();
if (token == '–')
return num();
}
| NUM

Show me the code right code
??
int expr() {
int res = expr();
if (token == '–')
return num();
}
int expr() {
int res = expr();
if (token == '–')
return num();
}
| NUM

int expr() {
int res = expr();
if (token == '–')
return num();
}
int expr() {
int res = expr();
if (token == '–')
return num();
}
| NUM
int expr() {
int res = num();
while (token == '–')
res = res – num();
return res;
}
int expr() {
int res = num();
res = res – num();
return res;
}

4 – 3 * 2 = ?

4 – 3 * 2 = -2
| expr * NUM
| NUM

4 –(3 * 2)= -2
(4 – 3)* 2 = 2
| expr * NUM
| NUM

subs ➙ subs – mult
| mult
mult ➙ mult * NUM
| NUM
4 –(3 * 2)= -2

Show me the code
int subs() {
res = mult() ;
res = res – mult();
return res;
}
int mult() {
int res = num();
while (token == '*')
res = res * num();
return res;
}
int subs() {
res = mult() ;
res = res – mult();
return res;
}
int mult() {
int res = num();
while (token == '*')
res = res * num();
return res;
}
subs ➙ subs – mult
| mult
mult ➙ mult * NUM
| NUM

LL(1)
●
back to 1969
●
one token lookahead
●
no left-recursion

What is the plan?
✔ arithmetic expressions
✔ LL(1)
• a few common constructions from Java
• C++ ;)

The real deal
expr-stmt ➙ expr ; obj.method();
a = obj.field;
obj.method();
a = obj.field;

The real deal
expr-stmt ➙ expr ;
expr ➙ field-access
| method-call
| assignment
obj.method();
a = obj.field;
obj.method();
a = obj.field;

The real deal
| method-call
| assignment
field-access ➙ qualified-id
qualified-id ➙ qualified-id . id
| id
obj.method();
a = obj.field;
obj.method();
a = obj.field;

The real deal
| method-call
| assignment
| id
method-call ➙ qualified-id ()
obj.method();
a = obj.field;
obj.method();
a = obj.field;

The real deal
| method-call
| assignment
| id
assignment ➙ qualified-id = expr
obj.method();
a = obj.field;
obj.method();
a = obj.field;

int qualified_id() { /* easy */ }
int field_access() { /* easy */ }
int method_call() { /* easy */ }
int assignment() { /* easy */ }
int expr() {
// ???
}
int qualified_id() { /* easy */ }
int field_access() { /* easy */ }
int method_call() { /* easy */ }
int assignment() { /* easy */ }
int expr() {
// ???
}
Show me the code
| method-call
| assignment
| id
assignment ➙ qualified-id = expr
obj.method();
a = obj.field;
obj.method();
a = obj.field;

int expr() {
String id = qualified_id();
if (token == '(')
return method_call();
else if (token == '=')
return assignment();
else
return field_access();
}
int expr() {
String id = qualified_id();
if (token == '(')
return method_call();
else if (token == '=')
else
}
The LL(1) way
| method-call
| assignment
obj.method();
a = obj.field;
obj.method();
a = obj.field;

Reality
http://hg.openjdk.java.net/jdk8/jdk8/langtools/.../JavacParser.java

The better way
| method-call
| assignment
int expr() {
try { return field_access(); }
catch (RE e1) {
try { return method_call(); }
catch (RE e2) {
}
}
}
int expr() {
try { return field_access(); }
catch (RE e1) {
try { return method_call(); }
catch (RE e2) {
}
}
}
obj.method();
a = obj.field;
obj.method();
a = obj.field;

int expr() {
try { return method_call() ; }
catch (RE e1) {
try { return assignment(); }
catch (RE e2) {
}
}
}
int expr() {
try { return method_call() ; }
catch (RE e1) {
try { return assignment(); }
catch (RE e2) {
}
}
}
expr ➙ method-call
/ assignment
/ field-access
obj.method();
a = obj.field;
obj.method();
a = obj.field;

Parsing Expression Grammars
●
2002
●
ordered choice «/»
●
backtracking
●
no left-recursion

enum Nonterminals { EXPR, METHOD_CALL, … }
void grammar() {
rule(EXPR).is(
firstOf(
METHOD_CALL,
ASSIGNMENT,
FIELD_ACCESS));
}
enum Nonterminals { EXPR, METHOD_CALL, … }
void grammar() {
rule(EXPR).is(
firstOf(
METHOD_CALL,
ASSIGNMENT,
FIELD_ACCESS));
}
DSL for PEG
expr ➙ method-call
/ assignment
/ field-access
obj.method();
a = obj.field;
obj.method();
a = obj.field;

What is the plan?
✔ arithmetic expressions
✔ LL(1)
✔ common constructions from Java
✔ PEG
• C++ ;)

@YourTwitterHandle#DVXFR14{session hashtag} @dbolkensteyn @_godin_#parsing
Tea
Break

if (false)
if (true) System.out.println("foo");
else System.out.println("bar");
if (false)
Quiz

if (false)
if (false)
«Dangling else»
if-stmt ➙ IF (cond) stmt ELSE stmt
/ IF (cond) stmt

Java is awesome
(A)*B(A)*B

C++ all the pains of the world
int *B;
typedef int A;
(A)*B; // cast to type 'A' ('int' alias)
// of dereference of expression 'B'
int A, B;
(A)*B; // multiplication of 'A' and 'B'
// with redundant parenthesis around 'A'
int *B;
typedef int A;
(A)*B; // cast to type 'A' ('int' alias)
// of dereference of expression 'B'
int A, B;
(A)*B; // multiplication of 'A' and 'B'
// with redundant parenthesis around 'A'
Java is good, because it
was influenced by bad experience of C++ (A)*B(A)*B

rule(MUL_EXPR).is(
UNARY_EXPR, zeroOrMore('*', UNARY_EXPR));
rule(UNARY_EXPR).is(
firstOf(
sequence('(', TYPE_ID, ')', UNARY_EXPR),
PRIMARY,
sequence('*', UNARY_EXPR)));
rule(PRIMARY).is(
firstOf(
sequence('(', EXPR, ')'),
ID));
rule(MUL_EXPR).is(
firstOf(
PRIMARY,
rule(PRIMARY).is(
firstOf(
ID));
Hit the wall !
(A)*B(A)*B

rule(MUL_EXPR).is(
firstOf(
PRIMARY,
rule(PRIMARY).is(
firstOf(
ID));
rule(MUL_EXPR).is(
firstOf(
PRIMARY,
rule(PRIMARY).is(
firstOf(
ID));
Hit the wall !
(A)*B(A)*B

Dream
mul-expr ➙ mul-expr * unary-expr
| unary-expr
unary-expr ➙ ( type-id ) unary-expr
| * unary-expr
| primary
primary ➙ ( expr )
| id
(A)*B(A)*B

Generalized parsers
●
Earley (1968)
●
slow
●
GLR (1984)
●
complex

Chicken and egg problem
(A)*B
unary-expr mul-expr
(A) (A)*B
B*...
(A)*B(A)*B
| unary-expr
unary-expr ➙ ( type-id ) unary-expr
| * unary-expr
| primary
primary ➙ ( expr )
| id

Back to the future «dangling else»
if (…)
if (…) then-stmt
else else-stmt
if (…)
if (…) then-stmt
else else-stmt
outer-if
inner-if inner-if
then-stmt else-stmt
inner-if · else-stmt

GLL : How does it work ?
| unary-expr

Generalized LL
●
2010
●
no grammar left behind
(left-recursive, ambiguous)
●
simpler than GLR
●
syntactic ambiguities

Sum
m
ary

Summary
LL(1)
• trivial
• major grammar changes
• only good for arithmetic expressions
• on steroids as in JavaCC usable for
real languages

Summary
PEG
• trivial
• fewer grammar changes
• no ambiguities
• usable for real languages
• nice tools such as SSLR
• dead-end for C/C++

Summary
GLL
• any grammar
• relatively simple
• ambiguities
• reasonable performances
• the only clean choice for C/C++
• only «academic» tools for now... ;)

Summary
Hand-written
●
based on LL(1)
●
precise error-reporting
and recovery
●
best performances
●
maintainance hell

Q
&
A

The Art Of Parsing @ Devoxx France 2014

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à The Art Of Parsing @ Devoxx France 2014

Similaire à The Art Of Parsing @ Devoxx France 2014 (20)

Dernier

Dernier (20)

The Art Of Parsing @ Devoxx France 2014