Successfully reported this slideshow.
The following presentation is provided under DHMB license 
(Do Not Hurt My Brain). 
Lecturer is not responsible for the fi...
Software engineers 
united we stand 
divided we fall
Patterns for JVM languages 
Jarosław Pałka
about me 
work://chief_architect@lumesse 
owner://symentis.pl 
twitter://j_palka 
blog://geekyprimitives.wordpress.com 
sc...
Pet project
Because having pet projects is like solving crosswords 
everyday 
Or going to a gym 
Helps your brain to stay in shape
JVM 
is 
the 
new 
assembler
Translator
source code 
↓ 
target language source code 
↓ 
machine code || interpreter
Interpreter
source code 
↓ 
execution results
Compiler
source code 
↓ 
machine code
So how does it all work?
Diving into the stream
The ALL(*) lexer and parser generator 
Which supports multiple target languages
source code 
↓lexer 
token stream 
↓parser 
IR
Intermediate Representation 
visitor 
listener 
tree
Syntax tree
local y = 1 
x = 1 + y
fragment 
DIGIT 
: 
[0-9] 
; 
fragment 
LETTER 
: 
[a-zA-Z] 
; 
String 
: 
''' .*? ''' 
| '"' .*? '"' 
; 
Number 
: 
DIGIT...
[@0,[0..1)='x',<49>] 
[@1,[2..3)='=',<36>] 
[@2,[4..5)='1',<48>] 
[@3,[6..7)='+',<33>] 
[@4,[8..15)='"Hello"',<47>] 
[@5,[...
varlist returns [Varlist result] 
: 
var 
{$result = createVarlist($var.result);} 
( 
',' var 
{$result = createVarlist($r...
beware of left recursion
translator 
interpreter 
compiler
Interpreter pattern 
Non terminal nodes 
Terminal nodes
package pl.symentis.lua.grammar; 
import pl.symentis.lua.runtime.Context; 
public interface Statement { 
void execute(Cont...
package pl.symentis.lua.grammar; 
Import pl.symentis.lua.runtime.Context; 
public interface Expression<T> { 
T evaluate(Co...
package pl.symentis.lua.runtime; 
public interface Context { 
VariableRef resolveVariable(String name); 
Scope enterScope(...
package pl.symentis.lua.runtime; 
public interface Scope { 
Scope getOuterScope(); 
VariableRef resolveVariable(String nam...
Into the forest of syntax tree
x = 1
public class Literal implements ExpressionNode { 
public final LuaType value; 
public Literal(LuaType value) { 
super(); 
...
public class VariableName extends Var { 
public final String name; 
public VariableName(String name) { 
this.name = name; ...
public class Assignment implements StatementNode { 
public final Varlist varlist; 
public final Explist explist; 
public A...
public void visit(Assignment assignment) { 
InterpreterExpressionVisitor varlist = createExpressionVisitor(); 
assignment....
public void visit(Literal literal) { 
result = literal.value; 
} 
public void visit(VariableName variableName) { 
result =...
translator 
interpreter 
compiler
compiler → byte code 
org.ow2.asm:asm:jar 
me.qmx.jitescript:jitescript:jar
JiteClass jiteClass = new JiteClass(classname, new String[] { p(Script.class) }); 
jiteClass.defineDefaultConstructor(); 
...
JiteClass jiteClass = new JiteClass(classname, new String[] { p(Script.class) }); 
jiteClass.defineDefaultConstructor(); 
...
JiteClass jiteClass = new JiteClass(classname, new String[] { p(Script.class) }); 
jiteClass.defineDefaultConstructor(); 
...
x = 1
public void visit(Assignment assignment) { 
Explist explist = assignment.explist; 
Varlist varlist = assignment.varlist; 
...
public void visit(Literal literal) { 
Object object = literal.value.asObject(); 
codeBlock.ldc(object); 
// box object int...
When writing bytecode compiler it helps to think about Java code 
But at the end you need to think like stack machine :)
.method public execute : (Lpl/symentis/lua4j/runtime/Context;)V 
; method code size: 21 bytes 
.limit stack 2 
.limit loca...
Symbol table 
maps variable local to its symbolic name
0xBA 
invokedynamic
print("Welcome","to","Jokerconf",2014)
Var var = functionCall.var; 
CompilerExpressionVisitor visitor = new CompilerExpressionVisitor(codeBlock, 
symbolTable); 
...
static final Handle HANDLE = new Handle( 
CodeBlock.H_INVOKESTATIC, 
p(Lua4JDynamicBootstrap.class), 
"bootstrap", 
sig(Ca...
public static CallSite bootstrap( 
MethodHandles.Lookup lookup, 
String name, 
MethodType type) throws Throwable { 
Contex...
Things to consider 
Type coercion 
Symbol table and scope 
Function handles and scope 
Global scope
https://github.com/headius/invokebinder 
A Java DSL for binding method handles forward, 
rather than backward 
https://git...
What's next? 
graal + truffle 
dynamic compiler + AST interpreter
Books
Thanks, 
stay cool, 
dynamically linked 
and...
Let the eternal stream of bytecode 
flow through your body
Q&A
Patterns for JVM languages JokerConf
Patterns for JVM languages JokerConf
Patterns for JVM languages JokerConf
Patterns for JVM languages JokerConf
Patterns for JVM languages JokerConf
Patterns for JVM languages JokerConf
Patterns for JVM languages JokerConf
Prochain SlideShare
Chargement dans…5
×

Patterns for JVM languages JokerConf

982 vues

Publié le

In recent years we have seen explosion of languages which run on Java Virtual Machine. We also have seen existing languages getting their implementations being rewritten to JVM. With all of the above we have seen rapid development of tools like parsers, bytecode generators and such, even inside JVM we saw initiatives like Da Vinci Machine Project, which led to invoke dynamic in JDK 7 and recent development of Graal and Truffle projects.

Is it really hard to write new programming language running on JVM? Even if you are not going to write your own I think it is worth to understand how your favorite language runs undercover, how early decisions can impact language extensibility and performance, what JVM itself and JVM ecosystem has to offer to language implementors.

During the session I will try to get you familiar with options you have when choosing parsers and byte code manipulation libraries. which language implementation to consider, how to test and tune your "new baby". Will you be able after this session to develop new and shiny language, packed with killer features language? No. But for sure you will understand difference between lexers and parsers, how bytecode works, why invoke dynamic and Graal and Truffle are so important to the future of JVM platform. Will we have time to write simple, compiled language?

Publié dans : Logiciels
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

Patterns for JVM languages JokerConf

  1. 1. The following presentation is provided under DHMB license (Do Not Hurt My Brain). Lecturer is not responsible for the financial and moral damages resulting from taking too seriously the content of the presentation. Including permanent damage to neurons, reduction of neurotransmitter activity at the molecular level and group dismissal.
  2. 2. Software engineers united we stand divided we fall
  3. 3. Patterns for JVM languages Jarosław Pałka
  4. 4. about me work://chief_architect@lumesse owner://symentis.pl twitter://j_palka blog://geekyprimitives.wordpress.com scm:bitbucket://kcrimson scm:github://kcrimson
  5. 5. Pet project
  6. 6. Because having pet projects is like solving crosswords everyday Or going to a gym Helps your brain to stay in shape
  7. 7. JVM is the new assembler
  8. 8. Translator
  9. 9. source code ↓ target language source code ↓ machine code || interpreter
  10. 10. Interpreter
  11. 11. source code ↓ execution results
  12. 12. Compiler
  13. 13. source code ↓ machine code
  14. 14. So how does it all work?
  15. 15. Diving into the stream
  16. 16. The ALL(*) lexer and parser generator Which supports multiple target languages
  17. 17. source code ↓lexer token stream ↓parser IR
  18. 18. Intermediate Representation visitor listener tree
  19. 19. Syntax tree
  20. 20. local y = 1 x = 1 + y
  21. 21. fragment DIGIT : [0-9] ; fragment LETTER : [a-zA-Z] ; String : ''' .*? ''' | '"' .*? '"' ; Number : DIGIT+ ; Name : ( '_' | LETTER )( '_' | LETTER | DIGIT )* ; COMMENT : '--' .*? 'n' -> skip ; NL : 'r'? 'n' -> skip ; WS : [ tf]+ -> skip ;
  22. 22. [@0,[0..1)='x',<49>] [@1,[2..3)='=',<36>] [@2,[4..5)='1',<48>] [@3,[6..7)='+',<33>] [@4,[8..15)='"Hello"',<47>] [@5,[15..16)=';',<37>] [@6,[17..17)='',<-1=EOF>]
  23. 23. varlist returns [Varlist result] : var {$result = createVarlist($var.result);} ( ',' var {$result = createVarlist($result,$var.result);} )* ; // var returns [Var result] : Name { $result = createVariableName($Name.text); } | prefixexp '[' exp ']' | prefixexp '.' Name ;
  24. 24. beware of left recursion
  25. 25. translator interpreter compiler
  26. 26. Interpreter pattern Non terminal nodes Terminal nodes
  27. 27. package pl.symentis.lua.grammar; import pl.symentis.lua.runtime.Context; public interface Statement { void execute(Context context); }
  28. 28. package pl.symentis.lua.grammar; Import pl.symentis.lua.runtime.Context; public interface Expression<T> { T evaluate(Context ctx); }
  29. 29. package pl.symentis.lua.runtime; public interface Context { VariableRef resolveVariable(String name); Scope enterScope(); Scope exitScope(); }
  30. 30. package pl.symentis.lua.runtime; public interface Scope { Scope getOuterScope(); VariableRef resolveVariable(String name); void bindVariable(String name, Object value); }
  31. 31. Into the forest of syntax tree
  32. 32. x = 1
  33. 33. public class Literal implements ExpressionNode { public final LuaType value; public Literal(LuaType value) { super(); this.value = value; } @Override public void accept(ExpressionVisitor visitor) { visitor.visit(this); } }
  34. 34. public class VariableName extends Var { public final String name; public VariableName(String name) { this.name = name; } @Override public void accept(ExpressionVisitor visitor) { visitor.visit(this); } }
  35. 35. public class Assignment implements StatementNode { public final Varlist varlist; public final Explist explist; public Assignment(Varlist varlist, Explist explist) { super(); this.varlist = varlist; this.explist = explist; } @Override public void accept(StatementVisitor visitor) { visitor.visit(this); } }
  36. 36. public void visit(Assignment assignment) { InterpreterExpressionVisitor varlist = createExpressionVisitor(); assignment.varlist.accept(varlist); InterpreterExpressionVisitor explist = createExpressionVisitor(); assignment.explist.accept(explist); Iterator<ExpressionNode> iterator = ((List<ExpressionNode>) explist.result()).iterator(); List<VariableRef> vars = (List<VariableRef>) varlist.result(); for (VariableRef variableRef : vars) { if (iterator.hasNext()) { ExpressionNode node = iterator.next(); InterpreterExpressionVisitor exp = createExpressionVisitor(); node.accept(exp); variableRef.set(exp.result()); } else { variableRef.set(LuaNil.Nil); } } }
  37. 37. public void visit(Literal literal) { result = literal.value; } public void visit(VariableName variableName) { result = context.resolveVariable(variableName.name); } public void visit(Varlist varlist) { List<VariableRef> refs = new ArrayList<>(); for (Var var : varlist.vars) { var.accept(this); refs.add((VariableRef) result()); } result = refs; }
  38. 38. translator interpreter compiler
  39. 39. compiler → byte code org.ow2.asm:asm:jar me.qmx.jitescript:jitescript:jar
  40. 40. JiteClass jiteClass = new JiteClass(classname, new String[] { p(Script.class) }); jiteClass.defineDefaultConstructor(); CodeBlock codeBlock = newCodeBlock(); // bind context into current runtime codeBlock.aload(1); codeBlock.invokestatic(p(RuntimeContext.class), "enterContext", sig(void.class, Context.class)); // parse file and start to walk through AST visitor StatementNode chunk = parseScript(readFileToString(script)); // generate byte code chunk.accept(new CompilerStatementVisitor(codeBlock, new HashSymbolTable())); codeBlock.voidreturn(); jiteClass.defineMethod("execute", JiteClass.ACC_PUBLIC, sig(void.class, new Class[] { Context.class }), codeBlock); return jiteClass.toBytes(JDKVersion.V1_8); }
  41. 41. JiteClass jiteClass = new JiteClass(classname, new String[] { p(Script.class) }); jiteClass.defineDefaultConstructor(); CodeBlock codeBlock = newCodeBlock(); // bind context into current runtime codeBlock.aload(1); codeBlock.invokestatic(p(RuntimeContext.class), "enterContext", sig(void.class, Context.class)); // parse file and start to walk through AST visitor StatementNode chunk = parseScript(readFileToString(script)); // generate byte code chunk.accept(new CompilerStatementVisitor(codeBlock, new HashSymbolTable())); codeBlock.voidreturn(); jiteClass.defineMethod("execute", JiteClass.ACC_PUBLIC, sig(void.class, new Class[] { Context.class }), codeBlock); return jiteClass.toBytes(JDKVersion.V1_8); }
  42. 42. JiteClass jiteClass = new JiteClass(classname, new String[] { p(Script.class) }); jiteClass.defineDefaultConstructor(); CodeBlock codeBlock = newCodeBlock(); // bind context into current runtime codeBlock.aload(1); codeBlock.invokestatic(p(RuntimeContext.class), "enterContext", sig(void.class, Context.class)); // parse file and start to walk through AST visitor StatementNode chunk = parseScript(readFileToString(script)); // generate byte code chunk.accept(new CompilerStatementVisitor(codeBlock, new HashSymbolTable())); codeBlock.voidreturn(); jiteClass.defineMethod("execute", JiteClass.ACC_PUBLIC, sig(void.class, new Class[] { Context.class }), codeBlock); return jiteClass.toBytes(JDKVersion.V1_8); }
  43. 43. x = 1
  44. 44. public void visit(Assignment assignment) { Explist explist = assignment.explist; Varlist varlist = assignment.varlist; Iterator<Var> vars = varlist.vars.iterator(); for (ExpressionNode exp : explist.expressions) { exp.accept(new CompilerExpressionVisitor(codeBlock, symbolTable)); CompilerExpressionVisitor visitor = new CompilerExpressionVisitor( codeBlock, symbolTable); vars.next().accept(visitor); String varName = (String) visitor.result(); codeBlock.astore(symbolTable.index(varName)); } }
  45. 45. public void visit(Literal literal) { Object object = literal.value.asObject(); codeBlock.ldc(object); // box object into Lua type if (object instanceof Integer) { codeBlock.invokestatic(p(Integer.class), "valueOf", sig(Integer.class, int.class)); } else if (object instanceof Double) { codeBlock.invokestatic(p(Double.class), "valueOf", sig(Double.class, double.class)); } codeBlock.invokestatic(p(LuaTypes.class), "valueOf", sig(LuaType.class, Object.class)); } public void visit(VariableName variableName) { result = variableName.name; }
  46. 46. When writing bytecode compiler it helps to think about Java code But at the end you need to think like stack machine :)
  47. 47. .method public execute : (Lpl/symentis/lua4j/runtime/Context;)V ; method code size: 21 bytes .limit stack 2 .limit locals 3 aload_1 invokestatic pl/symentis/lua4j/compiler/RuntimeContext enterContext (Lpl/symentis/lua4j/runtime/Context;)V ldc2_w 1.0 invokestatic java/lang/Double valueOf (D)Ljava/lang/Double; invokestatic pl/symentis/lua4j/types/LuaTypes valueOf [_28] astore_2 aload_2 return .end method
  48. 48. Symbol table maps variable local to its symbolic name
  49. 49. 0xBA invokedynamic
  50. 50. print("Welcome","to","Jokerconf",2014)
  51. 51. Var var = functionCall.var; CompilerExpressionVisitor visitor = new CompilerExpressionVisitor(codeBlock, symbolTable); var.accept(visitor); String functionName = (String) visitor.result(); Args args = functionCall.args; visitor = new CompilerExpressionVisitor(codeBlock, symbolTable); args.accept(visitor); Integer argNumber = (Integer) visitor.result(); Class<?>[] paramTypeArr = new Class[argNumber]; fill(paramTypeArr, LuaType.class); codeBlock.invokedynamic(functionName, sig(void.class, paramTypeArr), Lua4JDynamicBootstrap.HANDLE, new Object[] {});
  52. 52. static final Handle HANDLE = new Handle( CodeBlock.H_INVOKESTATIC, p(Lua4JDynamicBootstrap.class), "bootstrap", sig(CallSite.class, MethodHandles.Lookup.class, String.class, MethodType.class));
  53. 53. public static CallSite bootstrap( MethodHandles.Lookup lookup, String name, MethodType type) throws Throwable { Context context = currentContext(); MethodHandle target=lookup.findStatic(BasicFunctions.class,name, methodType( LuaType.class, Context.class, LuaType[].class)); target = MethodHandles.insertArguments(handler, 0, context) .asCollector(LuaType[].class, type.parameterCount()); return new ConstantCallSite(target.asType(type)); }
  54. 54. Things to consider Type coercion Symbol table and scope Function handles and scope Global scope
  55. 55. https://github.com/headius/invokebinder A Java DSL for binding method handles forward, rather than backward https://github.com/projectodd/rephract Another Java DSL for invokedynamic
  56. 56. What's next? graal + truffle dynamic compiler + AST interpreter
  57. 57. Books
  58. 58. Thanks, stay cool, dynamically linked and...
  59. 59. Let the eternal stream of bytecode flow through your body
  60. 60. Q&A

×