Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Scaling Xtext
Lieven Lemiengre
Sigasi
● IDE for Hardware Description Languages
○ VHDL, (System)Verilog
● Using Xtext for 4 years
● Large user base
○ (com...
Our company goal
● Assist hardware designer
● High quality interactive front-end compiler
○ Instant feedback
■ parsing, se...
Visualisations
The challenge
● Pre-specified languages
● Large projects
○ > 250 KLOC is not uncommon
○ design + external libraries
○ big ...
Adopting Xtext
● Started with the early Xtext 2.0 snapshots
● Initial performance analysis
○ Clean build performance of a ...
Adopting Xtext
● Started with the early Xtext 2.0 snapshots
● Initial performance analysis
○ Clean build performance of a ...
● Xtext framework improvements
● Measure → analyze → improve or cheat
○ faster build
○ reduce memory usage
○ UI responsive...
Overview
● Analysing build performance
○ Analyze the build
■ Macro build measurements
■ Key performance points
● Reduce wo...
Analyzing builds: builder overview
Global
indexing
Linking
Validation
Custom
Validation
Global
index
Eclipse
resources
war...
Analyzing builds: builder overview
Global
indexing
Global
index
resource
descriptions
resource
changes
● Usage
○ Location ...
Analyzing builds: builder overview
Linking
Validation
● Usage
○ Determine IScope all cross references
○ Link cross referen...
Analyzing builds: builder overview
● Usage
○ Execute all custom validations
○ Creates errors / warnings
● Implementation
○...
Analyzing builds: builder overview
Global
indexing
Linking
Validation
Custom
Validation
Global
index
Eclipse
resources
war...
Analyzing builds: metrics
● For each build
○ # of files being build
○ timing: Global index, Linking, Validation, Individua...
Analyzing builds: resource loads
● Observation:
○ Most time spent in resource loads
○ Certain files are loaded multiple ti...
Memory pressure
Global
index
Resource
Set
Memory pressure?
● Size of EMF models
○ All the resources loaded during the buil...
Memory pressure: EMF models
Reduce EMF size
○ Watch out for inferred model
http://www.sigasi.com/content/view-complexity-y...
Memory pressure: Global index
In YourResourceDescriptionStrategy
○ Export foo, foo.rec, foo.rec.field1, foo.rec.field2
○ A...
Optimize loading
● What is resource load?
○ Parse
○ build EMF model & install EMF proxies
○ build Node model
Optimize loading
● What is resource load?
○ Parse
○ build EMF model & install EMF proxies
○ build Node model
● Parallelise...
Optimize loading
● What is resource load?
○ Parse
○ build EMF model & install EMF proxies
○ build Node model
● Parallelise...
Linking
Global
indexing
Linking
validation
Custom
Validation
Builder
Participants
● Language specific
○ VHDL vs Verilog
● ...
Custom Validation
Global
indexing
Linking
Validation
Custom
Validation
Builder
Participants
● Combine validations to avoid...
Track performance
● Nightly build
● log build times
VHDL linking in depth
History of VHDL linking
● 1st version
○ AbstractDeclarativeScopeProvider
○ best effort scoping
● 2nd...
VHDL linking in depth
foo(baz) <= bar(bak.f(“?”), (‘1’, 2))) + 1;
● Most elements in an expression are overloaded
○ subpro...
VHDL linking in depth
Xtext lazy linking
● good
○ declarative: only a few rules
○ fine-grained: can be good for performanc...
VHDL linking in depth
Batch/Eager linking
● good
○ simple top-down algorithm
○ natural fit for vhdl
○ well described in li...
VHDL linking in depth
Our hybrid approach
● Eager/batch linking of design units
○ Big files are partially scoped
○ Parent-...
Vhdl linking in depth
Conclusion
● Easier to implement, debug and optimize
● Type error reporting during linking
● Memory ...
UI responsiveness
● Measuring: detect a blocked UI thread
○ initially Svelto https://github.com/dragos/svelto
○ now our ow...
Lightweight Editor (fallback)
● Syntax-highlighting + markers
● For files > 1 MB
● Based on ContentTypes extension point
Two ContentTypes (based on file size)
<extension point="org.eclipse.core.contenttype.contentTypes">
<content-type ...
desc...
Future work
● Continuous process
● Cache global index info per resource?
● Linking without node model?
● StoredResources
Q/A
Come talk to us about...
● Documentation generation
● Fancy linking algorithms / type systems
● Graphical views
● Cross-la...
Scaling xtext - XtextCon 2015
Prochain SlideShare
Chargement dans…5
×

Scaling xtext - XtextCon 2015

1 678 vues

Publié le

Using Xtext for the first time is usually a very positive experience. Although Xtext is a complex generic framework, it is very easy to create your first Xtext-based editor, because of Xtext’s smart defaults and intuitive APIs. Even with minimal initial effort, the results are quite spectacular. Unfortunately the initial excitement often turns into disillusion as soon as you use your plugin on a big project.
Many development teams hit a performance wall as their plugin gets deployed and has to support larger projects. Internally, Xtext is a complex beast. The internals are carefully hidden from the user, but understanding them is critical to understand where the performance bottlenecks come from.
At Sigasi we have built commercial tool support for complex hardware description languages (VHDL, Verilog, SystemVerilog) using the Xtext framework. Our plugin needs to handle big industrial sized projects (>400k lines of code) that include large generated files (2 to 10 MB). To handle these kinds of projects we have developed a set of techniques over the last four years.
In this talk we will cover some performance critical pieces of the Xtext framework and evaluate what can be done to optimize it (think: parallel loading, caching, fast linking,…). We will also discuss some workarounds that can be used if nothing else works (light-weight editors, reducing the workload of the compiler).

Publié dans : Logiciels
  • Login to see the comments

Scaling xtext - XtextCon 2015

  1. 1. Scaling Xtext Lieven Lemiengre
  2. 2. Sigasi ● IDE for Hardware Description Languages ○ VHDL, (System)Verilog ● Using Xtext for 4 years ● Large user base ○ (commercial, free, students)
  3. 3. Our company goal ● Assist hardware designer ● High quality interactive front-end compiler ○ Instant feedback ■ parsing, semantic, linting, style checking ○ IDE Services ■ visualisations ■ design exploration ■ documentation generation ○ Integrate with ecosystem ■ other compilers, simulators, synthesizers
  4. 4. Visualisations
  5. 5. The challenge ● Pre-specified languages ● Large projects ○ > 250 KLOC is not uncommon ○ design + external libraries ○ big files ■ some libraries are distributed as 1 file ■ generated hardware cores ●
  6. 6. Adopting Xtext ● Started with the early Xtext 2.0 snapshots ● Initial performance analysis ○ Clean build performance of a big project (330k LOC) ■ > 20 minutes ■ > 2 GB ○ Editing big files (> 1 MB) ■ unusable
  7. 7. Adopting Xtext ● Started with the early Xtext 2.0 snapshots ● Initial performance analysis ○ Clean build performance of a big project (330k LOC) ■ > 20 minutes → < 1 min ■ > 2 GB → ~ 1 GB memory ○ Editing big files (> 1 MB) ■ unusable → usable with reduced editor
  8. 8. ● Xtext framework improvements ● Measure → analyze → improve or cheat ○ faster build ○ reduce memory usage ○ UI responsiveness Improving performance
  9. 9. Overview ● Analysing build performance ○ Analyze the build ■ Macro build measurements ■ Key performance points ● Reduce workload ● Parallelize the build ○ Tracking performance ○ VHDL linking in depth ● Analyzing UI issues ○ Monitoring the UI thread ○ Saveguards
  10. 10. Analyzing builds: builder overview Global indexing Linking Validation Custom Validation Global index Eclipse resources warnings errors resource descriptions Builder Participants resource changes ?
  11. 11. Analyzing builds: builder overview Global indexing Global index resource descriptions resource changes ● Usage ○ Location of exported declarations ○ Incremental compilation ● Implementation ○ IResourceDescriptionsStrategy ○ Default: all declarations ○ Customize! ○ Runs before linking! ● IResourceDescriptions & IEObjectDescriptions ○ Always in memory ○ Persisted tot disk @ shutdown
  12. 12. Analyzing builds: builder overview Linking Validation ● Usage ○ Determine IScope all cross references ○ Link cross reference or create linking error ● Implementation ○ ILinkingService, IScopeProvider, LazyLinkingResource ○ Requires global index for global scope ○ Direct link or link to global scope ○ Linking may trigger linking and resource loading linking errors Eclipse resources
  13. 13. Analyzing builds: builder overview ● Usage ○ Execute all custom validations ○ Creates errors / warnings ● Implementation ○ AbstractDeclarativeValidator ○ Execute validations using reflection ○ Works against linked model ○ May trigger linking & resource loading Linking Validation errors warnings Eclipse resources
  14. 14. Analyzing builds: builder overview Global indexing Linking Validation Custom Validation Global index Eclipse resources warnings errors resource descriptions Builder Participants resource changes ? ● iterations ? ● order ?
  15. 15. Analyzing builds: metrics ● For each build ○ # of files being build ○ timing: Global index, Linking, Validation, Individual builder participants ● Instrument by overriding ClusteringBuilderState & XtextBuilder ● Example: Building 134 resources, timing: { global index=1806, linking=378, validation=823, totalLinkingAndValidation=1364 }
  16. 16. Analyzing builds: resource loads ● Observation: ○ Most time spent in resource loads ○ Certain files are loaded multiple times?! ● Solutions ○ Reduce memory pressure ○ Make loading faster Global indexing Linking validation Custom Validation Builder Participants resources LOAD POTENTIAL RELOADS POTENTIAL RELOADS
  17. 17. Memory pressure Global index Resource Set Memory pressure? ● Size of EMF models ○ All the resources loaded during the build ● Size of global index ○ Always loaded ○ Depends on number of open projects
  18. 18. Memory pressure: EMF models Reduce EMF size ○ Watch out for inferred model http://www.sigasi.com/content/view-complexity-your-xtext-ecore-model ○ Avoid ■ Emf classes with just one list of things ● ListOfThings : ‘(‘ things+=Thing (‘,’ things+=Thing)* ‘)’ ● class ListOfThings { contains Thing[] things } ■ Often unused fields ○ Code duplication in grammar vs efficient model ○ Fine-grained control with Xcore model
  19. 19. Memory pressure: Global index In YourResourceDescriptionStrategy ○ Export foo, foo.rec, foo.rec.field1, foo.rec.field2 ○ Add user-data: someType & anotherType ○ To reduce memory usage: don’t export child elements ■ export foo.rec + hash of fields ■ export foo + hash of contents of foo ■ can’t link these elements without loading anymore package foo is record rec is field1 : someType; field2 : anotherType(X downto Y); end; end;
  20. 20. Optimize loading ● What is resource load? ○ Parse ○ build EMF model & install EMF proxies ○ build Node model
  21. 21. Optimize loading ● What is resource load? ○ Parse ○ build EMF model & install EMF proxies ○ build Node model ● Parallelise ○ parse multiple files simultaneously ○ ~3 time faster loads on 4 core machine ○ only loading, not linking
  22. 22. Optimize loading ● What is resource load? ○ Parse ○ build EMF model & install EMF proxies ○ build Node model ● Parallelise ○ parse multiple files simultaneously ○ ~3 time faster loads on 4 core machine ○ only loading, not linking ● Cache ○ serialize EMF and Node model in a cache ○ originally 3-4 time faster loads ○ now 1.5x (no backtracking, simplified grammar)
  23. 23. Linking Global indexing Linking validation Custom Validation Builder Participants ● Language specific ○ VHDL vs Verilog ● Avoiding linking ○ library files, only linked when used in user-code ● Many iterations ○ lazy linking vs eager linking ○ From 40% of build time to 20%
  24. 24. Custom Validation Global indexing Linking Validation Custom Validation Builder Participants ● Combine validations to avoid model traversals ● Local analysis, do global validations moved into builder participant ● Avoid validation ○ disabled validations ○ libraries: errors & warnings are suppressed anyway ● Monitor
  25. 25. Track performance ● Nightly build ● log build times
  26. 26. VHDL linking in depth History of VHDL linking ● 1st version ○ AbstractDeclarativeScopeProvider ○ best effort scoping ● 2nd version ○ removed reflection ● 3rd version ○ special rule-based internal java dsl ○ first attempt to be 100% correct ● 4rth version ○ batch/eager linking ○ type errors
  27. 27. VHDL linking in depth foo(baz) <= bar(bak.f(“?”), (‘1’, 2))) + 1; ● Most elements in an expression are overloaded ○ subprograms, literals, enumliterals ● foo(baz) ○ 9 kinds of meanings ○ 4 of them can have subprogram overloading ● overloading includes return type ● overload resolution is very hard ○ you have to find 1 unambiguous solution ○ resolve all cross-references together
  28. 28. VHDL linking in depth Xtext lazy linking ● good ○ declarative: only a few rules ○ fine-grained: can be good for performance ○ re-use: scoping, auto-complete, serialisation ● bad ○ hard to debug ■ can call itself ■ lots of caching ■ indirection, huge stack traces ○ performance ■ build context for every cross-reference
  29. 29. VHDL linking in depth Batch/Eager linking ● good ○ simple top-down algorithm ○ natural fit for vhdl ○ well described in literature ● bad ○ resolve 1 reference = resolve all references ○ a lot of extra xtext customisation ■ auto-complete & serialization? ■ linking errors?
  30. 30. VHDL linking in depth Our hybrid approach ● Eager/batch linking of design units ○ Big files are partially scoped ○ Parent-scope of a design unit is the global scope ○ Local scoping is executed eagerly ● Global scope ○ Import declarations of other design units ○ Only query is find design unit x.y ■ load resource of x.y ■ create an ‘ExternalScope’ & cache it ● Always load dependent resources ○ needed for validation, hovers, highlighting anyway
  31. 31. Vhdl linking in depth Conclusion ● Easier to implement, debug and optimize ● Type error reporting during linking ● Memory intensive? ○ Every dependency is loaded ○ OK in practice for VHDL ● A lot of xtext customisation! ○ A lot of classes are affected ○ Forward compatible?
  32. 32. UI responsiveness ● Measuring: detect a blocked UI thread ○ initially Svelto https://github.com/dragos/svelto ○ now our own method & logging ○ Eclipse Mars ● Improvements ○ UI is for drawing only! ○ Make sure everything is cancellable ● Safeguards ○ certain services should never be executed on the UI thread => check & log
  33. 33. Lightweight Editor (fallback) ● Syntax-highlighting + markers ● For files > 1 MB ● Based on ContentTypes extension point
  34. 34. Two ContentTypes (based on file size) <extension point="org.eclipse.core.contenttype.contentTypes"> <content-type ... describer="com.sigasi...FullVhdlContentDescriber" name="VHDL editor" <describer class="...FullVhdlContentDescriber" /> </content-type> <content-type ... describer="com.sigasi....LightweightVhdlContentDescriber" name="Lightweight VHDL editor" <describer class="...LightweightVhdlContentDescriber" /> </content-type> </extension>
  35. 35. Future work ● Continuous process ● Cache global index info per resource? ● Linking without node model? ● StoredResources
  36. 36. Q/A
  37. 37. Come talk to us about... ● Documentation generation ● Fancy linking algorithms / type systems ● Graphical views ● Cross-language support ● Testing Xtext-plugins ● Lexical macros ● Manage large amount of validations ● ...

×