ASTs are an incredibly powerful tool for understanding and manipulating JavaScript. We'll explore this topic by looking at examples from ESLint, a pluggable static analysis tool, and Browserify, a client-side module bundler. Through these examples we'll see how ASTs can be great for analyzing and even for modifying your JavaScript. This talk should be interesting to anyone that regularly builds apps in JavaScript either on the client-side or on the server-side.
2. Our Basic Plan
1. High-level overview
2. Static Analysis with ASTs
3. Transforming and refactoring
4. A quick look at the Mozilla Parser API
(de-facto standard AST format)
11. Fix a bug. Add a unit test.
Fix a similar bug…
12. Write some really
solid static analysis.
Never write that same
type of bug again.
13. function loadUser(req, res, next) {
User.loadUser(function(err, user) {
req.session.user = user;
next();
});
}
Bad Example
We forgot to handle the error!
14.
15. handle-callback-err
1. Each time a function is declared check
if there is an error* parameter
If so set a count to 0;
Increment count when error is used
At the end of the function warn when
count is empty
* the parameter name can be defined by the user
17. History Lesson
• 1995: JavaScript
• 2002: JSLint started by Douglas Crockford
• 2011: JSHint comes out as a fork of JSLint.
Esprima AST parser released.
• 2012: plato, escomplex, complexity-report
• 2013: Nicholoas Zakas releases ESLint. Marat
Dulin releases JSCS.
23. no-loop-func
function checkForLoops(node) {
var ancestors = context.getAncestors();
if (ancestors.some(function(ancestor) {
return ancestor.type === "ForStatement" ||
ancestor.type === "WhileStatement" ||
ancestor.type === "DoWhileStatement";
})) {
context.report(node, "Don't make functions within a loop");
}
}
return {
"FunctionExpression": checkForLoops,
"FunctionDeclaration": checkForLoops
};
24. max-params
var numParams = context.options[0] || 3;
function checkParams(node) {
if (node.params.length > numParams) {
context.report(node, "This function has too many parameters
({{count}}). Maximum allowed is {{max}}.", {
count: node.params.length,
max: numParams
});
}
}
return {
“FunctionDeclaration”: checkParams,
“FunctionExpression”: checkParams
}
25. no-jquery
function isjQuery(name) {
return name === '$' || name === 'jquery' || name === 'jQuery';
}
return {
“CallExpression”: function(node) {
var name = node.callee && node.callee.name;
if (isjQuery(name)) {
context.report(node, 'Please avoid using jQuery here.’);
}
}
}
27. Other Areas for Static
Analysis
Code complexity and visualization is another
area where static analysis is really useful. Plato is
an exciting start, but I believe there are tons of
more interesting things that can be done in this
area.
28. Recap
• Static Analysis can help you catch real bugs and keep
your code maintainable
• ESLint and JSCS both use ASTs for inspecting your
code to make it easy to cleanly to add new rules
• Static analysis can also help you manage your code
complexity as well
• What exactly does a for loop sound like?
34. Tools like falafel and recast give
you an API to manipulate an AST
and then convert that back into
source code.
35. Two Types of AST
Transformations
Regenerative
Regenerate the full file from the AST. Often losing
comments and non-essential formatting. Fine for code
not read by humans (i.e. browserify transforms).
Partial-source transformation
Regenerate only the parts of the source that have
changed based on the AST modifications. Nicer for
one-time changes in source.
38. 4 Steps
1. Buffer up the stream of source code
2. Convert the source into an AST
3. Transform the AST
4. Re-generate and output the source
39. Step 1
Use through to grab the source code
var through = require(‘through');
var buffer = [];
return through(function write(data) {
buffer.push(data);
}, function end () {
var source = buffer.join(‘’);
});
40. Step 2
Use falafel to transform create an AST
var falafel = require(‘falafel’);
function end () {
var source = buffer.join(‘’);
var out = falafel(source, parse).toString();
}
41. Step 3
function parse(node) {
if (node.type === 'Identifier' &&
node.value === ‘ui’) {
node.update('browserify');
}
}
Use falafel to transform the AST
42. Step 4
Stream the source with through and close the stream
function end () {
var source = buffer.join(‘’);
var out = falafel(source, parse).toString();
this.queue(out);
this.queue(null); // end the stream
}
43. var through = require('through');
var falafel = require('falafel');
module.exports = function() {
var buffer = [];
return through(function write(data) {
buffer.push(data);
}, function end() {
var source = buffer.join('n');
var out = falafel(source, parse).toString();
this.queue(out);
this.queue(null); // close the stream
});
};
function parse(node) {
if (node.type === 'Identifier' &&
node.name === 'ui') {
node.update('browserify');
}
}
53. A Basic Map/Filter
var a = [1, 2, 3];
var b = a.filter(function(n) {
return n > 1;
}).map(function(k) {
return k * 2;
});
54. Faster Like This
var a = [1, 2, 3];
var b = [];
for (var i = 0; i < a.length; i++) {
if (a[i] > 1) {
b.push(a[i] * 2);
}
}
55.
56. A Basic Recast Script
var recast = require(‘recast’);
var code = fs.readFileSync(‘code.js', 'utf-8');
var ast = recast.parse(code);
var faster = transform(ast);
var output = recast.print(faster).code;
57. function transform(ast) {
var transformedAST = new MapFilterEater({
body: ast.program.body
}).visit(ast);
return transformedAST;
}
var Visitor = recast.Visitor;
var MapFilterEater = Visitor.extend({
init: function(options) {},
visitForStatement: function(ast) {},
visitIfStatement: function(ast) {},
visitCallExpression: function(ast) {},
visitVariableDeclarator: function(ast) {}
});
58. How Does it Work?
1. Move the right side of the b declaration into a for loop
2. Set b = []
3. Place the .filter() contents inside of an if statement
4. Unwrap the .map contents and .push() them into b
5. Replace all of the local counters with a[_i]
59.
60.
61.
62.
63. And Voila….
var a = [1, 2, 3];
var b = [];
for (var i = 0; i < a.length; i++) {
if (a[i] > 1) {
b.push(a[i] * 2);
}
}
80. • When debugging console.log(ast) will not print a
large nested AST properly. Instead you can use
util.inspect:
var util = require('util');
var tree = util.inspect(ast, { depth: null });
console.log(tree);
• When transforming code start with the AST you
want and then work backward.
• Often this means pasting code using the Esprima
online visualization tool or just outputting the trees
into JS files and manually diffing them.
81. Oftentimes it helps to print out the code
representation of a single node.
In recast you can do:
var source = recast.prettyPrint(ast, {
tabWidth: 2 }).code;
In ESLint you can get the current node with:
var source = context.getSource(node)
99. Tools like falafel and recast give
you an API to manipulate an AST
and then convert that back into
source code.
Notes de l'éditeur
Hi I’m Jamund. I work at PayPal doing node stuff and I have an environmental planning degree from the university of washington. Which hopefully qualifies me to talk you about trees today.
Before I get started, how many of you have used these tools before and know what i’m talking about? Cool, so the rest of you. please stick around. It’s to ally worth it and I’m sure these people can attest to that.
Instead of having to worry about string parsing and regexes and all that nonsense.
Here’s an example. This is using the mozilla parser API format used by esprima and respresents the following code.
This single statement creates a tree with 7 different nodes!!!
Each item in this tree is a node of one of several types. We’ll discuss these types in more depth later.
My goals is to help my team not introduce bugs in their code.
Anyone know what’w wrong with this with?
So we just built an ESLint rule that would ensure we never made that same mistake again! And it worked! We haven’t seen that type of bug crop up in our codebase.
I love static analysis and that example proves that it can do a lot more than complain about your formatting.
Essentially since 2011 ever new major static analysis tool has been based on esprima.
Mostly because of this…If I want to add a rule. I can put it in its own file/function. Adding that handle-callback-err thing was no problem at all. Even if ESLint didn’t take it at this point I can just NPM install rules and use them! It rocks.
Mixes the parser with the linter. It’s a great tool. But it’s not as easy to extend.
Everytime there’s a member expression, which is thing.thing you just check to see if the name is the console, in which case it complains…..super simple..
But who cares. You can do this with “grep” or something. Let’s do something more fun.
Pretty awesome. So every time you have a function expression or function declaration we check to see if has an ancestor that’s a loop. Pretty dang slick.
Here’s a custom one we have that we only apply to certain parts of the code-base. For example in our backbone models, to prevent people from using Backbone.sync.
Currently working with our globalization team on rules to help us avoid mistakes in handling dates, and phone numbers worldwide. and much more.
For example: what if you could turn your AST into a series of beats and you could listen to your code to help determine how complicated it is. Short song good. Consistent tones good. Too many high notes bad? I don’t know. Think about that.
We’ll use both of these in the following examples:
How many of you have used browserify?
Say we want to change this to this. Dynamically. Easy.
Through takes 2 callbacks. The first lets you buffer the stream data the 2nd is when you get everything.
This is our parse() function. We use falafel’s node.update to transform the node.
altogether it looks like this
And it works. And you can add them with npm and they just work.
Marihn haverbeke, author of Acorn.js parser and a genius. Asks this question. The answer is no, but don’t let that stop you from writing .forEach() and .map() and .filter(). CODE IS FOR HUMANS
https://github.com/xjamundx/perfify-recast/
But we can do hard things.
But we write code for humans, so we’ll let the transform take care of this.
https://github.com/benjamn/recast uses “partial source transformation” to safely
With browserify when you apply the transforms it rewrites the whole thing. You generally lose comments and stuff that isn’t really needed to form the AST. Recast (and some other tools like falafel) are nice for one-time refactoring because the employ techniques to limit the amount of code that is touched during the re-generation phase.
I can show you the full source it’s online…. https://github.com/xjamundx/perfify-recast/blob/master/index.js
There’s also a Facebook fork of Esprima that supports ES6, so it’s getting some support as well. I assume it will eventually bubble back up to the main branch…hopefully?
You probably just need esprima.
http://esprima.org/demo/parse.html#
It’s also called the SpiderMonkey API. Are th
This single statement creates a tree with 7 different nodes!!!
The key to being successful in working with the JavaScript AST is to study the node types. Memorize them. Get used to them. Think about your JavaScript in this way. Here are the basics.
Essentially acorn came out around the time Uglify2 was coming out and Esprima was still new, so he spent a lot of time improving Uglify and didn’t want to rewrite it again, so he wrote a compatability layer.