javascript parser infrastructure for code quality analysis
DESCRIPTION
TRANSCRIPT
JavaScript Parser Infrastructure for
Code Quality AnalysisAriya Hidayat
Twitter: @AriyaHidayathttp://ariya.ofilabs.com
Who Am I
Do you...
have a coding style/guidelines?use code quality tool such as JSLint or JSHint?actively prevent performance & coverage regressions?wish you could write your own quality rules?
Code Quality
High Quality: Practical Aspects
Avoid silly mistakes
Write readable code
Do not provoke ambiguities
Improve future maintenance
Learn better code pattern
Better Use of the Tools
EngineerTools
Feedback Cycle
Not everyone isa JavaScript ninja
•Boring•Repetitive•Time-consuming
From Spelling Checker to Grammar Enforcement Agent
Your so wrong,therefore you loose!
No misspelled word.Wrong choice of words!
Adaptive Quality Criteria
Explicit Implicit
Customize analysis optionsDefine new sets of rules
Infer from high-quality sampleObserve the engineer’s behavior
Next-Generation Code Quality Tools
To boldly analyze what no man has analyzed before...
Parser Infrastructure
JavaScript in the Browser
User Interface
Browser Engine
Graphics Stack
Data PersistenceRender Engine
JavaScript EngineNetworking I/O
JavaScript Engine Building Blocks
Virtual Machine/
InterpreterParser
Runtime
Source
Syntax Tree
Built-in objects,host objects, ...
Fast and conservative
Tokenization
keyword equal sign semicolon
identifier number
var answer = 42;
Syntax Tree
Variable Declaration
Identifier Literal Constant
answer 42
JavaScript Parser Written in JavaScript
UglifyJS
ZeParser
Esprima
TraceurNarcissus
Es-Lab
Produce Correct Output
ECMA-262 compliant Automatic semicolon insertion Strict Mode, e.g. “use strict” Unicode for identifiers
'use strict';var 日本語 = 1return 日本語
Heavily Tested > 500 unit tests Compatibility tests 100% code coverage Performance tests
Enforced during development
Sensible Syntax Tree
answer = 42
{ type: "Program", body: [ { type: "ExpressionStatement", expression: { type: "AssignmentExpression", operator: "=", left: { type: "Identifier", name: "answer" }, right: { type: "Literal", value: 42 } } } ]}
https://developer.mozilla.org/en/SpiderMonkey/Parser_API
http://esprima.org/demo/parse.html
Try online!
Syntax Tree Visualization
answer = 42
Specification, Parser Code, Syntax Tree
function parseWhileStatement() { var test, body; expectKeyword('while'); expect('('); test = parseExpression(); expect(')'); body = parseStatement(); return { type: Syntax.WhileStatement, test: test, body: body };}
while ( Expression ) StatementECMA-262 Annex A.4
High Performance
Esprima
UglifyJS parse-js 922 ms
567 ms
620 ms
233 ms
Speed ComparisonChrome 18Internet Explorer 9
http://esprima.org/test/compare.html
Benchmark corpus:jQuery, Prototype, MooTools, ExtCore, ...
Syntax Node Location{ type: "ExpressionStatement", expression: { type: "AssignmentExpression", operator: "=", left: { type: "Identifier", name: "answer", range: [0, 6] }, right: { type: "Literal", value: 42, range: [9, 11] }, range: [0, 11] }, range: [0, 11]}
answer = 42
Fit for Code Regeneration https://github.com/Constellation/escodegen
Esprima EscodegenSource Source
Syntax Tree
Syntax TransformationShorten variable nameAutomatically inline short functionObfuscate
Easily Tweaked http://mbebenita.github.com/LLJS/LLJS: Low Level JavaScript
let x = 0;Block scope
let u8 flag;let i32 position;
struct Point { int x, y;};
Data types
let u16 *p = q;Pointers
Error Tolerant Useful for IDE, editors, ...
var msg = "Hello’;
person..age = 18;
if (person.
'use strict';with (person) {}
Mismatched quote
Too many dots
Incomplete, still typing
Strict mode violation
Handle the Comments
https://github.com/thejohnfreeman/jfdocDocumentation tool Code annotation
https://github.com/goatslacker/node-typedjs
// Life, Universe, and Everythinganswer = 42
comments: [ { range: [0, 34], type: "Line", value: " Life, Universe, and Everything" }]
Forward Looking Experimental ‘harmony’ branch
Object initializer shorthand
{ let x; const y = 0;}
Block scope
var point = {x, y};
Module
module MathExtra {
// do something
}
Destructuring assignment
point = {14, 3, 77};{x, y, z} = point;
Code Quality Tools
Syntax Tree Visualization
Block statement
http://esprima.org/demo/parse.html
Most Popular Keywordsthis
functionif
returnvar
elsefor
newin
typeofwhilecase
breaktry
catchdeletethrow
switchcontinue
defaultinstanceof
dovoid
finally 4
10
12
14
15
25
35
38
72
84
84
115
122
143
188
225
232
436
562
2116
2878
3063
3108
3229
http://ariya.ofilabs.com/2012/03/most-popular-javascript-keywords.html
var fs = require('fs'), esprima = require('esprima'), files = process.argv.splice(2); files.forEach(function (filename) { var content = fs.readFileSync(filename, 'utf-8'), tokens = esprima.parse(content, { tokens: true }).tokens; tokens.forEach(function (token) { if (token.type === 'Keyword') { console.log(token.value); } });});
Most Popular Statements
http://ariya.ofilabs.com/2012/04/most-popular-javascript-statements.html
ExpressionStatementBlockStatement
IfStatementReturnStatement
VariableDeclarationFunctionDeclaration
ForStatementForInStatementWhileStatementBreakStatement
TryStatementEmptyStatementThrowStatement
SwitchStatementContinueStatementDoWhileStatementLabeledStatement 6
12
25
35
38
66
84
115
131
143
293
371
2116
2878
3063
6353
6728var fs = require('fs'), esprima = require('esprima'), files = process.argv.splice(2); files.forEach(function (filename) { var content = fs.readFileSync(filename, 'utf-8'), syntax = esprima.parse(content); JSON.stringify(syntax, function (key, value) { if (key === 'type') { if (value.match(/Declaration$/) || value.match(/Statement$/)) { console.log(value); } } return value; });});
Identifier Length Distribution
http://ariya.ofilabs.com/2012/05/javascript-identifier-length-distribution.html
0
250
500
750
0 5 10 15 20 25 30 35 40 45
mean of the identifier length is 8.27 characters
prototype-1.7.0.0.js SCRIPT_ELEMENT_REJECTS_TEXTNODE_APPENDINGprototype-1.7.0.0.js MOUSEENTER_MOUSELEAVE_EVENTS_SUPPORTED jquery-1.7.1.js subtractsBorderForOverflowNotVisiblejquery.mobile-1.0.js getClosestElementWithVirtualBindingprototype-1.7.0.0.js HAS_EXTENDED_CREATE_ELEMENT_SYNTAX
“Code Linting”
https://github.com/jshint/jshint-nextAnother example: next-generation JSHint
var fs = require('fs'), esprima = require('./esprima'), files = process.argv.splice(2); files.forEach(function (filename) { var content = fs.readFileSync(filename, 'utf-8'), syntax = esprima.parse(content, { loc: true }); JSON.stringify(syntax, function (key, value) { if (key === 'test' && value.operator === '==') console.log('Line', value.loc.start.line); return value; });});
if (x == 9) { // do Something}
Not a strict equal
Strict Mode Checking
http://ariya.ofilabs.com/2012/03/strict-mode-checks-with-esprima.html
'use strict';block = { color: 'blue', height: 20, width: 10, color: 'red'};
Duplicate data property in object literal not allowed in strict mode
“Boolean Trap” Finderhttp://ariya.ofilabs.com/2011/08/hall-of-api-shame-boolean-trap.html
Can you make up your mind? treeItem.setState(true, false);
event.initKeyEvent("keypress", true, true, null, null, false, false, false, false, 9, 0);The more the merrier?
Obfuscated choice var volumeSlider = new Slider(false);
Double-negativecomponent.setHidden(false);filter.setCaseInsensitive(false);
Style Formatter https://github.com/fawek/codepainter
CodePainterSource
Sample code
Formatted source
Infer coding styles IndentationQuote for string literalWhitespace
Rewrite and Regenerate
var syntax = esprima.parse('answer = 42;');syntax.body[0].expression.right.value = 1337;escodegen.generate(syntax)
answer = 1337;
answer = 42;
https://github.com/Constellation/escodegen
Code Coverage
http://ariya.ofilabs.com/2012/03/javascript-code-coverage-and-esprima.html
x = 42;if (false) x = -1;
https://github.com/itay/node-coverhttps://github.com/coveraje/coverajehttps://github.com/pmlopes/coberturajs
Instrumentation for Coverage http://itay.github.com/snug_codecoverage_slides/
var a = 5;
{ __statement_ZyyqFc(1); var a = 5;}
foo();
{ __statement_ZyyqFc(2); __expression_kC$jur(3), foo();}
function foo() { ...};
function foo() { __block_n53cJc(1); ...}
Statement
Expression
Block
Non-Destructive Partial Source Modification
Modified
Intact Do not remove commentsPreserve indentation & other formatting
Add “contextual” informationInject or change function invocation
String Literal Quotes
http://ariya.ofilabs.com/2012/02/from-double-quotes-to-single-quotes.html
console.log('Hello')
[ { type: "Identifier", value: "console", range: [0, 7] }, { type: "Punctuator", value: ".", range: [7, 8] }, { type: "Identifier", value: "log", range: [8, 11] }, { type: "Punctuator", value: "(", range: [11, 12] }, { type: "String", value: ""Hello"", range: [12, 19] }, { type: "Punctuator", value: ")", range: [19, 19] }]
console.log("Hello")
May need proper escaping
List of tokens
Tracking the Scalability
http://ariya.ofilabs.com/2012/01/scalable-web-apps-the-complexity-issue.html
Array.prototype.swap = function (i, j) { var k = this[i]; this[i] = this[j]; this[j] = k;}
Array.prototype.swap = function (i, j) {Log({ name: 'Array.prototype.swap', lineNumber: 1, range: [23, 94] }); var k = this[i]; this[i] = this[j]; this[j] = k;}
Execution Tracing
http://ariya.ofilabs.com/2012/02/tracking-javascript-execution-during-startup.html
https://gist.github.com/1823129jQuery Mobile startup log
4640 function calls
jquery.js 26 jQuery jquery.js 103 init undefined, undefined, [object Object] jquery.js 274 each (Function) jquery.js 631 each [object Object], (Function), undefined jquery.js 495 isFunction [object Object] jquery.js 512 type [object Object]jquery.mobile.js 1857 [Anonymous]jquery.mobile.js 642 [Anonymous]jquery.mobile.js 624 enableMouseBindingsjquery.mobile.js 620 disableTouchBindings
Transpilation http://esprima.googlecode.com/git-history/harmony/demo/transpile.html
// Object literal property shorthand.function Point(x, y) { return { x: x, y: y };}
// Object literal property shorthand.function Point(x, y) { return { x, y };}
Harmony
ES 5.1Intact
Inserted fragment
Application Structure
Ext.define('My.sample.Person', { name: 'Mr. Unknown', age: 42,
constructor: function(name) {}, walk: function(steps) {} run: function(steps) {}
});
http://www.sencha.com/learn/sencha-class-system/
Class manifest
{ className: 'My.sample.Person', functions: ['walk', 'run'], properties: ['name', 'age']}
Code Outline Eclipse
FunctionsVariables
Content Assist (aka Autocomplete, aka “Intellisense”)
http://contraptionsforprogramming.blogspot.com/2012/02/better-javascript-content-assist-in.html
Eclipse
Copy Paste Mistake
function inside(point, rect) { return (point.x >= rect.x1) && (point.y >= rect.y1) && (point.x <= rect.x2) && (point.y <= rect.y1);}
Wrong check
Refactoring Helper
// Add two numbersfunction add(firt, two) { return firt + two;}
// Add two numbersfunction add(first, two) { return first + two;}
Future
Tree traversal
“Richer” syntax treeScope analysis
Syntax transformation
Pattern Matching
Conclusion
Modern Parser
Smart editing
Source transformation
Minification & obfuscation
Instrumentation
Code coverage
Dependency analysis
Documentation generator
Conditional contracts
