javascript parser infrastructure for code quality analysis

JavaScript Parser Infrastructure for

Code Quality AnalysisAriya Hidayat

Twitter: @AriyaHidayathttp://ariya.ofilabs.com

Who Am I

Do you...

have a coding style/guidelines?use code quality tool such as JSLint or JSHint?actively prevent performance & coverage regressions?wish you could write your own quality rules?

Code Quality

High Quality: Practical Aspects

Avoid silly mistakes

Write readable code

Do not provoke ambiguities

Improve future maintenance

Learn better code pattern

Better Use of the Tools

EngineerTools

Feedback Cycle

Not everyone isa JavaScript ninja

•Boring•Repetitive•Time-consuming

From Spelling Checker to Grammar Enforcement Agent

Your so wrong,therefore you loose!

No misspelled word.Wrong choice of words!

Adaptive Quality Criteria

Explicit Implicit

Customize analysis optionsDefine new sets of rules

Infer from high-quality sampleObserve the engineer’s behavior

Next-Generation Code Quality Tools

To boldly analyze what no man has analyzed before...

Parser Infrastructure

JavaScript in the Browser

User Interface

Browser Engine

Graphics Stack

Data PersistenceRender Engine

JavaScript EngineNetworking I/O

JavaScript Engine Building Blocks

Virtual Machine/

InterpreterParser

Runtime

Source

Syntax Tree

Built-in objects,host objects, ...

Fast and conservative

Tokenization

keyword equal sign semicolon

identifier number

var answer = 42;

Syntax Tree

Variable Declaration

Identifier Literal Constant

answer 42

JavaScript Parser Written in JavaScript

UglifyJS

ZeParser

Esprima

TraceurNarcissus

Es-Lab

Produce Correct Output

ECMA-262 compliant Automatic semicolon insertion Strict Mode, e.g. “use strict” Unicode for identifiers

'use strict';var 日本語 = 1return 日本語

Heavily Tested > 500 unit tests Compatibility tests 100% code coverage Performance tests

Enforced during development

Sensible Syntax Tree

answer = 42

{ type: "Program", body: [ { type: "ExpressionStatement", expression: { type: "AssignmentExpression", operator: "=", left: { type: "Identifier", name: "answer" }, right: { type: "Literal", value: 42 } } } ]}

https://developer.mozilla.org/en/SpiderMonkey/Parser_API

http://esprima.org/demo/parse.html

Try online!

Syntax Tree Visualization

answer = 42

Specification, Parser Code, Syntax Tree

function parseWhileStatement() { var test, body;  expectKeyword('while'); expect('('); test = parseExpression(); expect(')'); body = parseStatement();  return { type: Syntax.WhileStatement, test: test, body: body };}

while ( Expression ) StatementECMA-262 Annex A.4

High Performance

Esprima

UglifyJS parse-js 922 ms

567 ms

620 ms

233 ms

Speed ComparisonChrome 18Internet Explorer 9

http://esprima.org/test/compare.html

Benchmark corpus:jQuery, Prototype, MooTools, ExtCore, ...

Syntax Node Location{ type: "ExpressionStatement", expression: { type: "AssignmentExpression", operator: "=", left: { type: "Identifier", name: "answer", range: [0, 6] }, right: { type: "Literal", value: 42, range: [9, 11] }, range: [0, 11] }, range: [0, 11]}

answer = 42

Fit for Code Regeneration https://github.com/Constellation/escodegen

Esprima EscodegenSource Source

Syntax Tree

Syntax TransformationShorten variable nameAutomatically inline short functionObfuscate

Easily Tweaked http://mbebenita.github.com/LLJS/LLJS: Low Level JavaScript

let x = 0;Block scope

let u8 flag;let i32 position;

struct Point { int x, y;};

Data types

let u16 *p = q;Pointers

Error Tolerant Useful for IDE, editors, ...

var msg = "Hello’;

person..age = 18;

 if (person.

 'use strict';with (person) {}

Mismatched quote

Too many dots

Incomplete, still typing

Strict mode violation

Handle the Comments

https://github.com/thejohnfreeman/jfdocDocumentation tool Code annotation

https://github.com/goatslacker/node-typedjs

// Life, Universe, and Everythinganswer = 42

comments: [ { range: [0, 34], type: "Line", value: " Life, Universe, and Everything" }]

Forward Looking Experimental ‘harmony’ branch

Object initializer shorthand

{ let x; const y = 0;}

Block scope

var point = {x, y};

Module

module MathExtra {

// do something

}

Destructuring assignment

point = {14, 3, 77};{x, y, z} = point;

Code Quality Tools

Syntax Tree Visualization

Block statement

http://esprima.org/demo/parse.html

Most Popular Keywordsthis

functionif

returnvar

elsefor

newin

typeofwhilecase

breaktry

catchdeletethrow

switchcontinue

defaultinstanceof

dovoid

finally 4

10

12

14

15

25

35

38

72

84

84

115

122

143

188

225

232

436

562

2116

2878

3063

3108

3229

http://ariya.ofilabs.com/2012/03/most-popular-javascript-keywords.html

var fs = require('fs'), esprima = require('esprima'), files = process.argv.splice(2); files.forEach(function (filename) { var content = fs.readFileSync(filename, 'utf-8'), tokens = esprima.parse(content, { tokens: true }).tokens;  tokens.forEach(function (token) { if (token.type === 'Keyword') { console.log(token.value); } });});

Most Popular Statements

http://ariya.ofilabs.com/2012/04/most-popular-javascript-statements.html

ExpressionStatementBlockStatement

IfStatementReturnStatement

VariableDeclarationFunctionDeclaration

ForStatementForInStatementWhileStatementBreakStatement

TryStatementEmptyStatementThrowStatement

SwitchStatementContinueStatementDoWhileStatementLabeledStatement 6

12

25

35

38

66

84

115

131

143

293

371

2116

2878

3063

6353

6728var fs = require('fs'), esprima = require('esprima'), files = process.argv.splice(2); files.forEach(function (filename) { var content = fs.readFileSync(filename, 'utf-8'), syntax = esprima.parse(content);  JSON.stringify(syntax, function (key, value) { if (key === 'type') { if (value.match(/Declaration$/) || value.match(/Statement$/)) { console.log(value); } } return value; });});

Identifier Length Distribution

http://ariya.ofilabs.com/2012/05/javascript-identifier-length-distribution.html

0

250

500

750

0 5 10 15 20 25 30 35 40 45

mean of the identifier length is 8.27 characters

prototype-1.7.0.0.js SCRIPT_ELEMENT_REJECTS_TEXTNODE_APPENDINGprototype-1.7.0.0.js MOUSEENTER_MOUSELEAVE_EVENTS_SUPPORTED jquery-1.7.1.js subtractsBorderForOverflowNotVisiblejquery.mobile-1.0.js getClosestElementWithVirtualBindingprototype-1.7.0.0.js HAS_EXTENDED_CREATE_ELEMENT_SYNTAX

“Code Linting”

https://github.com/jshint/jshint-nextAnother example: next-generation JSHint

var fs = require('fs'), esprima = require('./esprima'), files = process.argv.splice(2); files.forEach(function (filename) { var content = fs.readFileSync(filename, 'utf-8'), syntax = esprima.parse(content, { loc: true });  JSON.stringify(syntax, function (key, value) { if (key === 'test' && value.operator === '==') console.log('Line', value.loc.start.line); return value; });});

if (x == 9) { // do Something}

Not a strict equal

Strict Mode Checking

http://ariya.ofilabs.com/2012/03/strict-mode-checks-with-esprima.html

'use strict';block = { color: 'blue', height: 20, width: 10, color: 'red'};

Duplicate data property in object literal not allowed in strict mode

“Boolean Trap” Finderhttp://ariya.ofilabs.com/2011/08/hall-of-api-shame-boolean-trap.html

Can you make up your mind? treeItem.setState(true, false);

event.initKeyEvent("keypress", true, true, null, null, false, false, false, false, 9, 0);The more the merrier?

Obfuscated choice var volumeSlider = new Slider(false);

Double-negativecomponent.setHidden(false);filter.setCaseInsensitive(false);

Style Formatter https://github.com/fawek/codepainter

CodePainterSource

Sample code

Formatted source

Infer coding styles IndentationQuote for string literalWhitespace

Rewrite and Regenerate

var syntax = esprima.parse('answer = 42;');syntax.body[0].expression.right.value = 1337;escodegen.generate(syntax)

answer = 1337;

answer = 42;

https://github.com/Constellation/escodegen

Code Coverage

http://ariya.ofilabs.com/2012/03/javascript-code-coverage-and-esprima.html

x = 42;if (false) x = -1;

https://github.com/itay/node-coverhttps://github.com/coveraje/coverajehttps://github.com/pmlopes/coberturajs

Instrumentation for Coverage http://itay.github.com/snug_codecoverage_slides/

var a = 5;

{ __statement_ZyyqFc(1); var a = 5;}

foo();

{ __statement_ZyyqFc(2); __expression_kC$jur(3), foo();}

function foo() { ...};

function foo() { __block_n53cJc(1); ...}

Statement

Expression

Block

Non-Destructive Partial Source Modification

Modified

Intact Do not remove commentsPreserve indentation & other formatting

Add “contextual” informationInject or change function invocation

String Literal Quotes

http://ariya.ofilabs.com/2012/02/from-double-quotes-to-single-quotes.html

console.log('Hello')

[ { type: "Identifier", value: "console", range: [0, 7] }, { type: "Punctuator", value: ".", range: [7, 8] }, { type: "Identifier", value: "log", range: [8, 11] }, { type: "Punctuator", value: "(", range: [11, 12] }, { type: "String", value: ""Hello"", range: [12, 19] }, { type: "Punctuator", value: ")", range: [19, 19] }]

console.log("Hello")

May need proper escaping

List of tokens

Tracking the Scalability

http://ariya.ofilabs.com/2012/01/scalable-web-apps-the-complexity-issue.html

Array.prototype.swap = function (i, j) { var k = this[i]; this[i] = this[j]; this[j] = k;}

Array.prototype.swap = function (i, j) {Log({ name: 'Array.prototype.swap', lineNumber: 1, range: [23, 94] }); var k = this[i]; this[i] = this[j]; this[j] = k;}

Execution Tracing

http://ariya.ofilabs.com/2012/02/tracking-javascript-execution-during-startup.html

https://gist.github.com/1823129jQuery Mobile startup log

4640 function calls

jquery.js 26 jQuery jquery.js 103 init undefined, undefined, [object Object] jquery.js 274 each (Function) jquery.js 631 each [object Object], (Function), undefined jquery.js 495 isFunction [object Object] jquery.js 512 type [object Object]jquery.mobile.js 1857 [Anonymous]jquery.mobile.js 642 [Anonymous]jquery.mobile.js 624 enableMouseBindingsjquery.mobile.js 620 disableTouchBindings

Transpilation http://esprima.googlecode.com/git-history/harmony/demo/transpile.html

// Object literal property shorthand.function Point(x, y) { return { x: x, y: y };}

// Object literal property shorthand.function Point(x, y) { return { x, y };}

Harmony

ES 5.1Intact

Inserted fragment

Application Structure

Ext.define('My.sample.Person', { name: 'Mr. Unknown',  age: 42,

constructor: function(name) {},  walk: function(steps) {} run: function(steps) {}

});

http://www.sencha.com/learn/sencha-class-system/

Class manifest

{ className: 'My.sample.Person', functions: ['walk', 'run'], properties: ['name', 'age']}

Code Outline Eclipse

FunctionsVariables

Content Assist (aka Autocomplete, aka “Intellisense”)

http://contraptionsforprogramming.blogspot.com/2012/02/better-javascript-content-assist-in.html

Eclipse

Copy Paste Mistake

function inside(point, rect) { return (point.x >= rect.x1) && (point.y >= rect.y1) && (point.x <= rect.x2) && (point.y <= rect.y1);}

Wrong check

Refactoring Helper

// Add two numbersfunction add(firt, two) { return firt + two;}

// Add two numbersfunction add(first, two) { return first + two;}

Future

Tree traversal

“Richer” syntax treeScope analysis

Syntax transformation

Pattern Matching

Conclusion

Modern Parser

Smart editing

Source transformation

Minification & obfuscation

Instrumentation

Code coverage

Dependency analysis

Documentation generator

Conditional contracts

Thank You!

ariya.hidayat@gmail.com

@AriyaHidayat

ariya.ofilabs.com