ECMAScript tools

composable modules and transpiler infrastructure

Yusuke Suzuki (a.k.a Constellation)

self introduction

• ECMAScript engine iv / lv5 (written in C++) owner
• Esprima committer
• Escodegen owner
• Esmangle owner

background

ECMAScript everywhere

ECMAScript is now widely used

• client side
• server side
• databases

And ECMAScript tools are also developed widely

ECMAScript tools

• tools
  • UglifyJS
  • closure compiler
• transpilers
  • CoffeeScript
  • Roy
  • TypeScript
  • JSX developed by DeNA

Problem - Transpiler

• Debugging is very diffcult because of lack of source information
• We should map original code line/column information to generated code
• SourceMap solves this problem

Problem - SourceMap

• transpilers reproduce SourceMap attaching code
• transpilers becomes all-in-one
  reproduce all following modules
  • parser
  • code generator
  • source map generator
• And minified generated JS from transpiling language cannot provide SourceMap...

Problem - SourceMap

To solve it

Transpiler must also do minify task...

• create parser
• create code generator
• create source map
• create minifier
• create other language tools (such as coverage/profiling)

composable modules

Why aren't they composable?

Because they use raw script text as IR

IR

We suggest using Mozilla JS AST as IR

AST example

var i = 42;

{
    "type": "Program",
    "body": [
        {
            "type": "VariableDeclaration",
            "declarations": [
                {
                    "type": "VariableDeclarator",
                    "id": {
                        "type": "Identifier",
                        "name": "i"
                    },
                    "init": {
                        "type": "Literal",
                        "value": 42
                    }
                }
            ],
            "kind": "var"
        }
    ]
}

AST as IR

By using AST as IR, we can pass rich information

And we can develop modules using AST as IR and compose them

modules

Introduction

We'll introduce mainly 3 modules

• Esprima
• Escodegen
• Esmangle

Esprima

ECMAScript parsing infrastructure for multipurpose analysis : link

Esprima

Esprima provides complete ECMAScript parser (including strict-mode syntax check)

Parser demo

Esprima

Escodegen

ECMAScript code generator : link

Escodegen

Escodegen provides ECMAScript code generator

code generator demo

principle:

// structurally equal
parse(generate(parse(code))) === parse(code)

Escodegen do not mangle names and do other things

This keeps module simple and composable

Escodegen

Esmangle

ECMAScript mangler / minifier / optimizer : link

Esmangle

Esmangle doesn't parse / generate code. It's only responsible to optimizing AST

function upperBound(array, func) {
    var diff, len, i, current;

    len = array.length;
    i = 0;

    while (len) {
        diff = len >>> 1;
        current = i + diff;
        if (func(array[current])) {
            len = diff;
        } else {
            i = current + 1;
            len -= diff + 1;
        }
    }
    return i;
}

function upperBound(e,f){var b,a,c,d;a=e.length,c=0;while(a)b=a>>>1,d=c+b,f(e[d])?a=b:(c=d+1,a-=b+1);return c}

Esmangle

Esmangle consists of many small pass functions. We apply them to AST iteratively to resolve fixed point of AST like compiler.

Because it uses AST as IR, modules can preserve original line/column information

And other tools

For example...

• CoffeeScriptRedux can produce Mozilla JS AST

And we can develop other small (or large) modules using AST as IR

combine them to large tools

Modules are composable

Because they use standarized IR - Mozilla JS AST

We can use them and combine them into large tools

These modules become transpiler infrastructure!

Example 1 : Esmangle minifier

Esmangle minifier

It can produce SourceMap to original code

Example 2 : CoffeeScriptRedux

new CoffeeScript compiler - well-modularized compiler

CoffeeScriptRedux compiler uses Escodegen as its backend

Example 3 : sweet.js

Sweet.js brings hygienic macros from languages like Scheme and Rust to JavaScript

sweet.js also uses Escodegen as its backend

Example 4 : istanbul

istanbul is coverage tool using Esprima & Escodegen

istanbul screen shot

Other examples

• LLJS consists of modified Esprima and Escodegen
• JSHint uses Esprima for its ECMAScript parser
• Rumcoke uses Escodegen for its backend

And...

If we develop modules using AST as IR, we can apply them to all transpiling languages that produce AST

For example...

• We plan to create code coverage tool for CoffeeScript, by processing JS AST

conclusion

Conclusion

• We present composable modules for ECMAScript tooling
• By using them, we can construct large tools

Any Questions?