streamline.js is a language tool to simplify asynchronous Javascript programming.
tldr; See Cheat Sheet
Instead of writing hairy code like:
function archiveOrders(date, cb) {
db.connect(function(err, conn) {
if (err) return cb(err);
conn.query("select * from orders where date < ?", [date], function(err, orders) {
if (err) return cb(err);
helper.each(orders, function(order, next) {
conn.execute("insert into archivedOrders ...", [order.id, ...], function(err) {
if (err) return cb(err);
conn.execute("delete from orders where id=?", [order.id], function(err) {
if (err) return cb(err);
next();
});
});
}, function() {
console.log("orders have been archived");
cb();
});
});
});
}you write:
function archiveOrders(date, _) {
var conn = db.connect(_);
conn.query("select * from orders where date < ?", [date], _).forEach_(_, function(_, order) {
conn.execute("insert into archivedOrders ...", [order.id, ...], _);
conn.execute("delete from orders where id=?", [order.id], _);
});
console.log("orders have been archived");
}and streamline transforms the code and takes care of the callbacks!
No control flow APIs to learn! You just have to follow a simple rule:
Replace all callbacks by an underscore and write your code as if all functions were synchronous.
Streamline is not limited to a subset of Javascript.
You can use all the features of Javascript in your asynchronous code: conditionals,
loops, try/catch/finally blocks, anonymous functions, chaining, this, etc.
Streamline also provides futures, and asynchronous variants of the EcmaScript 5 array functions (forEach, map, etc.).
The latest cool feature is TypeScript support. See https://github.com/Sage/streamlinejs/wiki/TypeScript-support for details.
Streamline 1.0 was a major revamp as a Babel Plugin. Streamline 2.0 was a smaller step from Babel 5 to Babel 6. See https://github.com/Sage/streamlinejs/wiki/Babel-upgrade for details.
NPM, of course:
npm install streamline -gThe -g option installs streamline globally.
You can also install it locally (without -g) but then the _node and _coffee
commands will not be in your default PATH.
Note: If you encounter a permission error when installing on UNIX systems, you should retry with sudo.
Warning: you may get errors during install because fibers is now installed as an optional package and it may fail to build. But this package is optional and streamline itself should install fine.
Streamline modules have ._js or ._coffee extensions and you run them with _node or _coffee.
Example:
$ cat > hello._js
console.log('hello ...');
setTimeout(_, 1000);
console.log('... world');
^D
$ _node helloYou can also create standalone shell utilities. See this example.
You can also set up your code so that it can be run directly with node or coffee.
You have two options here:
The first one is to compile your source. The recommanded way is with babel's CLI (see babel-plugin-streamline). But you can still use streamline's CLI (_node -c myfile._js or _coffee -c myfile._coffee)
The second one is to create a loader which will register require hooks for the ._js and ._coffee extensions. See this example.
Compiling will give you the fastest startup time because node will directly load the compiled *.js files but the registration API has a cache option which comes close.
The recommandation is to use the loader during development but deploy precompiled files.
The runtime library is provided as a separate streamline-runtime package.
If you deploy precompiled files you only need streamline-runtime.
If your application/library uses a loader you will need to deploy both streamline-runtime and streamline with it.
You have two options to use streamline in the browser:
- You can transform and bundle your files with browserify. See how the build.js script builds the `test/browser/*-test.js files for an example.
- You can also transform the code in the browser with the
transformAPI. All the necessary JS code is available as a singlelib/browser/transform.jsfile. See the streamlineMe example.
Streamline can transform the code for several target runtimes:
- callbacks. The transformed code will be pure ES5 code. It should be compatible with all JavaScript engines.
- fibers. The transformed code will take advantage of the fibers library. This option is only available server-side.
- generators. The transformed code will take advantage of JavaScript generators. It will run in node.js 0.12 (with the
--harmonyflag), in node.js 4.0 (without any special flag) and in latest browsers. - await. The transformed code will take advantage of ES7 async/await.
The choice of a target runtime should be driven by benchmarks:
- The fibers mode gives superior development experience (because it uses real stacks for each fiber so you can step over async calls). It is also very efficient in production if your code traverses many layers of asynchronous calls.
- The callbacks transform is obtained by chaining the generators transform and the regenerator transform. It is less efficent than the generators transform and we recommend that you use generators if generators are supported by your target JavaScript engine and that you only use callbacks if you target a legacy JavaScript engine.
- The await mode is experimental at this stage. It relies on an emulation as async/await is not yet available natively in JavaScript engines.
You can control the target runtime with the --runtime (callbacks|fibers|generators|await) CLI option, or with the runtime API option.
You can call standard node functions from streamline code. For example the fs.readFile function:
function lineCount(path, _) {
return fs.readFile(path, "utf8", _).split('\n').length;
}You can also call streamline functions as if they were standard node functions. For example, the lineCount function that we just defined above can be called as follows in standard node.js style:
lineCount("README.md", function(err, result) {
if (err) return console.error("ERROR: " + err.message);
console.log("README has " + result + " lines.");
});You can mix streamline functions, classical callback based code and synchrononous functions in the same file.
Streamline only transforms the functions that have the special _ parameter.
Note: this works with all transformation options. Even if you use the fibers option, you can seamlessly call standard callback based node APIs and the asynchronous functions that you create with streamline have the standard node callback signature.
Streamline also provides seamless interoperability with Promise libraries, in both directions.
First, you can consume promises from streamline code, by passing two underscores to their then method:
function myStreamlineFunction(p1, p2, _) {
var result = functionReturningAPromise(p1, p2).then(_, _);
// do something with result
}
Note: if the promise fails the error will be propagated as an exception and you can catch it with try/catch.
In the other direction you can get a promise from any callback-based asynchronous function by passing void _ instead of _. For example:
function readFileWithPromise(path) {
var p = fs.readFile(path, 'utf8', void _);
// p is a promise.
p.then(function(result) {
// do something with result
}, function(err) {
// handle error
});
}Streamline also provides futures. Futures are like promises, without all the bells and whistles. They let you parallelize I/O operations in a very simple manner.
If you pass !_ instead of _ when calling a streamline function, the function returns a future. The future is just a regular node.js asynchronous function that you can call later to obtain the result. Here is an example:
function countLines(path, _) {
return fs.readFile(path, "utf8", _).split('\n').length;
}
function compareLineCounts(path1, path2, _) {
// parallelize the two countLines operations
var n1 = countLines(path1, !_);
var n2 = countLines(path2, !_);
// get the results and diff them
return n1(_) - n2(_);
}In this example, countLines is called twice with !_. These calls start the fs.readFile asynchronous operations and return immediately two futures (n1 and n2). The return statement retrieves the results with n1(_) and n2(_) calls and computes their difference.
See the futures wiki page for details.
The flows module contains utilities to deal with futures. For example flows.collect to wait on an array of futures and flows.funnel to limit the number of concurrent operations.
Streamline extends the Array prototype with asynchronous variants of the EcmaScript 5 forEach, map, filter, reduce, ... functions. These asynchronous variants are postfixed with an underscore and they take an extra _ argument (their callback too), but they are otherwise similar to the standard ES5 functions. Here is an example with the map_ function:
function dirLines(dir, _) {
return fs.readdir(dir, _).map_(_, function(_, file) {
return fs.readFile(dir + '/' + file, 'utf8', _).split('\n').length;
});
}Parallelizing loops is easy: just pass the number of parallel operations as second argument to the call:
function dirLines(dir, _) {
// process 8 files in parallel
return fs.readdir(dir, _).map_(_, 8, function(_, file) {
return fs.readFile(dir + '/' + file, 'utf8', _).split('\n').length;
});
}If you don't want to limit the level of parallelism, just pass -1.
See the documentation of the builtins module for details.
Streamline lets you do your exception handling with the usual try/catch construct. The finally clause is also fully supported.
Streamline overrides the ex.stack getter to give you complete comprehensive stacktrace information. In callbacks and generators modes you get two stack traces:
- the raw stack trace of the last callback.
- the async stack trace of the asynchronous calls that caused the exception.
In fibers mode there is a single stack trace.
Exception handling also works with futures and promises. If a future throws an exception before you try to read its result, the exception is memorized by the future and you get it at the point where your try to read the future's result. For example:
try {
var n1 = countLines(badPath, !_);
var n2 = countLines(goodPath, !_);
setTimeout(_, 1000); // n1 fails, exception is memorized
return n1(_) - n2(_); // exception is thrown by n1(_) expression.
} catch (ex) {
console.error(ex.stack); // exception caught here
}Some APIs return several results through their callback. For example:
request(options, function(err, response, body) {
// ...
});You can get all the results by passing [_] instead of _:
var results = request(options, [_]);
// will be better with destructuring assignment.
var response = results[0];
var body = results[1];Note: if you only need the first result you can pass _:
var response = request(options, _);Some APIs don't follow the standard error first callback convention of node.js. Instead, the accept a pair of callback and errback arguments. Streamline lets you call them by passing two _ arguments. For example:
function nodeStyleFn(arg, _) {
return callbackErrbackStyleFn(arg, _, _);
}As seen above, this feature is used in the promise interop: result = promise.then(_, _) is just a special case.
It can also be used to handle the special error-less callback of fs.exists:
function fileExists(path, _) {
// the second _ is ignored by fs.exists!
return fs.exists(path, _, _);
}CoffeeScript is fully supported.
You can seamlessly debug streamline code thanks to JavaScript source maps. See this video for a quick demo.
To activate this feature, pass the --source-map options to _node or _coffee, or set the sourceMap option if you register via a loader.
The tutorial shows streamline.js in action on a simple search aggregator application.
The diskUsage examples show an asynchronous directory traversal that computes disk usage.
The loader examples demonstrate how you can enable the ._js and ._coffee require hooks.
You can see how streamline transforms the code by playing with the online demo.
Read the FAQ.
If you don't find your answer in the FAQ, post to the mailing list, or file an issue in GitHub's issue tracking.
The following packages are installed together with streamline:
- babel-plugin-streamline: babel plug-in which transforms streamline code.
- streamline-runtime: runtime that you should distribute with compiled modules.
The following packages extend the power of streamline:
- express-streamline: interop with express
- streamline-express: interop with express
- ez-streams: streams and transforms for streamline.
- streamline-flamegraph: flamegraph monitoring.
The tutorial and FAQ are must-reads for starters.
The API is documented here.
For support and discussion, please join the streamline.js mailing list.
See the AUTHORS file.
Special thanks to Marcel Laverdet who contributed the fibers implementation and to Geoffry Song who contributed source map support (in 0.x versions).