6 ms·

That's all well and good ( increasing readability ) but the problem remains that each step has to finish before the next step can begin... sometimes the entire

That's all well and good ( increasing readability ) but the problem remains that each step has to finish before the next step can begin... sometimes the entire dataset won't fit into memory/machine/whatever... More useful, IMHO, would be a way to EASILY compose a true pipeline: const _pipe = (a, b) => (arg) => b(a(arg)), pipe = (...ops) => ops.reduce(_pipe) ...but have the behavior work like unix pipes ( a stream ), nodeJS supports this concept at it's most basic level using the pipe() abstraction, although you have to supply methods which handle being pipe'd to, and from... an example: const crypto = require('crypto'); // ... fs.createReadStream(file) .pipe(zlib.createGzip()) .pipe(crypto.createCipher('aes192', 'a_secret')) .pipe(reportProgress) .pipe(fs.createWriteStream(file + '.zz')) .on('finish', () => console.log('Done')); *ripped from: [source](https://medium.freecodecamp.org/node-js-streams-everything-you-need-to-know-c9141306be93 https://medium.freecodecamp.org/node-js-streams-everything-y...) Imagine reading a 100gb json file line-by-line via ajax on the client, and feeding into the pipeline of transformative methods -- iteratively introduce data in one end of the pipe, and gathering the results at the other end, and creating some visualization like a graph or whatever... without ever having to have the entire thing in memory at once...

piedar 8y ago
Have you seen https://github.com/labs42io/itiriri https://github.com/labs42io/itiriri? It does lazy queries on iterables, like IEnumerable from C#. import { query } from 'itiriri'; function* fibonacci() { let [a, b] = [0, 1]; while (true) { yield a; [a, b] = [b, a + b]; } } // Finding first 3 Fibonacci numbers that contain 42 const result = query(fibonacci()) .filter(x => x.toString().indexOf('42') !== -1) .take(3); for (const e of result) { console.log(e); } // outputs: 514229, 267914296, 7778742049
- dvlsg 8y ago
  I wrote a lib that does this too. It's been a while, but using generators tended to be way slower than just using arrays, except in the most obvious cases (array of 1000000 elements, only take 5, no sorting involved, etc). Maybe that's changed. It's been a while since I've checked.

piedar 8y ago

Have you seen https://github.com/labs42io/itiriri https://github.com/labs42io/itiriri? It does lazy queries on iterables, like IEnumerable from C#. import { query } from 'itiriri'; function* fibonacci() { let [a, b] = [0, 1]; while (true) { yield a; [a, b] = [b, a + b]; } } // Finding first 3 Fibonacci numbers that contain 42 const result = query(fibonacci()) .filter(x => x.toString().indexOf('42') !== -1) .take(3); for (const e of result) { console.log(e); } // outputs: 514229, 267914296, 7778742049

dvlsg 8y ago
I wrote a lib that does this too. It's been a while, but using generators tended to be way slower than just using arrays, except in the most obvious cases (array of 1000000 elements, only take 5, no sorting involved, etc). Maybe that's changed. It's been a while since I've checked.