Fp-ts: Call stack size exception with 5500 elements

[josete89]

🐛 Bug report

Current Behavior

If we have a not so big array and we use traverse or sequence is raising a call stack size exceed

Expected behavior

Don't throw RangeError: Maximum call stack size exceeded

Reproducible example

import * as fp from 'fp-ts'

const doProcessing = (number) => fp.task.task.of(number * 2)

const pipeline = fp.pipeable.pipe(

  fp.task.task.of(fp.array.range(1,5500)),
  fp.task.chain( (x) => {
    return fp.array.array.traverse(fp.task.task)(x,doProcessing)
  })
)

pipeline().then(console.log).catch(console.log)

Suggested solution(s)

I guess is trying to execute everything in parallel, can be executed in chunks or sequentially?

Additional context

Your environment

I've tried in [email protected] and fp-ts 2.5.3

| Software | Version(s) |
| ---------- | ---------- |
| fp-ts | 2.1.1, 2.5.3 |
| TypeScript | 3.8.3 |

[DenisFrezzato]

As mentioned in the documentation:

Data types are not stack safe and there is no trampolining implementation.

You can work around this using batchTraverse.

See also https://github.com/gcanti/fp-ts/issues/983 and https://github.com/gcanti/fp-ts/issues/907.

[josete89]

Thank you @DenisFrezzato this work when you have relative smaller arrays (10k) but in my base code I'm having an array of 660k elements and then the batchTraverse is not working

import * as fp from 'fp-ts'
import { batchTraverse } from 'fp-ts-contrib/lib/batchTraverse';

const doProcessing = (number) => fp.task.task.of(number * 2)

const pipeline = fp.pipeable.pipe(

  fp.task.task.of(fp.array.range(1,666500)),
  fp.task.chain( (x) => {
    const chunks = fp.array.chunksOf(2)
    return batchTraverse(fp.task.task)(chunks(x),doProcessing)
  })
)

pipeline().then(console.log).catch(console.log)

[DenisFrezzato]

2 doesn't sound like a reasonable number to work around this problem, you end up with two chunks of 330k elements each. I would try with bigger chunks, like 1000 or 100 elements.

[mikearnaldi]

You can also use @matechs/effect that works well with the fp-ts ecosystem and implements trampolining to get stack safety, your example is similar to https://github.com/mikearnaldi/matechs-effect/blob/master/packages/effect/tests/Effect.test.ts#L487

[josete89]

Thanks @DenisFrezzato you were right with 1000 it works, @mikearnaldi good library I will definitely have a look into this. Will be nice add a trampolin wrapper around in the code foundations of fp-ts. All good from my side this issue can be closed