Crystal: Data structers for large datasets

[wontruefree]

I just realized the implementations we have for arrays use integer types that would overflow for some data sets. I am not sure if there is a large need to have large datasets, but we do have BigInt.

A solution would be to update all the functions in Array to handle larger datasets. This would take more memory for allocating larger numbers and probably larger buffer. This seems like an option, but it might be optimizing for a small use case.

Another solution would be to have another class that has large data sets in mind, something like BigArray. This would be more fitted to the problem and would not slow down the development of normal data structures. However, This could become out of sync with the features of Array, and if dependent libraries did not use them, it would cause problems for consumers of libraries.

[asterite]

I think this is related to #8111 and possibly a duplicate of #4011

[wontruefree]

This is related. I proposed 2 solutions. Should I close this and repost these in the other issue?

[asterite]

Yes, please! I think #4011 is the best place to discuss it. 👍

[straight-shoota]

There is also #3209.

Solution no. 1 has already been discussed and that's the way to go. We need Int64 size for collections. The details are just how exactly to achieve this (probably via #8111).

Solution no. 2 is not a solution, it just shifts the problem. Not worth considering that.

[wontruefree]

I would agree I just threw it out there because it was the other obvious option.