Runtime: Iterator provides inconsistent results when being iterated multiple times

[mnmr]

Description

We experienced a weird issue while migrating data from one database to another, and seem to have narrowed down the issue to a problem with iterators.

Consider the following code (which can be pasted into LINQPad and executed as a C# Program):

void Main()
{
    var list = Enumerable.Range(1, 358);
    var splits = list.Split(100);

    foreach (var split in splits)
    {
        foreach (var nr in split)
            ;
        Console.WriteLine(split.Count());       
    }
}

public static class Extensions
{
    public static IEnumerable<IEnumerable<T>> Split<T>(this IEnumerable<T> source, int len)
    {
        if (len == 0)
            throw new ArgumentNullException();

        var enumer = source.GetEnumerator();
        while (enumer.MoveNext())
        {
            yield return Take(enumer.Current, enumer, len);
        }
    }

    private static IEnumerable<T> Take<T>(T head, IEnumerator<T> tail, int len)
    {
        while (true)
        {
            yield return head;
            if (--len == 0)
                break;
            if (tail.MoveNext())
                head = tail.Current;
            else
                break;
        }
    }
}

Running this code will print:

100
60

If you remove the first of the inner iterations (the foreach (var nr in split) ;), leaving just the split.Count() iteration, it correctly prints:

100
100
100
58

I would not expect the output to change depending on how often the iterator is consumed. It's also super weird that the outer loop returns fewer results (2 vs 4).

I have tested this code on Windows 10 x64 using .NET Core 3.1.102 and 3.1.302.

[msftbot[bot]]

Tagging subscribers to this area: @eiriktsarpalis
See info in area-owners.md if you want to be subscribed.

[svick]

The problem is that when you iterate the IEnumerable<T> returned from Take multiple times (you do that once in the second foreach and once in Count()), it continues iterating the same IEnumerator<T>. Maybe a clearer way to see this is:

```c#
var split = splits.First();
Console.WriteLine(split.Count());
Console.WriteLine(split.Count());
Console.WriteLine(split.Count());
Console.WriteLine(split.Count());

This prints:

100
100
100
61

If you want to make code like this work correctly, you will have to implement `Split` differently. Probably the easiest way is to eagerly enumerate the `IEnumerable<T>` returned from `Take` exactly once by using `ToList()`:

```c#
yield return Take(enumer.Current, enumer, len).ToList();

[eiriktsarpalis]

Correct, the implementation is reusing the same IEnumerator<T> instance. IEnumerators are stateful by design, and so it's very easy to make mistakes like that when manipulating them directly. I'd probably rewrite your Split method to something like the following:

public static class Extensions
{
    public static IEnumerable<IEnumerable<T>> Split<T>(this IEnumerable<T> source, int len)
    {
        if (len == 0)
            throw new ArgumentNullException();

        var list = new List<T>();
        foreach (T t in source)
        {
            if (list.Count == len)
            {
                yield return list.ToArray();
                list.Clear();
            }

            list.Add(t);
        }

        if (list.Count > 0)
        {
            yield return list.ToArray();
        }
    }   
}

This more or less guarantees referential transparency, you can enumerate both the outer and inner enumerables as many times as you like.

Closing the issue since this is by design.

EDIT: Changed wording just a bit, since I mistook @svick's comment for the OP.