Eventstore: $by_correlation_id projection faulted after upgrade to Event Store 5.0.1

[Narvalex]

Description:
The '$by_correlation_id' projection faults after upgrading from v5.0.0 to v5.0.1, even after following the upgrade instruction found in the official blog (we stopped the projections in version 5.0.0 before the rolling upgrade).

Steps to reproduce the behavior:

1. Follow the upgrade instructions from version 5.0.0 to 5.0.1 found in the blog.
2. After some time (it happened al least 3 times to us) the projection will fail, like when not following the update instructions.

Expected behavior
The projection should not fail, even after reseting the projection $by_correlation_id

Actual behavior
The projection fails, and we can not recover the $bc-corrId streams with, for instance, reseting the projection.

Screenshot:

EventStore details:

• EventStore server version: 5.0.1

• Operating system:
  Windows 10 pro, v 1809

Additional context
The database is small, like 11.000 events.

[Narvalex]

We are willing to provide the full database to reproduce the error, as this happens consistently every time we try to upgrade from v5.0.0 to v5.0.1

[Narvalex]

We have just tried to upgrade without following the instructions (stop the projections before upgrading) and now works, aparently... wich is wierd.

[Narvalex]

We have just tried to upgrade without following the instructions (stop the projections before upgrading) and now works, aparently... wich is wierd.

Update: It did not work every time. We are rolling back to version 5.0.0 until this issue is fixed.

[Narvalex]

I was watching carefully the stream $projections-$by_correlation_id-checkpoint, and only after watching it was updated I shutdown the server and proceed to do the rolling upgrade on our single node DB. And It worked! I will close this issue for now, but I will reopen if any issues arise again. Thanks @shaan1337 for your careful observation

[Narvalex]

It did happen again. Bellow the screenshot

[Narvalex]

It did happen again. Bellow the screenshot

This time, I did followed the instructions. I appeared to work. But as soon as I restarted the node, it happened again 😟

[shaan1337]

@Narvalex hello, thanks for all the details, we're trying to reproduce this issue.

Would you please be able to upload your database somewhere and send me a link to shaan.nobee [ a t ] eventstore.org? (S3 or Google drive should be OK) This will definitely help us fix this bug more quickly.

Thanks!

[Narvalex]

Mail sent! Thanks @shaan1337 !

[shaan1337]

As discussed with @Narvalex offline, it appears that setting the eventstore config option StartStandardProjections to true may be causing the above issues.

[Narvalex]

Thanks @shaan1337 . I will reopen if it happens again

[Narvalex]

I did happen again. Now the projection is faulted in production, as I already mentioned to @shaan1337 .

[Narvalex]

The pull request #1882 is the cause for this issue, at least with following projection:

fromAll()
    .when({ 
        $any: (s, e) => {
            if (e.streamId.substring(0, 1) === '$')
                return;

            linkTo('all', e);
       }
})

[Narvalex]

It happens after the node is restarted

[YoEight]

Should be fixed by #2447

[Narvalex]

I will test ASAP and let you know, @YoEight

[Narvalex]

Thanks again @YoEight for your patience. The #2447 does prevent the $by_correlation_id to get faulted, indeed, but, given the scenario described in #1952, it does not appear any new written events with corrIds in the $bc-[correlationId] streams.
Only after merging with #1952 the new events appears again in the $bc-[corId] streams.

[YoEight]

@Narvalex The patch I did was about discarding a link event targetting an event that no longer exists. It can happen if a stream has a max age or max count set in its metadata and that targetted event got scavenged. Is this description fits your situation?

[Narvalex]

@YoEight No, it does not fit my situation. It does not involve deleted events. My situation is about a projection like the following

fromAll()
    .when({ 
        $any: (s, e) => {
            if (e.streamId.substring(0, 1) === '$')
                return;

            linkTo('all', e);
       }
})

When the aformetioned projection runs, and after a database restart the $bc-[correlationId] streams does not make progress anymore. Your patch indeed prevents the $by_correlation_id to become faulted, buth still the $bc-[correlationId] does not make progress anymore.
You can reproduce this error by following the description of pull request #1952

[YoEight]

I asked you that question specifically because if my patch prevents $by_correlation_id from becoming faulted, it means that event wasn't supposed to be written. About $by_correlation_id, are you sure it doesn't make any more progress? Let me be clearer, are you certain $by_correlation_id has events to process? If you take that information from the UI, it is possible you are reading a cached value if there are no incoming events for $by_correlation_id to process. The UI is eventually consistent when reading that information from the database.

[Narvalex]

I'am certain. Double checked with fresh build from master (since your patch is already merged with it). With cache disabled in my browser I can see no progress in $bc-[correlationId]. After merging with #1952 the $by_correlation_id makes progress.