Eventstore: Projections stop until restart of GetEventStore

[kstrauss]

Since we've upgraded to ES 3.5 we've seen several cases where projections seem to stop processing. Looking at the projections web UI they say that they are all running and at 100% but that is not the case. This is happening almost daily at this point.

Restarting the clients makes no difference, so I don't believe it has anything to do with the subscriptions to ES.

I've attached an ES log from today where we had to restart twice to get projections actually to be caught up. I've annotated it slightly.
today.txt

Any ideas of why this is now happening or how to find out more information?

Thanks,

Karl

[gregoryyoung]

Will look through logs.

Projections are internal and have nothing to do with client API.

[pgermishuys]

Do you have logs for any previous situations where you had to restart the nodes due to projections not processing events.
What i'm seeing in the logs are

[PID:04236:018 2016.04.22 14:29:59.702 INFO  CoreProjectionCheckp] Failed to write events to stream $ce-Checkpoints. Error: CommitTimeout
[PID:04236:018 2016.04.22 14:29:59.702 INFO  CoreProjectionCheckp] Retrying write to $ce-Checkpoints
[PID:04236:019 2016.04.22 14:29:59.702 INFO  CoreProjectionCheckp] Failed to write events to stream XiAll. Error: CommitTimeout
[PID:04236:019 2016.04.22 14:29:59.702 INFO  CoreProjectionCheckp] Retrying write to XiAll

and then further in the log

[PID:04236:007 2016.04.22 14:30:38.418 TRACE QueuedHandlerThreadP] SLOW QUEUE MSG [StorageReaderQueue #3]: ReadStreamEventsForward - 2889ms. Q: 3/9.
[PID:04236:022 2016.04.22 14:30:38.418 TRACE QueuedHandlerThreadP] SLOW QUEUE MSG [StorageReaderQueue #2]: ReadStreamEventsForward - 17556ms. Q: 0/107.
[PID:04236:022 2016.04.22 14:30:38.418 ERROR QueuedHandlerThreadP] ---!!! VERY SLOW QUEUE MSG [StorageReaderQueue #2]: ReadStreamEventsForward - 17556ms. Q: 0/107.
[PID:04236:029 2016.04.22 14:30:38.438 TRACE QueuedHandlerThreadP] SLOW QUEUE MSG [StorageReaderQueue #4]: ReadStreamEventsForward - 10035ms. Q: 0/30.

Those reads are taking quite awhile and along with the commits timeouts it would be good to see if this particular scenario happened in your previous encounters with projections not processing events.

That information about projections however needs to be surfaced through the statistics so that the user is made aware of these types of issues.

[kstrauss]

Yes lots of logs! I had noticed the same thing, but I'm unsure as to what to do about the slowness. It kind of is what it is without decreasing the quantity of events. It's running in a VM on a host several physical spinning disks underneath.
Does that mean I should be adjusting the
--commit-timeout-ms Commit timeout (in milliseconds).
-CommitTimeoutMs

option? What's the tradeoff being made in increasing this value? I assume that the reason why we're seeing this more and more is that we have more and events ?

10.0.0.111-2113-cluster-node.txt

[kstrauss]

i was thinking about slowness and possible causes. We have a few events that are quite a bit larger than the others. Would these be likely a problem? Obviously in my mind it shouldn't fail silently, but I can understand that projections say have a limit of say 256 kb for each event.

Below is a distribution of event sizes (including metadata) in bytes.

| Bucket Range | Number of Occurences |
| --- | --- |
| Between 0 and 1 kb | 826174 |
| Between 1 and 4 kb | 515314 |
| Between 4 and 9 kb | 66490 |
| Between 9 and 16 kb | 913 |
| Between 16 and 25 kb | 2184 |
| Between 25 and 36 kb | 14 |
| Between 36 and 49 kb | 33 |
| Between 49 and 64 kb | 44 |
| Between 64 and 81 kb | 38 |
| Between 81 and 100 kb | 253 |
| Between 100 and 121 kb | 35 |
| Between 121 and 144 kb | 43 |
| Between 144 and 169 kb | 102 |
| Between 169 and 196 kb | 77 |
| Between 196 and 225 kb | 22 |
| Between 225 and 256 kb | 20 |
| Between 256 and 289 kb | 24 |
| Between 289 and 324 kb | 20 |
| Between 324 and 361 kb | 51 |
| Between 361 and 400 kb | 11 |
| Between 400 and 441 kb | 6 |
| Between 441 and 484 kb | 3 |
| Between 484 and 529 kb | 11 |
| Between 529 and 576 kb | 5 |
| Between 576 and 625 kb | 2 |
| Between 625 and 676 kb | 1 |
| Between 676 and 729 kb | 5 |
| Between 729 and 784 kb | 2 |
| Between 784 and 841 kb | 1 |
| Between 841 and 900 kb | 3 |
| Between 900 and 961 kb | 5 |
| Between 961 and 1024 kb | 7 |
| Between 1024 and 1089 kb | 9 |
| Between 1089 and 1156 kb | 4 |
| Between 1156 and 1225 kb | 2 |
| Between 1225 and 1296 kb | 4 |
| Between 1296 and 1369 kb | 1 |
| Between 1369 and 1444 kb | 1 |
| Between 1521 and 1600 kb | 3 |
| Between 1764 and 1849 kb | 1 |
| Between 2025 and 2116 kb | 1 |
| Between 2809 and 2916 kb | 1 |
| Between 2916 and 3025 kb | 2 |
| Between 9025 and 9216 kb | 1 |
| Between 9409 and 9604 kb | 1 |

[gregoryyoung]

Between 9409 and 9604 kb 1

a 9.5MB event?

On Sat, Apr 23, 2016 at 6:33 PM, Karl Strauss [email protected]
wrote:

i was thinking about slowness and possible causes. We have a few events
that are quite a bit larger than the others. Would these be likely a
problem? Obviously in my mind it shouldn't fail silently, but I can
understand that projections say have a limit of say 256 kb for each event.

Below is a distribution of event sizes (including metadata) in bytes.
Bucket Range Number of Occurences
Between 0 and 1 kb 826174
Between 1 and 4 kb 515314
Between 4 and 9 kb 66490
Between 9 and 16 kb 913
Between 16 and 25 kb 2184
Between 25 and 36 kb 14
Between 36 and 49 kb 33
Between 49 and 64 kb 44
Between 64 and 81 kb 38
Between 81 and 100 kb 253
Between 100 and 121 kb 35
Between 121 and 144 kb 43
Between 144 and 169 kb 102
Between 169 and 196 kb 77
Between 196 and 225 kb 22
Between 225 and 256 kb 20
Between 256 and 289 kb 24
Between 289 and 324 kb 20
Between 324 and 361 kb 51
Between 361 and 400 kb 11
Between 400 and 441 kb 6
Between 441 and 484 kb 3
Between 484 and 529 kb 11
Between 529 and 576 kb 5
Between 576 and 625 kb 2
Between 625 and 676 kb 1
Between 676 and 729 kb 5
Between 729 and 784 kb 2
Between 784 and 841 kb 1
Between 841 and 900 kb 3
Between 900 and 961 kb 5
Between 961 and 1024 kb 7
Between 1024 and 1089 kb 9
Between 1089 and 1156 kb 4
Between 1156 and 1225 kb 2
Between 1225 and 1296 kb 4
Between 1296 and 1369 kb 1
Between 1369 and 1444 kb 1
Between 1521 and 1600 kb 3
Between 1764 and 1849 kb 1
Between 2025 and 2116 kb 1
Between 2809 and 2916 kb 1
Between 2916 and 3025 kb 2
Between 9025 and 9216 kb 1
Between 9409 and 9604 kb 1

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
https://github.com/EventStore/EventStore/issues/899#issuecomment-213763868

Studying for the Turing test

[kstrauss]

yep, obviously something which we shouldn't do in general. However, i guess the question is does something that large cause failures?
In general we are looking to remove the chance of a mistake like that happening. Is a 2mb or 1mb event too large?

[gregoryyoung]

I would generally lean towards yes. What is in a 2mb event?

[kstrauss]

a verbose Json document that communicates the state of a customer and associated equipment that is created from an EDI document passed to us from a 3rd party. Obviously we could make it less verbose by putting it in say mongo/sql/a blob store and and referencing the documentId, but we didn't because:

1. wanted a single write to a single resource in our operation for consistency ease
2. didn't see anything in the docs that warned against documents that large. I believe the event limit is 64mb so we were well below that level.
3. less moving pieces/laziness justified by 1 & 2 :)

Obviously if things don't work we can and would adjust!

[gregoryyoung]

I believe its 16mb but need to check.

The problem with very large events is that they can screw with things like paging. So normally I want to page 100 events at a time but what if there are 9 4mb events in that page?

[kstrauss]

I definitely agree that it can cause large variations in size and therefore performance issues. Our current issue is that projections stop working. This is unexpected. I would expect either an error or slowing performance; not a silent failure.

At this point, I am unsure what is causing the problems and how to diagnose. Is it the number of messages or the size of a small % (.0022% > 1mb) of messages the culprit? @pgermishuys mentioned some considerations/issues from the log regarding commit timeouts and very slow queue msg. Does this help point to something we should be doing? We can provide more logs if needed.

[manlpn]

Hi,

Looks like we encountered the same issue. While we were doing data synchronization between ES 3.5 and other data storage, projections web UI said that they were all running at 100%, but actually there were stopped.

[PID:04792:017 2016.05.13 11:28:00.410 TRACE QueuedHandlerMRES   ] SLOW QUEUE MSG [Projection Core #2]: UnwrapEnvelopeMessage - 62ms. Q: 160/413.

[PID:04792:017 2016.05.13 11:28:00.504 TRACE CoreProjectionCheckp] Writing checkpoint for $stream_by_category at C:169972315812/P:169972315812 with expected version number 18502

[PID:04792:017 2016.05.13 11:28:00.693 TRACE CoreProjectionCheckp] Checkpoint has been written for projection $stream_by_category at sequence number 18503 (current)

[PID:04792:037 2016.05.13 11:28:00.801 TRACE CoreProjectionCheckp] Writing checkpoint for $streams at C:169962494013/P:169962494013 with expected version number 18497

[PID:04792:037 2016.05.13 11:28:00.910 TRACE CoreProjectionCheckp] Checkpoint has been written for projection $streams at sequence number 18498 (current)

[PID:04792:017 2016.05.13 11:28:00.989 TRACE CoreProjectionCheckp] Writing checkpoint for $stream_by_category at C:169975082917/P:169975082917 with expected version number 18503

[PID:04792:037 2016.05.13 11:28:01.004 TRACE CoreProjectionCheckp] Writing checkpoint for $by_event_type at C:169718028874/P:169718028874 with expected version number 18399

[PID:04792:017 2016.05.13 11:28:01.020 TRACE CoreProjectionCheckp] Checkpoint has been written for projection $stream_by_category at sequence number 18504 (current)

[PID:04792:037 2016.05.13 11:28:11.836 INFO  CoreProjectionCheckp] Failed to write projection checkpoint to stream $projections-$by_event_type-checkpoint. Error: CommitTimeout

[PID:04792:037 2016.05.13 11:28:11.836 INFO  CoreProjectionCheckp] Retrying write checkpoint to $projections-$by_event_type-checkpoint

[PID:04792:037 2016.05.13 11:28:11.836 TRACE CoreProjectionCheckp] Writing checkpoint for $by_event_type at C:169718028874/P:169718028874 with expected version number 18399

[PID:04792:036 2016.05.13 11:28:11.883 INFO  CoreProjectionCheckp] Failed to write events to stream $ce-Tag. Error: CommitTimeout

[PID:04792:037 2016.05.13 11:28:11.883 INFO  CoreProjectionCheckp] Failed to write events to stream $streams. Error: CommitTimeout

[PID:04792:036 2016.05.13 11:28:11.883 INFO  CoreProjectionCheckp] Retrying write to $ce-Tag

[PID:04792:037 2016.05.13 11:28:11.883 INFO  CoreProjectionCheckp] Retrying write to $streams

cluster-node.txt

[gregoryyoung]

@manlpn

What kind of hardware is this running on? almost every message in your log is about slow processing.

Also do you have any very large events?

93 seconds is a massive amount of time for a read.

I agree that projections should update the UI better with information here but there is something else going on underneath that is causing io to basically come to a crawl

[jpierson]

We've seen similar behavior in our test environment, events are visible on _$all_ but do not show under category projections even though they are 100% caught up. We recently moved from v 3.2 to 3.8 but suspect that we may have had the issue prior to the upgrade.

We are looking into whether large events possibly being the culprit based on the comments above.

[gregoryyoung]

Can you send a log?

On Tue, Aug 2, 2016 at 11:50 PM, Jeff [email protected] wrote:

We've seen similar behavior in our test environment, events are visible on
_$all_ but do not show under category projections even though they are
100% caught up. We recently moved from v 3.2 to 3.8 but suspect that we may
have had the issue prior to the upgrade.

We are looking into whether large events possibly being the culprit based
on the comments above however @kstrauss https://github.com/kstrauss
mentioned.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/EventStore/EventStore/issues/899#issuecomment-237039234,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAXRWoE--qYGAKeU8kcqJZAl1Wie7VNLks5qb62SgaJpZM4INsii
.

Studying for the Turing test

[jpierson]

Here are our logs from today.
2016-08-02.zip

[gregoryyoung]

o the $scavenges stream. Reason: WrongExpectedVersion
[PID:18252:004 2016.08.02 15:47:29.627 ERROR ProjectionManager   ] The '$by_event_type' projection faulted due to 'An event emitted in recovery differs from the originally emitted event.  Existing('$>', 'C:8144137954/P:8144137435'). New('$>', 'C:8144079340/P:8144077487')'
[PID:18252:004 2016.08.02 15:51:13.980 ERROR QueuedHandlerMRES   ] Error while processing message WRITE COMPLETED: CorrelationId: bf8c55fd-8c9f-44f0-8ca6-f33d1afd6cf5, Result: Success, Message: , FirstEventNumber: 2, LastEventNumber: 2 in queued handler 'Projections Master'.
System.Exception: Exception of type 'System.Exception' was thrown.
   at EventStore.Projections.Core.Services.Management.ManagedProjection.StartOrLoadStopped() in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Management\ManagedProjection.cs:line 970
   at EventStore.Projections.Core.Services.Management.ManagedProjection.WritePersistedStateCompleted(WriteEventsCompleted message, String eventStreamId) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Management\ManagedProjection.cs:line 729
   at EventStore.Core.Messaging.RequestResponseDispatcher`2.Handle(TResponse message) in c:\projects\eventstore\src\EventStore.Core\Messaging\RequestResponseDispatcher.cs:line 75
   at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message) in c:\projects\eventstore\src\EventStore.Core\Bus\MessageHandler.cs:line 33
   at EventStore.Core.Bus.InMemoryBus.Publish(Message message) in c:\projects\eventstore\src\EventStore.Core\Bus\InMemoryBus.cs:line 324
   at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o) in c:\projects\eventstore\src\EventStore.Core\Bus\QueuedHandlerMRES.cs:line 122
[PID:18252:004 2016.08.02 15:51:14.174 ERROR QueuedHandlerMRES   ] Error while processing message WRITE COMPLETED: CorrelationId: 2e49fcb7-8c46-4cab-bff5-87c6b2e8e2b0, Result: Success, Message: , FirstEventNumber: 2, LastEventNumber: 2 in queued handler 'Projections Master'.
System.Exception: Exception of type 'System.Exception' was thrown.
   at EventStore.Projections.Core.Services.Management.ManagedProjection.StartOrLoadStopped() in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Management\ManagedProjection.cs:line 970
   at EventStore.Projections.Core.Services.Management.ManagedProjection.WritePersistedStateCompleted(WriteEventsCompleted message, String eventStreamId) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Management\ManagedProjection.cs:line 729
   at EventStore.Core.Messaging.RequestResponseDispatcher`2.Handle(TResponse message) in c:\projects\eventstore\src\EventStore.Core\Messaging\RequestResponseDispatcher.cs:line 75
   at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message) in c:\projects\eventstore\src\EventStore.Core\Bus\MessageHandler.cs:line 33
   at EventStore.Core.Bus.InMemoryBus.Publish(Message message) in c:\projects\eventstore\src\EventStore.Core\Bus\InMemoryBus.cs:line 324
   at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o) in c:\projects\eventstore\src\EventStore.Core\Bus\QueuedHandlerMRES.cs:line 122
[PID:18252:004 2016.08.02 15:51:14.174 ERROR QueuedHandlerMRES   ] Error while processing message WRITE COMPLETED: CorrelationId: 7bde6f71-c418-4fe5-bb52-890114ed1e93, Result: Success, Message: , FirstEventNumber: 2, LastEventNumber: 2 in queued handler 'Projections Master'.
System.Exception: Exception of type 'System.Exception' was thrown.
   at EventStore.Projections.Core.Services.Management.ManagedProjection.StartOrLoadStopped() in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Management\ManagedProjection.cs:line 970
   at EventStore.Projections.Core.Services.Management.ManagedProjection.WritePersistedStateCompleted(WriteEventsCompleted message, String eventStreamId) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Management\ManagedProjection.cs:line 729
   at EventStore.Core.Messaging.RequestResponseDispatcher`2.Handle(TResponse message) in c:\projects\eventstore\src\EventStore.Core\Messaging\RequestResponseDispatcher.cs:line 75
   at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message) in c:\projects\eventstore\src\EventStore.Core\Bus\MessageHandler.cs:line 33
   at EventStore.Core.Bus.InMemoryBus.Publish(Message message) in c:\projects\eventstore\src\EventStore.Core\Bus\InMemoryBus.cs:line 324
   at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o) in c:\projects\eventstore\src\EventStore.Core\Bus\QueuedHandlerMRES.cs:line 122
[PID:18252:004 2016.08.02 15:51:35.858 ERROR ProjectionManager   ] The '$by_event_type' projection faulted due to 'An event emitted in recovery differs from the originally emitted event.  Existing('$>', 'C:8144137954/P:8144137435'). New('$>', 'C:8144079340/P:8144077487')'
[PID:18252:004 2016.08.02 15:57:47.189 ERROR QueuedHandlerMRES   ] Error while processing message WRITE COMPLETED: CorrelationId: 04eff5c0-6c94-4409-b729-81ad18801291, Result: Success, Message: , FirstEventNumber: 5, LastEventNumber: 5 in queued handler 'Projections Master'.
System.Exception: Exception of type 'System.Exception' was thrown.
   at EventStore.Projections.Core.Services.Management.ManagedProjection.StartOrLoadStopped() in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Management\ManagedProjection.cs:line 970
   at EventStore.Projections.Core.Services.Management.ManagedProjection.WritePersistedStateCompleted(WriteEventsCompleted message, String eventStreamId) in c:\projects\eventstore\src\EventStore.Projections.Core\Services\Management\ManagedProjection.cs:line 729
   at EventStore.Core.Messaging.RequestResponseDispatcher`2.Handle(TResponse message) in c:\projects\eventstore\src\EventStore.Core\Messaging\RequestResponseDispatcher.cs:line 75
   at EventStore.Core.Bus.MessageHandler`1.TryHandle(Message message) in c:\projects\eventstore\src\EventStore.Core\Bus\MessageHandler.cs:line 33
   at EventStore.Core.Bus.InMemoryBus.Publish(Message message) in c:\projects\eventstore\src\EventStore.Core\Bus\InMemoryBus.cs:line 324
   at EventStore.Core.Bus.QueuedHandlerMRES.ReadFromQueue(Object o) in c:\projects\eventstore\src\EventStore.Core\Bus\QueuedHandlerMRES.cs:line 122

There are already being looked at I believe @pgermishuys are these the same?

Also @jpierson what kind of hardware are you running on/what is size of database? See many very slow messages.

[jpierson]

It's a VM hosted with HyperV, I'm told the disk it is on is a virtual disk and I'm told it is shared with another VM. We started some tests eliminating that disk and other factors from the test to see if we get different results and will likely look more into that soon. As far as the size of the event store, it's about 9 GB at the moment.

[pgermishuys]

It would appear that there are 2 separate issues here which has the same result whereby the status of the projection is not being reflected correctly.

The commit timeout being one and then there is another which pertains to the emitted stream from a projection which throws An event emitted in recovery differs from the originally emitted event.. Both of these are being looked at.

Thanks for the logs @jpierson and @kstrauss.

[pgermishuys]

The issue has been resolved by #1129

[kstrauss]

i don't think that this has been resolved. At least I'm seeing similar symptoms of projections saying they are up to date but projection streams not being updated until a restart of GES.

I have been running a build that I did myself from the release-v4.0.0 branch on 1/5/2017.

In the log I do see messages like:
PID:06528:019 2017.01.24 16:43:24.786 INFO CoreProjectionCheckp] Failed to write events to stream NonSystem. Error: CommitTimeout [PID:06528:019 2017.01.24 16:43:24.786 INFO CoreProjectionCheckp] Retrying write to NonSystem [PID:06528:020 2017.01.24 16:43:24.786 INFO CoreProjectionCheckp] Failed to write events to stream $et-TimeToPackageRequests. Error: CommitTimeout [PID:06528:020 2017.01.24 16:43:24.786 INFO CoreProjectionCheckp] Retrying write to $et-TimeToPackageRequests [PID:06528:020 2017.01.24 16:43:26.786 INFO CoreProjectionCheckp] Failed to write events to stream $et-TimeToPackageRequests. Error: CommitTimeout [PID:06528:020 2017.01.24 16:43:26.786 INFO CoreProjectionCheckp] Retrying write to $et-TimeToPackageRequests [PID:06528:019 2017.01.24 16:43:26.786 INFO CoreProjectionCheckp] Failed to write events to stream NonSystem. Error: CommitTimeout [PID:06528:019 2017.01.24 16:43:26.786 INFO CoreProjectionCheckp] Retrying write to NonSystem [PID:06528:019 2017.01.24 16:43:28.802 INFO CoreProjectionCheckp] Failed to write events to stream NonSystem. Error: CommitTimeout [PID:06528:019 2017.01.24 16:43:28.802 INFO CoreProjectionCheckp] Retrying write to NonSystem [PID:06528:020 2017.01.24 16:43:28.802 INFO CoreProjectionCheckp] Failed to write events to stream $et-TimeToPackageRequests. Error: CommitTimeout [PID:06528:020 2017.01.24 16:43:28.802 INFO CoreProjectionCheckp] Retrying write to $et-TimeToPackageRequests [PID:06528:020 2017.01.24 16:43:30.802 INFO CoreProjectionCheckp] Failed to write events to stream $et-TimeToPackageRequests. Error: CommitTimeout [PID:06528:020 2017.01.24 16:43:30.802 INFO CoreProjectionCheckp] Retrying write to $et-TimeToPackageRequests [PID:06528:019 2017.01.24 16:43:30.802 INFO CoreProjectionCheckp] Failed to write events to stream NonSystem. Error: CommitTimeout [PID:06528:019 2017.01.24 16:43:30.802 INFO CoreProjectionCheckp] Retrying write to NonSystem

Is there another setting that I need to change? Or something else?

[pgermishuys]

@kstrauss thanks for the feedback. What are you running Event Store on (e.g. Hardware specifications etc)?

The one issue that pertains to read timeouts from the projection has been resolved by the mentioned PR, however in the case of the CommitTimeout issue, the issue needs to be surfaced to the user.

Thank you once again.

[kstrauss]

it's a 4 cpu hyper-v vm configured with 16gb ram and a scsi virtual disk, running windows 2012r2.
Host is a dual cpu - total 16 core host with 96 gb ram - 12 physical spinning disks in a raid array i don't know the details on the redundancy level.
There are a total of about 8 vms on the box, so it's not immune to a noisy neighbour but in general they are not that busy as it's a dev host. If you need more details let me know.

By surface to the user, do you mean showing that the projections are not at 100% in the web-ui ?
What would a user then be expected to do ? Hit restart ? What's the use case of where a user would like their projection to stop intermittently until a manual action is performed?

@pgermishuys does that mean you would like to assign this under a different issue then?

[pgermishuys]

Let's tackle this under a new issue if you don't mind @kstrauss.

What I meant by surfacing it to the user is not that we will just leave it there, but rather let's make sure we make these things visible to the user first of all and then separately look into how best to handle the case where we cannot recover without manual intervention.

[pgermishuys]

@kstrauss mind if we handle the commit timeout issue here? https://github.com/EventStore/EventStore/issues/1190

[kstrauss]

sounds good.