Flow: Full resynchronization of client is not implemented

[fluorumlabs]

Occasionally, due to connectivity issues, client RPCs might get duplicated (?) or lost (?) (_according to the ServerRpcHandler.handleRpc_). The server must initiate client resync in this case, but it's not there yet.

No matter what, the exception is thrown instead.

java.lang.UnsupportedOperationException: FIXME: Implement resync and call it above
at com.vaadin.server.communication.ServerRpcHandler.handleRpc(ServerRpcHandler.java:298)
at com.vaadin.server.communication.UidlRequestHandler.synchronizedHandleRequest(UidlRequestHandler.java:87)

[samie]

Maybe remove the exception in the meantime? Or can this be safely ignored? It is happening inside Flow communication, which makes it harder to catch.

[Legioth]

@Peppe Has a reproduction case for this. The problematic class looks like this:

public class TodoIndexTemplate extends PolymerTemplate<TodoIndexTemplate.TodoIndexModel>
        implements View {

    public TodoIndexTemplate() {
        List<Todo> todos = new ArrayList<>();
        todos.add(new Todo("Buy Vaadin stocks", true));
        todos.add(new Todo("Get rich", false));
        getModel().setTodos(todos);

        // Listen for value change events. The name property is updated by logic
        // in the template.
        getElement().addPropertyChangeListener("newTodo",
                event -> updateNewTodo());

        // Set up a DOM id value used for integration tests
        setId("template");
    }


    @EventHandler
    private void updateNewTodo() {
        String name = getModel().getNewTodo();
        System.out.println("new: " + name);
        getModel().getTodos().add(new Todo(name, false));
        getModel().setNewTodo("");
    }

    /**
     * A model interface that defined the data that is communicated between the
     * server and the client.
     */
    public interface TodoIndexModel extends TemplateModel {
        String getNewTodo();
        void setNewTodo(String todo);
        List<Todo> getTodos();
        void setTodos(List<Todo> todos);
    }
}

The problem is triggered by the loop between doing setNewTodo("") and listening to events to the same property.

[caalador]

I only got a java.lang.RuntimeException: java.lang.StackOverflowError for my update loop. Could I have the template also?

[caalador]

The exception is due to the fact that the clientId sent from the client doesn't match the server side expectation on what it should be.

[caalador]

Question. Is there in the logs a message Unexpected message id from the client. Expected: ...?

[fluorumlabs]

Question. Is there in the logs a message Unexpected message id from the client. Expected: ...?

I can't see any.

[caalador]

Any Ignoring old message from the client. Expected: then?

[paulroemer]

No, unfortunately not.

[fluorumlabs]

Any news on that?

[fluorumlabs]

BTW Ignoring old message from the client. Expected: probably not in the logs because it's logged with a debug level

[mpilone]

I'm also seeing this error on a couple of my v13 applications. I attached a browser client log from where I was able to reproduce the issue. In my case, the error is triggered by the client missing a message from the server. The client logs that it is missing a message and waiting, then eventually times out and attempts to resync which throws the exception on the server side.

Received message with server id 212 but expected 211. Postponing handling until the missing message(s) have been received
...
Gave up waiting for message 211 from the server
Resynchronizing from server

I can reliably reproduce the problem using long polling push and unplugging my network connection for 10 seconds or so, causing a server message to be missed, then connecting to the network again. I'm sure there are probably other scenarios that cause a message to get dropped but network connectivity seems to be the easiest to reproduce.

It really seems like this functionality needs to be implemented on the server side if the client can still send the resync request. Or the client needs to be modified to handle this gracefully. It seems like this could happen anytime a push message is dropped. Right now it will always generate the exception:

ServerRpcHandler.java:313

 if (rpcRequest.isResynchronize()) {
            // FIXME Implement
            throw new UnsupportedOperationException("FIXME: Implement resync");
        }

Resync Error Client Log.txt

[mhosio]

Verified. Easy to reproduce with V14 as well exactly as described by mpilone. Seems a whole bunch of push messages can be missed.

Received message with server id 100 but expected 61. Postponing handling until the missing message(s) have been received

In my case the situation was related to getting net::ERR_CONNECTION_RESET 200 (OK) in response to the push connection request for long polling. Looks like server has been writing push messages to the open connection and incrementing the sequence numbers after which the connection has been reset. Thus client and server end up not being in sync. Attaching some screenshots from chrome debugger.

Environment:
Mac OS Mojave
jetty-maven-plugin v9.4.15.v20190215
Both server and the client running on the same machine.

[m-emelchenkov]

Please fix it if possible, it's very annoying bug. I use @Push(transport = Transport.LONG_POLLING) instead WebSocket Push due to bug in Safari which prevents using WebSocket with client-side certificates.

[mpilone]

I implemented a very poor workaround that handles the resynchronization request and simply sends a full refresh command to the client. It is far from perfect and not the correct implementation, but I rather have the user's page reload than have them see an internal error notification and call our help desk. You might be able to use it directly or at least get some ideas for your own workaround.

You can use it by implementing VaadinServiceInitListener and adding the handler in the serviceInit method like:

// This is a workaround for bug:
// https://github.com/vaadin/flow/issues/2210
serviceInitEvent.addRequestHandler(new RefreshingResynchronizeHandler());

RefreshingResynchronizeHandler.java.zip

[m-emelchenkov]

Thank you, @mpilone, for sharing.

[miroslavvojtus]

I also vote for urgent resolution of this bug. It is happening to us multiple times a day with pretty stable connection over WS. Is there any more straight forward solution than @mpilone's?
The suggested workaround is more temporary hack than solution.

Using V13 vaadin.

[mehdi-vaadin]

Can someone please provide instructions or an example to reproduce this issue?

[mpilone]

Here's a simple test case to reproduce the problem.

1. Load the view
2. Wait for the counter to start counting
3. Unplug your network connection (or take down all your interfaces)
4. Wait for 5 seconds
5. Plug in your network connection
6. Get an error.

It looks like you might get a

Trying to start a new request while another is active

OR

Internal error
Please notify the administrator.
Take note of any unsaved data, and click here to continue.

Bug2210View.java.zip

[mehdi-vaadin]

@mpilone Thanks :)

[mehdi-vaadin]

I couldn't reproduce it in Chrome 77. But, it happens in Firefox 60 ESR.

[mpilone]

The issue with resynchronization is a server side issue, see ServerRpcHandler.java:304:

if (rpcRequest.isResynchronize()) {
            // FIXME Implement
            throw new UnsupportedOperationException("FIXME: Implement resync");
        }

As long as the client can generate a resynchronization request and this code exists on the server, there are going to be problems. It's possible that there are multiple bugs here (i.e. "trying to start a new request" and "internal error") but the core resync issue should be browser type and version independent. If the client sends a resync request because it missed one or more push messages, the server throws an exception.

[ujoni]

I've used a clumsy to trigger the bug. 100% packet loss (if the timing happens to be correct) triggers the resync quite effectively. The trouble is, as far as I can tell, that if the packet is sent successfully but not received, there is no recovery scheme. Atmosphere does not cache the message (since sending was successful) and client-side does not have any recourse for retrieving the lost message. The client simply waits for it to arrive and eventually times out the missing message and raising resync.

@Legioth For this particular reproduction case, I don't see any real bug in the system. The message is lost. To cover the problem, we'd either need to actually implement resync or some bandaging to the cache; Keep recent messages, especially if they are close to messages that could not be delivered, and allow client to re-request specific messages. Or something like that.

[Legioth]

So we'd either need the sledgehammer of full resync (which impacts UX since things may flicker or unsubmitted data may be lost when everything is rebuilt) or some mechanism that preserves messages until the server can be sure the message has been processed (which increases server memory use).

[samie]

Surprising that Atmosphere does not have any built-in/optional ACK of somekind. Could that be a contribution? Anyway, full sync might be good to have as a last resort and for other use cases, but beyond that this sounds like a very traditional communication protocol design.

[ujoni]

There seems to be this about ACK messages:
https://github.com/Atmosphere/atmosphere/issues/1034

So for example Atmosphere Pro, Postman would seem to have that support.