Laravel-websockets: Invalid auth signature provided Exception

[irazasyed]

So I have a search engine built into my app that fetches results from multiple sources from across the web from different engines and broadcasts the results in real-time as we get a response.

I was using laravel-echo-server earlier and it was all good. I migrated to this package and have been facing this issue. At first, I thought it could be a third-party issue and tested the APIs and everything else but it seems like everything is fine at that end but when it's broadcasting results in a loop, it shows this error for few broadcasts.

What you think could be breaking the flow and causing this issue? I'm sending JSON payload of results after we've parsed and transformed as per the format we need on the front-end.

Exception `Symfony\Component\HttpKernel\Exception\HttpException` thrown: `Invalid auth signature provided.`

[irazasyed]

FYI, I have no other issues with the package and other issues for which I had, I've sent PRs for it. Everything else and other parts of the app works fine. Just this part I'm having trouble and there are no logs for some reason. So it's quite challenging to debug the issue.

[coolcodemy]

Inside WebSockets/Channels/Channel.php, you have verifySignature method. It is trying to compare hash_hmac of the payload and app's secret. This will fail because $payload->auth take the value inside config/broadcasting.php. But the hash_hmac('sha256', $signature, $connection->app->secret) returns the value for the app you are currently selecting.

[mattiasgeniar]

I can confirm we're seeing the same behaviour: for _some_ of our events, we're getting this invalid auth signature;

{"error":"Invalid auth signature provided."} {"exception":"[object] (Illuminate\\Broadcasting\\BroadcastException(code: 0): {\"error\":\"Invalid auth signature provided.\"}

[stayallive]

I think this might be caused by the signatures not being validated correctly according to what the Pusher server does, might be fixed with #38.

[irazasyed]

@stayallive I tested your code and unfortunately, It's throwing that exception for every event now instead of specific ones so that has not resolved the issue rather introduced a new one.

[coolcodemy]

@irazasyed checkout my PR #39

[irazasyed]

@coolcodemy That didn't resolve the issue either plus it's not specific to dashboard related events or auth.

[coolcodemy]

@irazasyed oopss. sorry. That fix was for invalid signature. I've read it wrong.

[stayallive]

I am seeing that too... this has to do with the route params being added as query parameter causing the validation to fail... damn. Back to the drawing board...

[irazasyed]

So I've been debugging this issue all day today and here are a few things I found that may help us get to the root:

• When this invalid signature issue occurs, the request body is null/empty even tho there is data.
• I increased the max request kb just in case my payload might've become huge but it still failed.
• Modified my data that was being broadcast and it works. The data that was removed from the payload was the links that were urlencoded and had various chars in them. Tested with decoded links and every other ways but it always failed. However, Other data with links also worked fine. But when I was using laravel-echo-server with the same data, it always used to work fine. So something here somewhere is discarding the request body due to which it's causing this issue. Any pointers would be good?

[mattiasgeniar]

Any pointers would be good?

Would you be able to provide the payload that got dropped?

There are various json_encode's in the code, if that fails to encode properly it'll return false and might cause an empty body. But, in order to troubleshoot, the exact payload would be useful.

[harrynewsome]

• Modified my data that was being broadcast and it works.

Experiencing this same issue and can confirm changing the data being broadcast fixes the issue.

[mattiasgeniar]

@harrynewsome do you happen to have an example of a payload in the broadcast that reliably causes the issue?

[ulfie22]

I have the web socket server talking to three deployments of my code (one server with three socket apps defined). For my local and staging servers, it is working well. For my production server, I am getting the Invalid auth signature provided error on every broadcast attempt. I am using private channels and I see the auth request hit my Laravel server. In the socket server dashboard, I never see my app successfully occupied and subscribed to the channel (as it does in the other deployments). The same type of data is being sent as the payload in all three deployments so I don't think it's anything there. This is a real showstopper for me - any thoughts how to get around this?

On the actual socket server, there is an error in the log for each request: Exception `BeyondCode\LaravelWebSockets\WebSockets\Exceptions\InvalidSignature` thrown: `Invalid Signature` 661879: exception `BeyondCode\LaravelWebSockets\WebSockets\Exceptions\InvalidSignature` thrown: `Invalid Signature`.

[mattiasgeniar]

I can confirm this Exception is thrown when the payload you're trying to broadcast is too large. Somewhere along the way, the body of the POST to the websocket server gets truncated. I'm still trying to find _where_ exactly this happens.

It's not the max_request_size_in_kb parameter, that appears to be correctly handling things.

[mpociot]

Not sure if this is related, but I just merged a PR that fixes the way we generate and validate the auth signatures.
I haven’t tagged a new release yet - but maybe you can try it out.

[mattiasgeniar]

I'll give it a try later today, here's what I saw when debugging this issue;

1) The payload being sent to the laravel-websockets server wasn't complete, it only contained the first 65KB or so of the request
2) It's unclear whether the payload was truncated at the request (in Laravels' broadcasting) or dropped partially on the recipient (laravel-websockets)

For instance, here's a part of the payload that failed. Note how the #content suddenly stops, this is the data in $request->getBody(). This request was dump()'d in the laravel-websockets server.

Illuminate\Http\Request {#1810
  [...]
  +server: Symfony\Component\HttpFoundation\ServerBag {#1811
    #parameters: array:5 [
      "SERVER_NAME" => "ohdear.app.test"
      "SERVER_PORT" => 6001
      "REQUEST_URI" => "/apps/1/events"
      "QUERY_STRING" => "appId=1&auth_key=abc&auth_signature=def&auth_timestamp=1546463497&auth_version=1.0&body_md5=ghi"
      "REQUEST_METHOD" => "POST"
    ]
  }
  #content: "{"name":"App\\Events\\RunEnded","data":"{\"run\":{\"id\":39,\"check_id\":38,\"parameters\":[],\"result\":\"failed\",\"result_payload\":{\"crawledPages\":\"a:3:{i:0;s:44:\\\"http:\\\/\\\/immutable.be\\\/mixed-content-test-page\\\/\\\";i:1;s:44:\\\"https:\\\/\\\/immutable.be\\\/mixed-content-test-page\\\";i:2;s:45:\\\"https:\\\/\\\/immutable.be\\\/mixed-content-test-page\\\/\\\";}\",\"foundMixedContent\":\"a:999:{i:0;O:39:\\\[...(truncated by mattias)...]{s:27:\\\"\\u0000GuzzleHttp\\\\Psr7\\\\Uri\\u0000scheme\\\";s:4:\\\"http\\\";s:29:\\\"\\u0000GuzzleH"    <===== Payload got truncated here (note the GuzzleH) after ~65KB
  [...]
}

I will track it down further to see if the data in the event got truncated at the source or at the destination, to narrow it down.

[stayallive]

^ that looks like the output of a dd() or dump() be aware that it could also truncate the string (maybe not on cli but definitely in the HTML variant: https://github.com/symfony/symfony/blob/4.2/src/Symfony/Component/VarDumper/Dumper/HtmlDumper.php#L72), might want to double check that too :)

[mattiasgeniar]

that looks like the output of a dd() or dump() be aware that it could also truncate the string

Hmm good point, this was caught by a dump() indeed using the dump-server

[ulfie22]

The payload I am sending is not anywhere near 65K. I have two more or less identical servers sending data to the websocket server - same Laravel code base (one staging, one production) - and one works fine and the other generates the Invalid auth signature provided. error. I am happy to try to do any diagnosis you suggest to figure out why this is happening. Thanks.

[stayallive]

@mattiasgeniar also looking at your auth_signature and body_md5 in the QUERY_STRING they're really borked if that is real... please ignore if you fingered the values for the sake of sharing :)

@ulfie22 Would be great if you can validate if that's still the case with the code in master (~set dev-master@dev in your composer.json as the version number~ 1.0.4 was just released with those changes) which changes the signature validation (#38) to be more compliant with how Pusher generates / validates it. This might be the fix.

[ulfie22]

@stayallive I loaded 1.0.4 (I had been using dev-master) with the same result. Can you please give me some guidance as to how/where in code to capture these requests? I have these two servers that are virtually identical - it seems to be a perfect spot to sort this out. Thank you for your help!

[ulfie22]

UPDATE: @mpociot was extraordinarily generous with his time and helped me discover that the issue I was having was totally pilot error by me. Thank you all for your suggestions and for me this is confirmation again that checking your configuration data for the hundredth time is probably worthwhile. This is an incredibly important project and again thank the authors and community for making it possible.

[stayallive]

@ulfie22 would you be able to share what was wrong in you config since you are not the only one that might have done this and other can benefit from knowing where to double-check?

Related: https://xkcd.com/979/

[yizen]

@ulfie22 Running also into the same issue, could you share what was wrong with your config ?
@mpociot If you could share what was wrong with @ulfie22 , that might help here, where everything runs fine on my local, and fails on AWS (with ElasticBeanstalk). I surely have overlooked something in my config, but cannot figure out what.

[ulfie22]

@stayallive @Yizen it's almost too embarrassing to admit: i am running 3 socket apps on one server (the server is at digital ocean; i have my local machine, plus staging and production servers at AWS). my configuration error was having the wrong app key in the .env file on my production server. i know I checked it multiple times and was shocked when my brain finally registered it was the wrong one - I had the staging server app key on the production server which had the correct app id. bottom line: in my case, the authentication error was EXACTLY the right message due to my screw up.

[yizen]

Thanks @ulfie22 , that was helpful : I'm going to double check all my AWS Beanstalk vars !

[ulfie22]

@stayallive @Yizen @mpociot I'm baaaack... :)

I have my socket server working correctly and am using it to great effect .... until I started adding more events. My app now sends one HTTP request that triggers a bunch of events which cause different pieces of data to be broadcast through the socket server. As I expanded one of these data payloads, it started failing with the {"error":"Invalid auth signature provided."} error - in this case, I am CERTAIN that the error message is not correct as multiple other events get broadcast and received as expected at the same time. I changed the 'max_request_size_in_kb' => 1000, in my config/websockets.php file to 1000 and it did not solve the problem. Here are two gists:

failing:
https://gist.github.com/ulfie22/bcab04fb9202f861661ba28dd9427d3a
successful:
https://gist.github.com/ulfie22/e8575aefcc7d48251a395caab47cfb0e

The failing one has an additional data item (recurrence) which is just a bunch of strings. The saved file size is 24.5k - well under the 1000 specified. There are some emoji characters in the "base" part of the data, so the character set isn't the issue. Any ideas very welcome. Thanks!

does Laravel somehow limit the size of the packet that can be sent?

[stayallive]

Laravel certainly does not but it could be in the Pusher SDK or even in the Ratchet HTTP server implementation.

(have not had the time to dig in but awesome you are able to provide some payloads!)

[mpociot]

Alright. So I did a lot of digging and here's what I found out:

The way I could reproduce this issue:

• Spin up a new server through Laravel Forge (I'm using the lowest DO server specs)
• Start the websocket server there and give access to the port
• Broadcast an event from a Laravel app on your local machine to the Forge server (using the IP / hostname)

The error occurs and the payload gets truncated (at some point)
Be aware that the payload needs to be rather long in order for this issue to take place.

It works though, if I broadcast the same event on the Forge server to the local websocket server (127.0.0.1 as the Pusher host).

This must be some kind of network configuration as it works locally, but not from a remote connection.

@mattiasgeniar do you have any idea?

[stayallive]

That is class A sherlock work! Mattias as a server guru hopefully has something smart to add that could point to a reason 👍

After a bit of Googling I found these very old issues:

• https://github.com/reactphp/react/issues/324
• https://github.com/reactphp/event-loop/issues/26

They do seem to point to something similar, but as far as I can tell this has long been fixed, although I know Ratchet is a bit older so something about it could somehow affect it still if it's not something in the networking unrelated to the PHP code :) Just wanted to drop it here in case it rings a bell for anyone.

[ulfie22]

@stayallive @Yizen @mpociot as an experiment, I made a Laravel event where I could control the size of the payload that was sent to the laravel-websockets server. On the direction of @mpociot , I put a bunch of log statements in vendor/beyondcode/laravel-websockets/src/HttpApi/Controllers/Controller.php in the ensureValidSignature(Request $request) method. At that point, you can see the $request that the server receives and examine the content. The results of the experiment are that any content payload smaller than 14,110 characters will succeed; anything greater fails. I also logged the payload on the Laravel side (in the event's broadcastWith() method, and it was the full desired length. (There is probably some more magic on the Laravel side before the data gets sent - it must go through the Pusher server - I looked through vendor/pusher/pusher-php-server/src/Pusher.php and didn't see anything that obviously would truncate the data). So - does anyone know where to look on the laravel-websockets server to track the data from where it comes in to Controller.php? It feels like our gremlin is on that path somewhere...

[mpociot]

Alright. I finally figured out what was causing the issue.
The problem is that we (in some cases) do not receive the full HTTP request in a single "message" but split into multiple parts. I am now using the content length header to verify that the incoming request is fully loaded.
This is now fixed in version 1.1.0 🎉

[DougThwaites]

In case it helps anyone I had the same error because I was running my event through a queue (Redis/Horizon) and hadn't restarted the queue worker (php artisan horizon:terminate) since I added the needed environment variables so things like the PUSHER_APP_ID hadn't updated on the queue.

[canhkieu]

Remember enable this line
'secret' => env('PUSHER_APP_SECRET')

[abhijitghogre]

This issue is happening again. I am on version 1.4. For long payload, It fails with the exception "Invalid auth signature provided Exception".