Sentry: Sentry 9.1 - expected an ip address at line 1 column 16

[DeyV]

Important Details

How are you running Sentry?

• [ ] On-Premise docker [Version xyz]
• [ ] Saas (sentry.io)
• [x] Other [docker, latest - 9.1]

Description

After update from 9.0 I have internal error:
Two instances, both with the same problem.

Unknown/api/{project_id}/store/
error
expected an ip address at line 1 column 16

156b69febe4a4ca8b71d71a41cde20b1-symbolicated.txt

and no error is stored in any other (than internal)

[markstory]

Is it possible that you are putting an invalid value into an ip field in the events you are storing?

[DeyV]

I don’t think so.

All events are broken, today I had more than 3k this kind of errors
And on 9.0 all this type of events are logged correctly.

[YtvwlD]

I had the same issue after upgrading to 9.1.0. After looking into the Sentry source code, I came up with the following solution which works for me:

Replace this line in sentry/web/api.py:

    remote_addr = request.META['REMOTE_ADDR']

with this line:

    remote_addr = request.META['REMOTE_ADDR'] or None

That's because in some cases, request.META['REMOTE_ADDR'] doesn't contain a IP address and instead an empty string. And semaphore.processing.StoreNormalizer fails if the IP address is an empty string. None works though.

[markstory]

Closing as this seems to be a configuration error in how you have sentry deployed. Your load balancer should be forwarding or setting that header.

[DeyV]

I think that its not the problem
My proxy automaticly do this, and on 9.0 evrything is ok

[untitaker]

@DeyV because Python is sloppy in handling falsy values compared to Rust. Somewhere you have a proxy wsgi middleware that does this, right?

[DeyV]

I have sentry in docker swarm with traefik and haProxy

But i’m not sure - its problem with env or with error event sended from my (php) application?

Btw I can dump env on this docker to show all them

[untitaker]

If your php sdk sends invalid data it should fail way more gracefully (showing error in UI). I think the env dump will not help. Needs proper debugging

[YtvwlD]

Apparently, my proxy doesn't set this header correctly - but only, if the request is made with IPv4.

But Sentry does save and display the events even without the IP address. I think this would be preferable to failing with this not very descriptive exception.

[YtvwlD]

No, the proxy sets the X-Forwarded-For header correctly. But the IPv4 addresses are passed as ::ffff:<address>. Django < 1.8 seems to have problems with this.

So, indeed this seems to be an issue in Sentry.

[YtvwlD]

I think, I found the problem: it's here:

if ':' in real_ip and '.' in real_ip:
    # Strip the port number off of an IPv4 FORWARDED_FOR entry.
    real_ip = real_ip.split(':', 1)[0]
request.META['REMOTE_ADDR'] = real_ip

... and this returns an empty string on e.g. ::ffff:10.7.0.3.

[untitaker]

That's brilliant, thanks for figuring it out! I would merge a PR that fixes it. I hope it is not too much extra code though (like backporting stuff from newer Django)

[DeyV]

we waiting for release 👍

[DeyV]

Is any chance for release this fix? Bug still exists in 9.1.2
sentry.zip

[chocolatkey]

Can confirm that it's still a bug (self-hosted sentry 9.1.2) that causes events to not ever appear on the event stream (was a source of big mistery) and that monkey-patching proxy.py fixes it, for now. Would very much appreciate an official fix

[BYK]

The fix was first released with Sentry 10.0.0. This is now fixed on all recent versions.