Sentry-ruby: REMOTE_ADDR behind a proxy

[troelskn]

I'm proxying requests through nginx to my rails app, which means that REMOTE_ADDR will always be 127.0.0.1. That's not overly helpful to have show up in the overview section within Sentry. Would it be possible to mend raven-ruby to pass X-Real-IP along as an environment variable, if present?

[nateberkopec]

@troelskn Ever figure this out? I never roll my own server setups so I have no idea how to help you here.

[troelskn]

I totally forgot about this. It looks like I'm getting the right IP's in sentry though, so either I did some patching that I've since forgot about or someone changed sentry in the meanwhile. I'll see if I can figure out which it is.

[nateberkopec]

FWIW, you can always use context to add this in:

Raven.extra_context real-ip: request.headers["X-Real-IP"]

[seanlinsley]

I'm seeing this issue with NGINX. Here are the headers passed to my Rails app (1.2.3.4 being a real, correct IP)

{
  "REMOTE_ADDR" => "127.0.0.1", 
  "HTTP_X_REAL_IP" => "1.2.3.4",
  "HTTP_X_FORWARDED_FOR" => "1.2.3.4",
}

It looks like raven-ruby sends all headers to Sentry, but Sentry only considers REMOTE_ADDR when determining the IP.

[seanlinsley]

The Sentry server itself correctly uses X-Forwarded-For, so it's odd that when processing events it only looks at REMOTE_ADDR. I found that commit by reading through https://github.com/getsentry/sentry/issues/1254.

[dentarg]

it's odd that when processing events it only looks at REMOTE_ADDR

I agree.

On Heroku, REMOTE_ADDR is the internal IP, and the real user IP is in X-Forwarded-For. I think it would be useful if Sentry was smart about this.

[dcramer]

We cant make assumptions about X-Forwared-For as its not secure to just use that as the IP when its present. The Sentry server generally uses an Nginx module and is configured based on the installation to simply set the correct IP.

[seanlinsley]

For the life of me, I can't seem to get that NGINX module to rewrite REMOTE_ADDR. I settled on this monkeypatch instead:

Raven::HttpInterface.prepend Module.new {
  def from_rack(env)
    super
    self.env['REMOTE_ADDR'] = env['action_dispatch.remote_ip'].to_s
  end
}

This works by using remote_ip, which is generated by filtering out private IP addresses, and also attempts to detect spoofing. The only obvious difference between it and the NGINX module is that the list of filtered IPs can't be changed without modifying the constant.

I would suggest that Raven use this value when used in a Rails project, since Rails has already done all the work for you, and it makes it so everyone who uses Raven behind a proxy server like NGINX locally and those who use Heroku automatically get the correct IP address showing up on their Sentry events.

I'd be happy to create a PR if we can agree on the change.

[dcramer]

@seanlinsley I think this makes sense if it's a "this is true by default in Rails". I'm not sure if we'll need a way to override it or not, but generally I'm always on the side of the best defaults as long as they dont cause security, privacy, or other data consistency issues.

[EricCarlstrom]

@dcramer I struggle to understand the hard and fast rule against logging proper client IP's if coming from a trusted source. Are you saying it's just not possible, or is there an actual better solution.

@seanlinsley I like your suggested solution on paper, but for a relative newb like myself, could you provide more direction on where one would best put the Raven::HttpInterface.prepend snippet you've included?

Our stack involves ELB + nginx + unicorn, and I'm hopeful this is the missing piece to getting complete exception reports.

Thanks kindly in advance.

[dcramer]

@MixedRice the problem is the trusted location bit -- theres no way to _know_ that the header is trusted unless you explicitly tell us it is.

i.e. I could send it to any webserver as a custom header, and unless they've explicitly said "I know this is the IP sent from my load balancer, and I know it wasnt a custom header" it's not safe.

[dcramer]

Reopening this since the server itself actually does a bunch of things with REMOTE_ADDR. We should aim to provide a simple way to specify it for users.

I dont think any client actually deals w/ this at the moment, though in things like Django it's easy to work around.

I think it might be enough to have something like config.use_x_forwarded_for.

https://docs.djangoproject.com/en/1.9/ref/settings/#std:setting-USE_X_FORWARDED_HOST

[dcramer]

see also getsentry/raven-java#164

[EricCarlstrom]

Totally get it... I guess in my head, it comes down to three choices:

1) For those using proxies, the user IP is simply unavailable per the inherent limitations of REMOTE_ADDR

2) Replicate some of the set_real_ip_from logic used in nginx, in an effort to adopt common practices around using trusted headers - likely an optional config.

3) I'd throw out a third option - simply letting Raven consumers manually intervene and explicitly set the client ip, similar to how they set user_context... offloading any logic, integrity, etc to the consumers of the library.

[dcramer]

@MixedRice that might make sense just to provide #3 and have it be the final say. I would be open to adding this to our client specification and getting that adopted throughout, as it's much simpler than most other things. We may still want to provide something like "use_x_forwarded_for", but t hats tangential.

[dcramer]

getsentry/sentry#2434

[EricCarlstrom]

:+1:

[davedash]

FYI, my workaround for this was to place in my initializer:

module Raven
  class Processor::Custom < Processor

    # Override the rack provided environment with user defined attributes.
    # See application_controller.rb.
    def process(value)
      if value['request']
        value['request']['url'] = value['user'].delete(:url)
        value['request']['env']['REMOTE_ADDR'] = value['user'].delete(:ip)
      end
      value
    end

  end
end

# https://github.com/getsentry/raven-ruby/issues/136#issuecomment-143000465
Raven.configure do |config|
  config.processors << Raven::Processor::Custom
end

In my application controller I do the following:

  before_action :set_raven_context

  def set_raven_context
    context = {
      url: request.original_url,
      ip: request.ip
    }
    Raven.user_context(context)
  end

It's not perfect, but it works for now.

[NobodysNightmare]

@davedash sentry-raven does support setting the :ip_address in the user_context, which will basically have the same effect as your custom processor.

@dcramer Since we needed support for X-Forwarded-For for many of our services, we built a (very very simple) middleware that passes the right IP to Sentry, without changing the rest of the user_context. If anyone is interested: https://rubygems.org/gems/sentry_real_ip.

[nateberkopec]

PR in #546, looking for feedback.

I'll note that @NobodysNightmare is right, shortly after the opening of this issue, Sentry allowed you to set the :ip_address in the user context, which overrides REMOTE_ADDR when determining an Event IP.

[nateberkopec]

Closed by #546.