Apm: Aligning on context.request.socket.remote_address

[eyalkoren]

Description of the issue

Some agents (Python and maybe .NET) send the immediate connection remote IP obtained from the incoming HTTP request in context.request.socket.remote_address and some (Go and Java) parse headers (like Forwarded and X-Forwarded-For) to find the origin of the request route.

What we are voting on

Following a discussion, it was agreed that all agents should send the remote IP of the immediate socket in the intake API context.request.socket.remote_address field.
No change is required currently by the APM server that already uses the value of this field to store in ES as both client.ip and source.ip.
If the need arises, other IPs on the route can be parsed from the values of headers already sent by all agents in the APM server or any other backend entity.
No change at the moment for the default configurations related to header capture.

Vote

@elastic/apm-agent-devs please check what is it you send currently and link to the issue if it requires work, or mark as done.

| Agent | Done | N/A | Link to agent issue
| --------|:----:|:---:|:-----------:|
| .NET |

• [x]

|

• [ ]

|
| Go |

• [x]

|

• [ ]

| https://github.com/elastic/apm-agent-go/issues/659
| Java |

• [x]

|

• [ ]

| https://github.com/elastic/apm-agent-java/issues/886
| Node.js |

• [x]

|

• [ ]

|
| Python |

• [x]

|

• [ ]

|
| Ruby |

• [x]

|

• [ ]

| https://github.com/elastic/apm-agent-ruby/issues/583
| RUM |

• [ ]

|

• [x]

|

[felixbarny]

The remote address might be the address of the local load balancer as opposed to the user's IP. Is that how it's supposed to be and if yes, why?

[axw]

@felixbarny yes, that's how it's supposed to be. The reason is so we don't allow untrusted sources to spoof the source address. This relates to SIEM integrations.

As the agents send request headers, the server can identify the originating client address in a unified manner, with per-service configuration to determine whether the source is or is not trusted.

[mikker]

The Ruby agent tried to parse forward headers too (via a lib). Is changing this a (data) breaking change?

[axw]

@mikker IMO it's not, as we haven't previously specified its definition. If it's undocumented, it's undefined.

[Qard]

Node.js already works this way, so nothing to do here.

[eyalkoren]

Updated the table at the top based on what we know, please correct it if I was wrong:
@mikker I unmarked the done column, as I believe there is only the issue with no implementation.
@beniwohli I marked as done for Python based on your feedback.
@gregkalapos Did you get the chance to check?

[gregkalapos]

@gregkalapos Did you get the chance to check?

I did some reading on this and I think we are good here in .NET - the API that we use returns the remote IP of the immediate socket.

You could do some configuration to make the framework read the X-Forwarded header and use that in the property we read, but that'd be applied after the agent reads the remote address. At the point when we read the value, it's not yet applied. So 👍 from me too.

[mikker]

Forgot to check when closing the Ruby issue which was also fixed. Seems like every agent is done?