Reqwest: DNS bypass

Created on 8 Jul 2019 · 13Comments · Source: seanmonstar/reqwest

It would be nice if there were a way we could bypass DNS resolution and connect directly to an IP address we specify during the request. Something like...

client.skip_dns('127.0.0.1').get("https:/google.com").send()

SNI / cert validation could still work off the domain in the URI.

rfc

Source

frenchtoastbeer

All 13 comments

That seems like a fair desire, unsure about the best API to expose.

seanmonstar on 8 Jul 2019

So I toyed with this a little by modifying the async_impl/request struct like:

pub struct Request {
    ip: Option<std::net::Ipv4Addr>,
    method: Method,
    url: Url,
    headers: HeaderMap,
    body: Option<Body>,
}

and then tried compiling / updating over and over again, making the necessary changes at each step to get past whatever errors were introduced. I tried to stay true to how the rest of the library treated an instance of a Request at each step, but I got stuck at...

C:\dns_bypass (master -> origin)λ cargo build
   Compiling hyper v0.12.27 (C:\dns_bypass\.cargo\hyper)
error[E0308]: mismatched types
   --> .cargo\hyper\src\proto\h1\dispatch.rs:464:47
    |
464 |                         Ok(Async::Ready(Some((ip, head, body))))
    |                                               ^^ expected struct proto::MessageHead, found struct http::Ip
    |
    = note: expected type proto::MessageHead<proto::RequestLine>
               found type http::Ip

I didn't think it should be merged into request "headers" as its rather part of the IP header, but taking this approach meant I needed to modify the h2, hyper, and http libraries as well, and possibly more?

frenchtoastbeer on 8 Jul 2019

Before you go to far down that rabbit hole, I think it'd be best to think about the exposed API to the user. The implementation likely doesn't need to bother touching h2/hyper/http, and would probably just adjust reqwest's connect.rs stuff...

seanmonstar on 8 Jul 2019

Ahh. Well it's too late for me I already went down the rabbit hole! Anyhow, I think you're right it's best to avoid changes on any dependent libraries. I'll try again sometime this week hopefully and report back.

For the API, I already don't like .skip_dns() because "DNS" is just one of the ways by which an IP could be resolved. I do like that the default behavior is to attempt resolution, since I believe the users of reqwest would prefer not to worry about it.

Perhaps skip_resolution would be better? I still like 'skip', since it implies that the standard behavior is to attempt IP resolution, and shows that by calling that function, no subsequent resolution would occur.

I don't think support for non-IP protocols would be anything in-scope for this library, so accepting an std::net::IpAddr would be best?

Having anything more than a .skip_resolution(std::net::IpAddr) builder method to support this behavior, I think, would be unnecessary.

frenchtoastbeer on 8 Jul 2019

Does curl and other libraries provide an easy option to do this?

seanmonstar on 8 Jul 2019

curl has --resolve <host:port:address> Resolve the host+port to this address
wget does similar by over-riding the host header, but I imagine that isn't going to be compatible with SNI?

Now that I'm looking at it, I can see why curl would have a --resolve as opposed to a skip_resolution, as it would matter with redirects. If you skip resolution on the first host, but then get redirected, do you then use DNS for the redirect if it goes to a new host?

In this case, you'd want to simply set up a resolution mask where host A maps to IP A, and let all other names resolve normally. In curls example, they are allowing host A to map to IP A, only on a specific port, otherwise use DNS as normal (I think even for host A on a different port).

That seems a bit more complex than what I was envisioning, where I would simply want to establish a static Host->IP resolution, ignoring the port.

Now that I've seen how curl does it, I think I'd want to leave DNS resolution "on" for any names that weren't already defined, and have a .resolve(&str, std::net::IpAddr) to define name->ip's that would be usable prior to issuing each request. Each .resolve() would append to an internal dns cache that would then be checked prior to doing DNS resolution. I anticipate that there wouldn't be so many DNS over-rides that one would want .resolve() to accept an array of names->IP's.

Since the default resolver is already replaceable with a custom one, maybe all this is really a bogus feature request? All I'd need to do is figure out how to mimic the default resolver and then add my very basic "mask" in front of that resolver for an early return. I'm not sure how to do that, but already it seems that would be a more acceptable direction to take, rather than have all reqwest requests perform a dns cache check for something that probably doesn't exist 99% of the time.

frenchtoastbeer on 8 Jul 2019

👍1

Maybe I was confused, I'm looking now at the reqwest source and I don't see how to set my own resolver on either the client or the request. Was that part of hyper?

frenchtoastbeer on 9 Jul 2019

@seanmonstar Can you reopen this issue? Specifying custom DNS resolution is quite helpful.