AdAway 🚀 - Amazon app error when web server enabled (5.0.10)

I am also seeing the same thing. I installed this AdAway version to try to prevent my phone from freezing up, which is a separate issue reported in this github repo.

philipfong on 30 Oct 2020

Workaround is to add *amazon* to your whitelist. That seems to allow the app to function again.

philipfong on 30 Oct 2020

❤2

@philipfong I've also found that whitelisting is a workaround for this issue. Ideally this would be accounted for in some official rule set as this wasn't something I noticed in previous versions of AdAway (<5.0).

Specifically, I use the following settings (though I would prefer to have an even more specific whitelist if possible):

{
    "sources": [
        {
            "label": "AdAway official hosts",
            "url": "https:\/\/adaway.org\/hosts.txt",
            "enabled": true,
            "allow": false,
            "redirect": false
        },
        {
            "label": "Pete Lowe blocklist hosts",
            "url": "https:\/\/pgl.yoyo.org\/adservers\/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext",
            "enabled": true,
            "allow": false,
            "redirect": false
        },
        {
            "label": "StevenBlack Unified hosts",
            "url": "https:\/\/raw.githubusercontent.com\/StevenBlack\/hosts\/master\/hosts",
            "enabled": true,
            "allow": false,
            "redirect": false
        }
    ],
    "blocked": [],
    "allowed": [
        {
            "host": "*.amazon.com",
            "enabled": true
        },
        {
            "host": "*.amazon-adsystem.com",
            "enabled": true
        }
    ],
    "redirected": []
}

lhriley on 30 Oct 2020

👍1

Definitely not an ideal solution. Now I'm seeing a bunch of Amazon ads in a bunch of other apps.

philipfong on 31 Oct 2020

Hi all,

First, have you checked the solution here: #2066 ?

Under no circumstances should an error occur when using the local web server redirection.

Oh yes, there is a lot of way to mess with network and I guess Amazon is doing something weird with their application...

PerfectSlayer on 1 Nov 2020

@PerfectSlayer I hadn't found that specific issue when I searched before creating this one.

However, I did try this from https://github.com/AdAway/AdAway/issues/2066#issuecomment-695767291:

There was report on XDA forum for root ad-block user to disable the web server and change redirect to 0.0.0.0 instead of the default 127.0.0.1 fix the isuse. Can you confirm?

And it seems to function as a workaround.

For clarification I did the following:

Disable the redirection webserver in AdAway.
Set the ipv4 redirection IP to 0.0.0.0.
Uncheck the whitelisted Amazon entries, and apply when prompted.
Force close the Amazon app.
Clear the Amazon app cache.
Launch the Amazon app.
Observe that no error occurs.

I haven't done extensive testing on this yet, these are just my initial findings.

One problem is that when validating this, for whatever reason I never get any entries in the AdAway DNS log. So, it is difficult to know what (if anything) is being caught. The logging issue seems like it might be a separate bug in 5.x

Do you happen to have any insight into how using 0.0.0.0 without the webserver acts differently than 127.0.0.1 with the webserver?

The image placeholder seems like an obvious difference that I would miss if I continue to use the workaround.

lhriley on 1 Nov 2020

One problem is that when validating this, for whatever reason I never get any entries in the AdAway DNS log. So, it is difficult to know what (if anything) is being caught. The logging issue seems like it might be a separate bug in 5.x

For root adblocking method, it won't catch blocked requests if you don't disable adblocking (only allowed requests will appear). I should add a warning in the log UI...

If you setup 0.0.0.0, the webserver won't be used. 0.0.0.0 tells to not connect to a server.
If you setup 127.0.0.1, it will query the webserver.

The latest version of webserver will server an SVG app icon by default. It can be change to serve an empty web page (0 byte) instead.

I don't know what tricks Amazon is doing with its app... but redirecting their traffic to another web server that does not answer as expected broke the app. They might handle no connection error (0.0.0.0) but not the invalid server response (127.0.0.1).

PerfectSlayer on 6 Nov 2020

or root adblocking method, it won't catch blocked requests if you don't disable adblocking (only allowed requests will appear). I should add a warning in the log UI...

Okay, that makes sense. I was really confused as to why I didn't see anything.

Thanks for the clarification on the rest of the topics as well. I'm trying to think about how I could test for the different response types to find out what is really going on.

What webserver are you using internally? Is it something robust or just the app listening on a port and serving back 200 for every request? (I guess I could just look at the code 😉...)

lhriley on 6 Nov 2020

You can check the webserver implementation here: https://github.com/AdAway/AdAway/blob/cdf733393c2f990f6e29812f3853cb85646303fc/webserver/jni/webserver.c#L18

It's based on Mongoose and I linked you the response callback.
It answers a test page for certificate test page, SVG image if icon parameter is set or an empty page otherwise.

PerfectSlayer on 6 Nov 2020

@PerfectSlayer thanks. I'll take a look when I have some time. I'm also trying to find a good way to capture packets to compare a good and bad response.

lhriley on 6 Nov 2020

To follow up, the code for the embedded webserver is a bit beyond my expertise, so I don't really have anything substantive to offer.

I have also been unable to get any sort of man-in-the-middle packet capturing due to limitations in Android 11. From what I do see, though, I found that there is some DRM (care of Microsoft) in the Amazon APK. I suspect that there might be an internal certificate mechanism in place that doesn't play well with the self signed certificate provided by AdAway. While watching requests from the APK, it seems that Amazon contacts a large number of different hosts and when they fail to respond correctly that's when the error message appears.

What is strange is that this is apparently not an issue when blocking the requests outright (not using the local webserver), but only when there is an unexpected response from the local webserver. One approach might be to have a mechanism that allows certain hosts to use the blocking directly and bypass the local webserver, if enabled. Something like a "blocklist for local webserver" which doesn't attempt to respond with an empty page / image. I'm not familiar with AdAway code, so I don't know if this is a realistic approach or not.

lhriley on 16 Nov 2020

👍1

Thanks for your investigation!

One approach might be to have a mechanism that allows certain hosts to use the blocking directly and bypass the local webserver, if enabled

You can achieve this behavior by using the redirect list. For the amazon related domains, simply redirect them to 0.0.0.0 to do not send them to the web server. Could you give it a try?

PerfectSlayer on 18 Nov 2020

You can achieve this behavior by using the redirect list. For the amazon related domains, simply redirect them to 0.0.0.0 to do not send them to the web server. Could you give it a try?

@PerfectSlayer I looked into your suggestion, but it seems that the redirect expects a FQDN and not a regex matcher. So, while in theory this might work, it isn't a manageable solution as there are many Amazon hosts in multiple domains that would need to be included. I could write a script to translate the domains from the blocklist into the JSON for the redirect list and import it into AdAway, but in reality that isn't a solution that any user would find acceptable.

Before I start going down that road... Does the redirect list redirect everything under a named domain? For example, if I were to set foo.com, would it also redirect bar.foo.com?

lhriley on 18 Nov 2020

Yes, it requires a domain name, not a regex. If it's working, I think we will update the AdAway hosts file to use 0.0.0.0 for those domains instead of 127.0.0.1. That will be a better long term solution.

An easy way to test is to create a temporary new hosts file from the hosts file you use with amazon domains only and replace the default 127.0.0.1 by 0.0.0.0. Then add this new hosts file to your AdAway source on top of those you already use (redirections will be used prior to block rules).
You could certainly create this new hosts file with grep, cut and commands.

And sadly no, foo.com is not the same so it wont redirect bar.foo.com 😞

PerfectSlayer on 18 Nov 2020

@PerfectSlayer Okay. So, this is what I ended up doing:

su
sed -r 's|127.0.0.1(.+amazon.+)|0.0.0.0\1|g' /system/etc/hosts | sed -r 's|::1(.+amazon.+)|::\1|g' > /storage/emulated/0/hosts.edited
cp  /storage/emulated/0/hosts.edited /system/etc/hosts
reboot

For whatever reason (probably how Magisk handles mounts) I can't sed -i, which is why I create the temporary file and then copy back to /system/etc/hosts.

However, this does not seem to work with the Amazon app. There doesn't seem to be any issue with the Amazon mobile web page (www.amazon.com).

These modifications seem to work fine when the webserver is disabled. It is only when the webserver is enabled does the Amazon app display an error.

Looking at netstat (I've removed my IPv6 addresses), there doesn't seem to be any difference in the connections that it opens:

Webserver enabled

OnePlus8Pro:/ # netstat -plant | grep amazon     
Active Internet connections (established and servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program Name

tcp        0      0 10.42.4.5:49056         52.46.145.112:443       ESTABLISHED 12382/com.amazon.mShop.android.shopping
tcp        0      0 10.42.4.5:42566         52.46.143.155:443       ESTABLISHED 12382/com.amazon.mShop.android.shopping
tcp        0      0 10.42.4.5:41984         13.227.65.4:443         ESTABLISHED 12382/com.amazon.mShop.android.shopping
tcp6       0   2703 ::ffff:10.42.4.5:49778  ::ffff:52.46.129.15:443 ESTABLISHED 12382/com.amazon.mShop.android.shopping
tcp6       0      0 ::ffff:10.42.4.5:41978  ::ffff:13.227.65.4:443  ESTABLISHED 12382/com.amazon.mShop.android.shopping
tcp6       0      0 x:x:x:x:48800 2a04:4e42:200::272:443  ESTABLISHED 12382/com.amazon.mShop.android.shopping
tcp6       0      0 ::ffff:10.42.4.5:39472  ::ffff:52.94.226.16:443 ESTABLISHED 12382/com.amazon.mShop.android.shopping
tcp6       0      0 x:x:x:x:40828 2a04:4e42:600::272:443  ESTABLISHED 12382/com.amazon.mShop.android.shopping
tcp6       0      0 ::ffff:10.42.4.5:44000  ::ffff:52.94.237.16:443 ESTABLISHED 12382/com.amazon.mShop.android.shopping

Webserver disabled:

OnePlus8Pro:/ # netstat -plant | grep amazon     
Active Internet connections (established and servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program Name

tcp        0      0 10.42.4.5:49056         52.46.145.112:443       ESTABLISHED 12382/com.amazon.mShop.android.shopping
tcp        0      0 10.42.4.5:42566         52.46.143.155:443       ESTABLISHED 12382/com.amazon.mShop.android.shopping
tcp        0      0 10.42.4.5:41984         13.227.65.4:443         ESTABLISHED 12382/com.amazon.mShop.android.shopping
tcp6       0      0 ::ffff:10.42.4.5:41978  ::ffff:13.227.65.4:443  ESTABLISHED 12382/com.amazon.mShop.android.shopping
tcp6       0      0 x:x:x:x:48800 2a04:4e42:200::272:443  ESTABLISHED 12382/com.amazon.mShop.android.shopping
tcp6       0      0 ::ffff:10.42.4.5:39472  ::ffff:52.94.226.16:443 ESTABLISHED 12382/com.amazon.mShop.android.shopping
tcp6       0      0 x:x:x:x:40828 2a04:4e42:600::272:443  ESTABLISHED 12382/com.amazon.mShop.android.shopping
tcp6       0   2746 ::ffff:10.42.4.5:44000  ::ffff:52.94.237.16:443 ESTABLISHED 12382/com.amazon.mShop.android.shopping

However, when I was watching tcpdump -Q out (and with private DNS disabled in Android 11 network settings) I found that with the webserver enabled, the Amazon app spams an IP address with no DNS entries associated with it (according to reverse IP lookups).

This seems to repeat somewhat regularly:

14:23:42.268923 IP 10.42.4.5.39596 > 52.46.155.11.https: Flags [.], ack 28276, win 241, length 0
14:23:42.268951 IP 10.42.4.5.41686 > 52.94.236.45.https: Flags [.], ack 153, win 131, length 0
14:23:42.270542 IP 10.42.4.5.41686 > 52.94.236.45.https: Flags [P.], seq 517:568, ack 153, win 131, length 51
14:23:42.272056 IP 10.42.4.5.41686 > 52.94.236.45.https: Flags [.], seq 568:3488, ack 153, win 131, length 2920
14:23:42.272398 IP 10.42.4.5.41686 > 52.94.236.45.https: Flags [.], seq 3488:4948, ack 153, win 131, length 1460
14:23:42.279303 IP 10.42.4.5.39596 > 52.46.155.11.https: Flags [.], ack 29736, win 247, length 0
14:23:42.438589 IP 10.42.4.5.39596 > 52.46.155.11.https: Flags [.], ack 31196, win 252, length 0
14:23:42.438709 IP 10.42.4.5.39596 > 52.46.155.11.https: Flags [.], ack 32656, win 258, length 0
14:23:42.438799 IP 10.42.4.5.39596 > 52.46.155.11.https: Flags [.], ack 34116, win 264, length 0
14:23:42.438878 IP 10.42.4.5.39596 > 52.46.155.11.https: Flags [.], ack 35576, win 270, length 0
14:23:42.438953 IP 10.42.4.5.39596 > 52.46.155.11.https: Flags [.], ack 37036, win 275, length 0
14:23:42.439185 IP 10.42.4.5.39596 > 52.46.155.11.https: Flags [.], ack 38496, win 281, length 0
14:23:42.439370 IP 10.42.4.5.39596 > 52.46.155.11.https: Flags [.], ack 39956, win 287, length 0
14:23:42.439447 IP 10.42.4.5.39596 > 52.46.155.11.https: Flags [.], ack 41416, win 292, length 0
14:23:42.439517 IP 10.42.4.5.39596 > 52.46.155.11.https: Flags [.], ack 42876, win 298, length 0
14:23:42.439585 IP 10.42.4.5.39596 > 52.46.155.11.https: Flags [.], ack 44336, win 304, length 0
14:23:42.439637 IP 10.42.4.5.39596 > 52.46.155.11.https: Flags [.], ack 45796, win 309, length 0
14:23:42.439855 IP 10.42.4.5.41686 > 52.94.236.45.https: Flags [P.], seq 4948:5088, ack 153, win 131, length 140
14:23:42.519590 IP 10.42.4.5.39596 > 52.46.155.11.https: Flags [.], ack 47256, win 315, length 0
14:23:42.519869 IP 10.42.4.5.39596 > 52.46.155.11.https: Flags [.], ack 48716, win 321, length 0
14:23:42.520389 IP 10.42.4.5.39596 > 52.46.155.11.https: Flags [.], ack 50176, win 327, length 0
14:23:42.520527 IP 10.42.4.5.39596 > 52.46.155.11.https: Flags [.], ack 51636, win 332, length 0
14:23:42.520690 IP 10.42.4.5.39596 > 52.46.155.11.https: Flags [.], ack 53096, win 338, length 0

Unsurprisingly this IP address belongs to AWS:

#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/resources/registry/whois/tou/
#
# If you see inaccuracies in the results, please report at
# https://www.arin.net/resources/registry/whois/inaccuracy_reporting/
#
# Copyright 1997-2020, American Registry for Internet Numbers, Ltd.
#


#
# Query terms are ambiguous.  The query is assumed to be:
#     "n 52.46.155.11"
#
# Use "?" to get help.
#

NetRange:       52.32.0.0 - 52.63.255.255
CIDR:           52.32.0.0/11
NetName:        AT-88-Z
NetHandle:      NET-52-32-0-0-1
Parent:         NET52 (NET-52-0-0-0-0)
NetType:        Direct Allocation
OriginAS:       
Organization:   Amazon Technologies Inc. (AT-88-Z)
RegDate:        2015-09-02
Updated:        2015-09-02
Ref:            https://rdap.arin.net/registry/ip/52.32.0.0



OrgName:        Amazon Technologies Inc.
OrgId:          AT-88-Z
Address:        410 Terry Ave N.
City:           Seattle
StateProv:      WA
PostalCode:     98109
Country:        US
RegDate:        2011-12-08
Updated:        2020-03-31
Comment:        All abuse reports MUST include:
Comment:        * src IP
Comment:        * dest IP (your IP)
Comment:        * dest port
Comment:        * Accurate date/timestamp and timezone of activity
Comment:        * Intensity/frequency (short log extracts)
Comment:        * Your contact details (phone and email) Without these we will be unable to identify the correct owner of the IP address at that point in time.
Ref:            https://rdap.arin.net/registry/entity/AT-88-Z

[...]

With the webserver disabled, tcpdump shows that after a few packets are passed with another AWS IP address then akamai is being spammed instead (this repeats hundreds of times):

14:28:32.771338 IP 10.42.4.5.37894 > a104-68-104-52.deploy.static.akamaitechnologies.com.https: Flags [.], ack 206854, win 1028, options [nop,nop,TS val 1432756450 ecr 2972064528], length 0

Eventually it pushes some data:

14:28:35.220055 IP 10.42.4.5.37894 > a104-68-104-52.deploy.static.akamaitechnologies.com.https: Flags [P.], seq 89370:89630, ack 390542, win 1532, options [nop,nop,TS val 1432758898 ecr 2972066950], length 260

Other interesting requests that seem to happen when the app starts requesting data:

14:27:51.060587 IP 10.42.4.5.41984 > server-13-227-65-4.sfo20.r.cloudfront.net.https: Flags [F.], seq 57932, ack 209421, win 1534, options [nop,nop,TS val 527335795 ecr 213498782], length 0
14:27:51.119376 IP6 x:x:x:x.17632 > x:x:x:x.domain: 22140+ AAAA? msh.amazon.com. (32)
14:27:51.136017 IP6 x:x:x:x.30225 > x:x:x:x.domain: 30796+ A? msh.amazon.com. (32)
14:27:51.157911 IP 10.42.4.5.48054 > 52.94.237.167.https: Flags [S], seq 2935306475, win 65535, options [mss 1460,sackOK,TS val 4270129450 ecr 0,nop,wscale 9], length 0
14:27:51.243181 IP 10.42.4.5.48054 > 52.94.237.167.https: Flags [.], ack 3567377492, win 128, length 0
14:27:51.246041 IP 10.42.4.5.48054 > 52.94.237.167.https: Flags [P.], seq 0:517, ack 1, win 128, length 517
14:27:51.300614 IP6 x:x:x:x.35444 > x:x:x:x.domain: 21767+ AAAA? appx.transient.amazon.com. (43)
14:27:51.332193 IP6 x:x:x:x.27422 > x:x:x:x.domain: 61171+ A? appx.transient.amazon.com. (43)
14:27:51.339958 IP 10.42.4.5.48054 > 52.94.237.167.https: Flags [.], ack 1461, win 134, length 0
14:27:51.341830 IP 10.42.4.5.48054 > 52.94.237.167.https: Flags [.], ack 2921, win 140, length 0
14:27:51.341952 IP 10.42.4.5.48054 > 52.94.237.167.https: Flags [.], ack 4381, win 146, length 0

It is very odd to me that the behavior changes so drastically between the state of the webserver being enabled. I'm pretty much at a loss as to what is happening at this point other than "there's some adblocking detection happening inside the Amazon app". Hopefully at least some of this information is useful.

lhriley on 19 Nov 2020

Device

AdAway version: 5.1.0
Device: OnePlus 6T
Android version: 10 (OxygenOS 10.3.6)
Root version: Magisk 20.4

I can confirm I am also affected by this issue in the same way as this ticket outlines, and also uniquely. It all seems to come down to whether the websever is enabled or not -- if it is disabled the amazon app works immediately.

Funny thing is I'm getting this issue even when using Private DNS (with NextDNS DoT because they do adblocking back to 0.0.0.0 and ::) with the AdAway webserver enabled! Why is it hitting the local webserver when I have both DoT and AdAdaway enabled...

From my linux PC I can confirm the response for a blocked domain (from NextDNS) is:

Non-authoritative answer:
Name:   mads.amazon.com
Address: 0.0.0.0
Name:   mads.amazon.com
Address: ::

As it's supposed to be.

On Android, the local webserver is listening on the appropriate loopback addresses:

netstat -ntlp | grep LIST                                    
tcp        0      0 127.0.0.1:80            0.0.0.0:*               LISTEN      3329/libwebserver_exec.so
tcp        0      0 127.0.0.1:443           0.0.0.0:*               LISTEN      3329/libwebserver_exec.so
tcp6       0      0 ::1:80                  :::*                    LISTEN      3329/libwebserver_exec.so
tcp6       0      0 ::1:443                 :::*                    LISTEN      3329/libwebserver_exec.so

In this case, as it does on most systems, 0.0.0.0 just routes to 127.0.0.1
On linux, you can run a simple 3-line python http.server and curl http://0.0.0.0 to confirm or
On android you can open up termux and curl http://0.0.0.0 and see for yourself...

So, the DNS response I am getting from NextDNS ("Private DNS") is the same or similar to that of what AdAway would produce with its own block lists, and ends up causing it to connect to AdAaway's local http server.

I've tried with the icon enabled/disabled, and with the cert enabled/disabled, and it doesn't have any effect on whether the Amazon app works, in all cases _if the webserver is enabled_ it breaks.

xenithorb on 29 Nov 2020

It all seems to come down to whether the websever is enabled or not -- if it is disabled the amazon app works immediately.

@xenithorb I would agree with this statement.

I use pihole for global DNS adblocking on my home network. This gives me occasional issues with random sites, but it never gives me issues with the Amazon app.

This issue seems very specifically related to something that the embedded webserver does when enabled. Or perhaps routing rules that are added to enable redirection to the webserver?

@PerfectSlayer are you manipulating iptables or some other Android routing mechanism when enabling the webserver?

If anyone reading this has an older version of Android running, I'd be curious if this issue happens before Android 10.

lhriley on 29 Nov 2020

I am not manipulating iptables or any other routing mechanism when enabling the web server, only the hosts file (web server enabled or not).

I wonder if the web server answers a 4xx or 5xx could help the Amazon app?
Right now, it seems we have:

If the server is running, answering a blank page or an issue blocks the Amazon app to launch,
If the server is not running, no answer does not block the Amazon app to launch.

Would you like me to create a branch with a web server that answer an http error to test if the Amazon app can work with?
If it works, I will try to improve it to answer an http error only for Amazon domains.

PerfectSlayer on 30 Nov 2020

Can you check this build and report if it is better? It contains the commit referenced above.
AdAway-webserver-amazon.zip

After installing the app, ensure to manually stop and restart the web server to be sure the right binary is running.

_Note:_ It was signed with the same key as GH releases (ie not the same as F-Droid one).

PerfectSlayer on 30 Nov 2020

@PerfectSlayer Thanks. I'll give your test build a try. How difficult would it be to make the response code configurable? For instance, it might make testing show more clear answers if we could set 503 or 200 or whatever to compare the different behavior.

Update: I might be getting ahead of myself, but unless I'm doing something wrong, that seems to work! @xenithorb can you verify as well?

Update 2: I take it back. I was doing something wrong. I cleared the cache in the Amazon app and it broke again =\

Update 3: After playing around a bit more, I can enable the whitelisting of Amazon domains while the webserver is running and the Amazon app works. I'm going to try to sed the hosts file to change Amazon hosts to 0.0.0.0 and see if I can remove the whitelist entries.

Update 4: Using 0.0.0.0 doesn't help. Whitelisting the following while the test version of the webserver is running does work, though:

*.amazon.com
*.amazon-adsystem.com
amazon-adsystem.com

lhriley on 1 Dec 2020

I would be quite long to create the whole UI settings and the command line parsing. But I come with another idea 😄

I inflate a file called error.txt in /data/user/0/org.adaway/files/webserver which contains the error code to use.
Use an adb (root) shell to change the content of the file then reboot the web server (to parse the file again).
For example:

echo 404 > /data/user/0/org.adaway/files/webserver

Here is the new test build: AdAway-webserver-amazon2.zip and the related commit 1528e6ce71742e24381b719ca9a0569628b35248

Tell me if you can achieve some result with it 😅

PerfectSlayer on 5 Dec 2020

This is awesome, thanks @PerfectSlayer .

I went through all of the common response codes with both a certificate in place and with the certificate removed. They all cause the error. Interestingly some of the codes take longer to display the error than others. That might just be random system timings rather than indicative of a pattern.

I'm curious if there's a way to just not respond to some request. As in no response code, no payload / no empty response, just complete radio silence. Maybe by putting none or something in the error file.

My thought on this is that the traditional hosts based blocking seems to work and there is nothing responding there. Maybe the app has some kind of error checking in place in order to provide some security around transactions. So any unexpected response is considered suspicious.

lhriley on 5 Dec 2020

It's definitely something related to the app. I don't think ad-blocking has something to do with it... But Amazon app seems to be used a lot and I might be worth it to implement some work around in the web server to prevent it to crash...

Another solution could be to use Charles or burp proxies in order to check what is expected by the Amazon app in order to fake payload...

Related resources:

PerfectSlayer on 6 Dec 2020

I'll look into this more when I have some time. I had already tried to find a similar solution via an on-device / app based solution, but modern Android intentionally makes this difficult.

Even one of the solutions you propose appears to be a non-starter:

From https://www.charlesproxy.com/documentation/using-charles/ssl-certificates/ :

Android
As of Android N, you need to add configuration to your app in order to have it trust the SSL certificates generated by Charles SSL Proxying. This means that you can only use SSL Proxying with apps that you control.

Hopefully this doesn't apply to all methods as well.

The burp proxy might be viable based on a quick read of the page. This is probably the method that I'll be looking at when I can.

lhriley on 7 Dec 2020

Yes, that is why I point you this article so you won't spend too much time on a dead end.

But maybe using an emulator with an old Android version could help if you have an Android development environment?

PerfectSlayer on 7 Dec 2020

Okay. So, I had an opportunity to try out the burp proxy, and the results are not conclusive. I know that everything was working as I was able to capture TLS encrypted packets for various apps like Gmail, but if at any point I was trying to capture calls to Amazon domains the app would immediately display the error.

I did find that some requests were going to a site called 5bf24a1c-baa1-5c85-913c-d351e15fd40d.prod.bundlestore.a2z.com which gives a 403 if you try to access it. This seems to me like the app might be downloading a cert bundle at runtime and then only trusts certificates from that bundle. ~~I haven't been able to locate any related files, but I also don't know what I'm looking for.~~

Looks like I spoke too soon. I just found some PEM files in the app data:

OnePlus8Pro:/data/data/com.amazon.mShop.android.shopping/app_ssnap-cert-store/6 # ls -lah
total 13K
drwx------ 2 u0_a256 u0_a256 3.4K 2020-11-13 09:01 .
drwxrwx--x 3 u0_a256 u0_a256 3.4K 2020-04-22 18:55 ..
-rw------- 1 u0_a256 u0_a256 5.2K 2020-04-22 18:55 1574812800_112720_MMDDYY.91a0d7950f0f7512f483ffc31851d1f3.pem.0
-rw------- 1 u0_a256 u0_a256 5.1K 2020-11-13 09:01 1636070400_110521_MMDDYY.a845a23fddeeafaa3e7c702b41e35513.pem.0
-rw------- 1 u0_a256 u0_a256  447 2020-11-13 09:01 journal

Also, while digging through the Amazon APK, I did find references to a debug menu which might be helpful, but haven't been able to figure out how to access it. It looks like you send a deeplink intent to it, but so far I haven't been successful in triggering it.

lhriley on 12 Dec 2020

I found another app where this issue occurs and it gives a legitimate error message.

This is the XDA app from https://play.google.com/store/apps/details?id=com.xda.labs.play

When the webserver is enabled the following error appears after being logged in and navigating to the profile page:

This error disappears after disabling the AdAway webserver.

This is leading me to believe that this is really an HSTS related error.

I'm not sure if it is possible to force disable HSTS in a response from the embedded webserver since the max-age might be a large number and this value supercedes any new values from my understanding of the way HSTS functions.

According to the Mozilla docs this can be disabled by sending a header with a max-age of 0 in the response:

Strict-Transport-Security: max-age=0; includeSubDomains

Additionally it appears if the preload option is set, then this might not be something that can be altered: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security#Preloading_Strict_Transport_Security

Including the header might be a generic fix for this problem in the case that preload is not enabled, however.

lhriley on 20 Dec 2020

Glad you successfully set up a burp proxy!

If the Amazon app is downloading a certificate file, it should be because its http client uses certificate pinning.

5bf24a1c-baa1-5c85-913c-d351e15fd40d.prod.bundlestore.a2z.com

Looks like the app is not only using amazon.com so it will be difficult to distinguish all Amazon domains...

Also, while digging through the Amazon APK, I did find references to a debug menu which might be helpful, but haven't been able to figure out how to access it. It looks like you send a deeplink intent to it, but so far I haven't been successful in triggering it.

You can list all exported activities (the Android application entry points) using dumpsys and start anyone using pm start:

adb shell "dumpsys package | grep com.amazon.mShop.android.shopping"

But I didn't find explicit debug activity. There are some related to deeplink but we don't know what data send to them to trigger the debug menu.

This is leading me to believe that this is really an HSTS related error.

What makes you think it is related to HSTS? It's a server related feature and the AdAway web server does not use it. I would rather think it is a certificate pinning issue. It is a settings on the client side to require specific certificates (not only a valid one according any CA). And as it's a client feature, the server can't do anything about it. This is the kind of feature to you use to prevent man in the middle attack.

PerfectSlayer on 24 Dec 2020

I just wanted to mention that for me all issues with Amazon app were resolved after disabling the web server and leaving the resolution to localhost 127.0.0.1, I never had problems with this setup.

itismo on 29 Dec 2020

@PerfectSlayer Sorry for the delayed response. Holidays and all that...

tldr

My read of all this research leads me to believe that you may be correct that it isn't explicitly HSTS, but the Android method for trusting CAs (post Nougat) seems (to me) conceptually similar to how I've experienced HSTS issues manifest in the past.

It might be that there is simply no way to enable the webserver feature of AdAway unless 1) apps target an older SDK version (increasingly unlikely for maintained apps), or 2) the app developer explicitly trusts the AdAway CA (which seems highly unlikely as it is a security issue).

Or maybe there's just a SAN "misconfiguration" in the AdAway SSL certs and generating a cert with a broader range of hosts might be enough to work around these problems.

(Apologies in advance as this post is a lot longer than I expected it would be...)

What makes you think it is related to HSTS? It's a server related feature and the AdAway web server does not use it. I would rather think it is a certificate pinning issue. It is a settings on the client side to require specific certificates (not only a valid one according any CA). And as it's a client feature, the server can't do anything about it. This is the kind of feature to you use to prevent man in the middle attack.

It might not be explicitly HSTS/PKP, however my suspicion comes from a combination of the error message that the XDA app gives, searching for similar issues with related messages, and the HSTS/PKP information displayed by Chrome when in their query tool (chrome://net-internals/#hsts). Also, since Amazon and XDA are very different organizations, I would find it unlikely that they would produce their apps in a way that would act so similarly. The more likely scenario is there was some security policy in common between them.

forum.xda-developers.com:

static_sts_domain:
static_upgrade_mode: UNKNOWN
static_sts_include_subdomains:
static_sts_observed:
static_pkp_domain:
static_pkp_include_subdomains:
static_pkp_observed:
static_spki_hashes:
dynamic_sts_domain: forum.xda-developers.com
dynamic_upgrade_mode: FORCE_HTTPS
dynamic_sts_include_subdomains: true
dynamic_sts_observed: 1603309479.763398
dynamic_sts_expiry: 1618861479.763393

www.amazon.com:

static_sts_domain: www.amazon.com
static_upgrade_mode: FORCE_HTTPS
static_sts_include_subdomains: true
static_sts_observed: 1606868910
static_pkp_domain:
static_pkp_include_subdomains:
static_pkp_observed:
static_spki_hashes:
dynamic_sts_domain: www.amazon.com
dynamic_upgrade_mode: FORCE_HTTPS
dynamic_sts_include_subdomains: true
dynamic_sts_observed: 1607652374.044239
dynamic_sts_expiry: 1639188374.044235

Alternatively a site that is blocked and doesn't seem to break anything is ads.google.com:

static_sts_domain:
static_upgrade_mode: DEFAULT
static_sts_include_subdomains: false
static_sts_observed: 0
static_pkp_domain: google.com
static_pkp_include_subdomains: true
static_pkp_observed: 1606868910
static_spki_hashes: sha256/IPMbDAjLVSGntGO3WP53X/zilCVndez5YJ2+vJvhJsA=,sha256/YZPgTZ+woNCCCIW3LH2CxQeLzB/1m42QcCTBSdgayjs=,sha256/hxqRlPTu1bMS/0DITB1SSu0vd4u/8l8TjPgfaAp63Gc=,sha256/Vfd95BwDeSQo+NUYxVEEIlvkOlWY2SalKK1lPhzOx78=,sha256/QXnt2YHvdHR3tJYmQIr0Paosp6t/nggsEGD4QJZ3Q0g=,sha256/mEflZT5enoR1FuXLgYYGqnVEoZvmf9c2bVBpiOjYQ0c=,sha256/iie1VXtL7HzAMF+/PVPR9xzT80kQxdZeJ+zduCB3uj0=
dynamic_sts_domain:
dynamic_upgrade_mode: UNKNOWN
dynamic_sts_include_subdomains:
dynamic_sts_observed:
dynamic_sts_expiry:

When I looked at the SSL cert that AdAway uses, I see this:

openssl x509 -in localhost.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            8b:4f:b1:60:8a:0f:61:93
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = localhost
        Validity
            Not Before: May  2 15:02:08 2020 GMT
            Not After : Apr 30 15:02:08 2030 GMT
        Subject: CN = localhost
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:a9:1e:5a:37:b3:3b:86:a0:fa:cb:c4:80:24:5f:
                    e2:4e:cf:d7:00:0f:d1:19:22:ce:06:4a:c7:9e:72:
                    db:83:28:8b:91:21:da:11:2a:ff:df:25:a2:8c:28:
                    48:de:e3:aa:a4:69:b0:8a:ec:99:d2:ab:69:ad:39:
                    1e:64:a8:e3:a6:17:9b:c4:a2:f4:e8:3d:b5:07:05:
                    27:cd:4c:4d:08:d4:9a:ef:38:ae:7d:39:78:97:62:
                    01:fe:77:74:73:e9:1a:87:41:ab:32:d3:71:93:ee:
                    78:e8:05:a5:0f:61:1d:f3:e3:63:ac:f3:81:3e:58:
                    c9:23:56:8a:b6:db:b3:c4:39:b1:ef:8c:57:c3:e3:
                    49:7e:43:25:82:01:a5:19:e4:74:86:68:8c:15:bc:
                    7f:42:8e:db:4a:33:84:0b:2e:f1:96:a1:ce:e4:62:
                    71:6a:40:4c:fa:3b:7a:94:da:b3:44:f8:f1:73:53:
                    1c:24:39:cd:48:0a:2b:34:db:42:28:07:d8:9b:b8:
                    d1:2c:c2:5f:d0:8c:29:62:93:d3:1d:ab:c5:3a:ed:
                    46:1e:78:61:ef:42:19:31:7f:7c:55:38:30:ba:35:
                    d1:3e:8f:78:ad:d4:3e:7a:b0:87:39:d6:9e:46:8c:
                    47:a4:da:ed:a2:70:9a:47:c0:15:ca:14:18:d6:ec:
                    30:d9
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name: 
                DNS:localhost, DNS:*.doubleclick.net, DNS:*.g.doubleclick.net, IP Address:127.0.0.1
            X509v3 Key Usage: 
                Digital Signature
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication
            X509v3 Basic Constraints: 
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         a7:3d:64:d2:7c:39:83:e8:e7:c3:18:10:ce:de:dc:cf:a0:a0:
         3c:9e:82:1e:2c:e9:92:3a:ba:d9:93:3f:0b:ed:f7:75:6b:46:
         52:ff:fe:b2:c5:2c:bd:a7:f1:8c:ee:7f:55:68:0e:fd:3c:02:
         79:a9:8c:30:b4:41:b1:45:8d:b2:68:1f:f6:33:7b:47:ca:fe:
         2c:98:05:d3:00:37:7c:25:e5:51:3e:1f:8c:44:6a:d2:9d:2c:
         bc:52:b9:45:9c:29:2f:6b:a4:2b:22:a4:c4:3a:6c:41:a8:a2:
         6a:c3:56:7c:ae:1a:39:72:8c:53:1e:e5:a2:3d:81:49:1f:c4:
         a3:ef:4a:4c:07:fd:13:85:0f:73:69:04:8c:c5:f7:d1:de:2a:
         12:be:eb:05:31:48:d8:f5:ea:4c:e3:f2:89:62:28:dd:2d:14:
         f0:e0:3b:37:f3:69:f5:8d:16:6a:91:6c:8d:7a:4a:03:77:8e:
         2b:73:e0:04:2a:e5:55:27:8a:df:fb:e9:bf:b9:69:77:1b:76:
         e8:3f:5a:7b:91:46:35:fc:3a:e2:59:df:18:22:c2:8d:43:f5:
         17:98:e3:e9:37:f7:6e:27:9f:26:8f:cb:d7:cf:c2:56:ea:2e:
         14:44:e5:a6:76:cc:c3:6f:ac:e2:2d:3b:65:a1:6c:df:26:1f:
         03:54:94:05

Specifically SAN part:

X509v3 Subject Alternative Name: 
                DNS:localhost, DNS:*.doubleclick.net, DNS:*.g.doubleclick.net, IP Address:127.0.0.1

I'm wondering if this is not inclusive enough, though, I'm not sure what the limits are for SAN names. For instance, I wonder if you could use * as a hostname. This seems unlikely, but I didn't find any documentation saying that it wasn't allowed.

Here's an interesting related article about certs with very large lists of SAN entries: https://discuss.httparchive.org/t/san-certificates-how-many-alt-names-are-too-many/1867

I tried to generate my own CA and certs. I got the CA to install correctly, but when I replaced the AdAway certs in /data/data/org.adaway/files/webserver, the webserver wasn't happy about it. This might just be my limited understanding of how the embedded webserver expects these files to be formatted, etc.

Additionally, I think I tracked down what might be the XDA app error page in the react-native-webview project. It looks identical to the error page in all of the related issues that I looked at and shows up when searching for the exact error message: https://github.com/react-native-webview/react-native-webview/blob/194c6a2335b12cc05283413c44d0948eb5156e02/android/src/main/java/com/reactnativecommunity/webview/RNCWebViewManager.java#L918-L920

This then links to https://developer.android.com/reference/android/net/http/SslError#SSL_UNTRUSTED for the actual error class and https://developer.android.com/reference/android/webkit/WebViewClient#onReceivedSslError(android.webkit.WebView,%20android.webkit.SslErrorHandler,%20android.net.http.SslError).

When doing some additional research today with the SSL_UNTRUSTED error I came across this thread which has a few suggestions that seem plausible: https://stackoverflow.com/questions/50544113/android-webview-get-sslerror-ssl-untrusted-but-certificate-is-valid

Specifically someone links to this page: https://android-developers.googleblog.com/2016/07/changes-to-trusted-certificate.html

User-added CAs
Protection of all application data is a key goal of the Android application sandbox. Android Nougat changes how applications interact with user- and admin-supplied CAs. By default, apps that target API level 24 will—by design—not honor such CAs unless the app explicitly opts in. This safe-by-default setting reduces application attack surface and encourages consistent handling of network and file-based application data.

This page also helped me find which seems would apply only to requests in AdAway proper: https://github.com/AdAway/AdAway/blob/4359d6d7a436af9d96cc4212b75ecdda42ca47d6/app/src/main/res/xml/network_security_config.xml

Some additional references to issues appear to be discussed in the XDA app forum thread: https://forum.xda-developers.com/t/check-out-the-new-xda-app-let-us-know-what-you-think-app-rolling-out-now.4199599/page-38#post-84166545

Unfortunately it seems that XDA might not be developing their current app in the open as they don't seem to be looking for contributors and I can't find any related code repositories to examine.

lhriley on 30 Dec 2020

@lhriley My turn for the late answer 😋

It might be that there is simply no way to enable the webserver feature of AdAway unless 1) apps target an older SDK version (increasingly unlikely for maintained apps), or 2) the app developer explicitly trusts the AdAway CA (which seems highly unlikely as it is a security issue).

Installing the AdAway CA would allow bad people to easily alter content of the domain the certificate signed.
That is why I only set localhost (which could be an issue on PC for example with program running localhost servers but should be quite safe for Android) and Google Double Click domain.

I'm wondering if this is not inclusive enough, though, I'm not sure what the limits are for SAN names. For instance, I wonder if you could use * as a hostname. This seems unlikely, but I didn't find any documentation saying that it wasn't allowed.

I am not sure either but I quickly encounter issues with wildcard.
For example, I can't ad top level domain with wildcard (ex. .google.com) to match all subdomains at any level (.google.com will match aaa.google.com, bbb.google.com but not xyz.ccc.google.com. I need to add *.ccc.google.com.

About the number

I tried to generate my own CA and certs. I got the CA to install correctly, but when I replaced the AdAway certs in /data/data/org.adaway/files/webserver, the webserver wasn't happy about it. This might just be my limited understanding of how the embedded webserver expects these files to be formatted, etc.

There is really nothing special about it.
It requires two files:

localhost.crt the certificate itself
localhost.key the certificate private key

Both files must be located in /data/data/org.adaway/files/webserver folder.
You will need to run the web server once to create the files (they are copied from apk internal assets).
Check the permissions of the files you copied. Owner and group are the id of the app 600 permissions.

When doing some additional research today with the SSL_UNTRUSTED error I came across this thread which has a few suggestions that seem plausible: https://stackoverflow.com/questions/50544113/android-webview-get-sslerror-ssl-untrusted-but-certificate-is-valid

Yes, you are right. Adding the CA into the system is no more enough for app to trust them.
You also need to declare the CA or certificate in the network_security_config.xml of all apps that must trust your CA...

Would you like we test to add some Amazon ad/tracker domains to the AdAway certificate and see how Amazon app behaves?
Do you need my help to create a such build if you can't install the certificate you create?

Oh, and I totally forget: _Happy new year!_ 🎉 😉

PerfectSlayer on 17 Jan 2021

I am not sure either but I quickly encounter issues with wildcard.
For example, I can't ad top level domain with wildcard (ex. .google.com) to match all subdomains at any level (.google.com will match aaa.google.com, bbb.google.com but not xyz.ccc.google.com. I need to add *.ccc.google.com.

Yeah, this was also something that I was concerned about. There's just no good way to handle multiple levels of subdomains in any predictable way.

There is really nothing special about it.
It requires two files:
localhost.crt the certificate itself
localhost.key the certificate private key
Both files must be located in /data/data/org.adaway/files/webserver folder.
You will need to run the web server once to create the files (they are copied from apk internal assets).
Check the permissions of the files you copied. Owner and group are the id of the app 600 permissions.

Hmm. It has been a while now since I tried it, but I feel like I did all of this as described. Maybe I generated the cert / private key incorrectly. Can you either give some example commands for how you generate the CA / cert / key combination?

Would you like we test to add some Amazon ad/tracker domains to the AdAway certificate and see how Amazon app behaves?
Do you need my help to create a such build if you can't install the certificate you create?

I haven't done a build for this project before. I assume that it would be pretty straightforward, though. I'm not sure when I would have the time to tackle it at the moment.

Alternatively, it would be cool to have a file like the error code one you added for testing purposes which would accept a list of hostnames that the cert generator could include when generating a certificate? That way I could test adding hostnames in various combinations until I see all of the known bad apps working correctly (or not) to determine if this is a viable solution.

Also, Happy New Year to you as well ^_~

lhriley on 17 Jan 2021

Can you either give some example commands for how you generate the CA / cert / key combination?

You know what? You scared me a lot with this question!
I pushed the certificate just before I build my new computer and I though I lost all the knowledge around it...
Hopefully, I find I stored all the details in the repository itself: https://github.com/AdAway/AdAway/tree/master/Resources/certificate

I haven't done a build for this project before. I assume that it would be pretty straightforward, though. I'm not sure when I would have the time to tackle it at the moment.

Two options for you:

If you are an android dev and you have a SDK/NDK install, it will be quite easy: https://github.com/AdAway/AdAway/blob/master/CONTRIBUTING.md#building-the-project
If you don't have a built environment, just fork the project and let GitHub build it for you 😉 I set up all the workflow to do everything by itself.

Alternatively, it would be cool to have a file like the error code one you added for testing purposes which would accept a list of hostnames that the cert generator could include when generating a certificate?

It would require to have the openssl CLI on the Android device. Which sadly is not available 😞

PerfectSlayer on 19 Jan 2021

You know what? You scared me a lot with this question!
I pushed the certificate just before I build my new computer and I though I lost all the knowledge around it...
Hopefully, I find I stored all the details in the repository itself: https://github.com/AdAway/AdAway/tree/master/Resources/certificate

I know that feeling. Luckily past you had the foresight to solve this problem. ^_~

I got the SDK/NDK up and running. It has been a while since I last needed it. The project seems to compile, so that's good. I'm ot sure that I will actually need it though, since I really just need to generate a new certificate and put it in the correct location for testing. We'll see how it goes.

It would require to have the openssl CLI on the Android device. Which sadly is not available disappointed

Hmm. I hadn't considered that. I assumed there was a library available that could be used to generate the certificates within Java / Kotlin.

lhriley on 22 Jan 2021

Quick update. I tested this and it still seems that Amazon is unhappy.

[dn]
CN=localhost
[req]
distinguished_name = dn
[EXT]
subjectAltName=@alternate_names
keyUsage=digitalSignature
extendedKeyUsage=serverAuth
basicConstraints=CA:true
[alternate_names]
DNS.1=localhost
DNS.2=*.doubleclick.net
DNS.3=*.g.doubleclick.net
DNS.4=*.amazon.com
DNS.5=amazon.com
DNS.6=*.amazon-adsystem.com
DNS.7=amazon-adsystem.com
DNS.8=*.xda-developers.com
DNS.9=xda-developers.com
DNS.10=a2z.com
DNS.11=*.a2z.com
DNS.12=prod.bundlestore.a2z.com
DNS.13=*.prod.bundlestore.a2z.com
DNS.14=transient.amazon.com
DNS.15=*.transient.amazon.com
DNS.16=*.deploy.static.akamaitechnologies.com
IP.1=127.0.0.1

XDA worked, which I thought was encouraging, but then I restored the certificates that shipped with AdAway to verify, and XDA continued to function, so I think they may have fixed something on their end with a recent update (they were getting review bombed with many users mentioning AdAway breaking their app and a variety of other bugs). That's both good and bad as it removes a second example to test against.

Something that will be really annoying to test would be to parse the host file that is generated by AdAway and use those hostnames to produce a list of SANs for everything matching Amazon domains. I don't have the brain power to tackle this particular problem at the moment, however.

lhriley on 22 Jan 2021

Adaway: Amazon app error when web server enabled (5.0.10)

Most helpful comment

All 35 comments

Device

Related issues