Collect: Introduction of OkHttp changed authorization type over https

[qlands]

Hi,

The authorization type changed from Digest to Authorization-Basic over HTTPS when OKHttp was introduced.

@lognaturel / @yanokwa do you post anywhere potential breaks for 3rd party software that uses ODK Collect (Like Ona, kobotoolbox, etc) when introducing changes?

[lognaturel]

Can you please say more about what you are experiencing? https://forum.opendatakit.org/t/odk-collect-v1-23-beta/20731 is where the change was described and a beta provided. Compatibility with the servers you have listed has been verified. The intent is for basic auth to be a fallback.

[qlands]

There are the headers sent by ODK Collect over HTTPS:

User-Agent-Dalvik/2.1.0 (Linux; U; Android 7.0; MHA-L29 Build/HUAWEIMHA-L29) org.odk.collect.android/v1.23.1
X-Openrosa-Version-1.0
Date-Mon, 05 Aug 2019 09:24:25 GMT+00:00
Authorization-Basic a3RhdmVubmVyOjEyMw==
Host-5e338166.ngrok.io
Accept-Encoding-gzip
X-Forwarded-Proto-https
X-Forwarded-For-201.206.55.96
Content-Length-0

No other is sent. It does not seems to be a fallback. Did the change to OKHttp changed the authorization process?

[lognaturel]

Thank you. That does look unintentional and problematic. I have another issue that I need to prioritize but I think we will be able to fix this within 24 hours.

[qlands]

Thank you.

[yanokwa]

@qlands Do you have a public server that QA could verify this fix against? It'd be good for us to do an end-to-end test so we don't miss anything else.

[qlands]

We traced the error to the authentication of internal URLs in the form list (downloadUrl and manifestUrl):

<xforms xmlns="http://openrosa.org/xforms/xformsList">
<xform>
<formID>LB_Sequia_MAG_20190123</formID>
<name>Sequia_LB_20190123</name>
<majorMinorVersion/>
<version>20190802</version>
<hash>md5:45bcc4e377adeff644a2facef832b3c6</hash>
<descriptionText>Sequia_LB_20190123</descriptionText>
<downloadUrl>
https://5e338166.ngrok.io/user/cquiros/project/prueb/LB_Sequia_MAG_20190123/xmlform
</downloadUrl>
<manifestUrl>
https://5e338166.ngrok.io/user/cquiros/project/prueb/LB_Sequia_MAG_20190123/manifest
</manifestUrl>
</xform>
</xforms>

In the form list url (/formList) we check if your credentials are valid, if not then we request your credentials using Digest with the following headers:

[('WWW-Authenticate', 'Digest realm="[email protected]",qop="auth,auth-int",nonce="95d9227a63c9f978aa6d9ff6720f1103",opaque="8ee0f2c2cbba476db0a8a81fcde4cbee"')]

The screen for user and password appears in ODK Collect. When pressing OK. We receive the following headers from Collect

User-Agent~Dalvik/2.1.0 (Linux; U; Android 7.0; MHA-L29 Build/HUAWEIMHA-L29) org.odk.collect.android/v1.23.1
X-Openrosa-Version~1.0
Date~Mon, 05 Aug 2019 10:51:00 GMT+00:00
Authorization~Digest username="ktavenner", realm="[email protected]", nonce="5fefb999c7f606e67ca0eba2c9de73ff", uri="/user/cquiros/project/prueb/formList", response="f03103e1de2149a357f25116451ccd46", qop=auth, nc=00000001, cnonce="9db389d643a955ef", algorithm=MD5, opaque="8ee0f2c2cbba476db0a8a81fcde4cbee"
Host~5e338166.ngrok.io
Accept-Encoding~gzip
X-Forwarded-Proto~https
X-Forwarded-For~201.206.55.96
Content-Length~0

Then ODK Collect request for the XML file. But ODK collect does not use Digest for that. We receive the following headers from Collect:

User-Agent~Dalvik/2.1.0 (Linux; U; Android 7.0; MHA-L29 Build/HUAWEIMHA-L29) org.odk.collect.android/v1.23.1
X-Openrosa-Version~1.0
Date~Mon, 05 Aug 2019 10:52:11 GMT+00:00
Authorization~Basic a3RhdmVubmVyOjEyMw==
Host~5e338166.ngrok.io
Accept-Encoding~gzip
X-Forwarded-Proto~https
X-Forwarded-For~201.206.55.96
Content-Length~0

Therefore this breaks our authentication process because we assume that ODK Collect will continue to use Digest through out like in 1.22 and previous versions.

@yanokwa you can check the problem with FormShare at https://formshare.qlands.com/app/

[qlands]

Hi @yanokwa, we had to patch our code to allow our user base to download forms. We forced ODK Colllect to send the user and password with Digest by revoking the Authorization:Basic and sending the Digest headers.

These are three things that are critical for 3rd party integrations:

• Authentication process
• XML file structures (FormList, Manifest, and Data file)
• Expected routes.

If your team change any of these in the future it would be ideal to communicate it in a special/specific section of the forum (e.g., 3rd party compatibility issues) so companies like ours can provide input to your QA process.

[seadowg]

@qlands it looks like this was an unintentional change in Collect - we had tried to keep Collect's behaviour identical with the shift to OkHttp but it looks like this change went undiscovered in our tests and QA.

It would be really helpful towards us understanding what happened here to see the headers your server was sending to Collect during the interactions you described. I'm going to take a look at using the server link you posted but I'm guessing you've now deployed the fix.

[qlands]

@seadowg you would be able still to see the headers that our app send to Collect. Basically is:

headers = [
                (
                    "WWW-Authenticate",
                    'Digest realm="'
                    + self.realm
                    + '",qop="auth,auth-int",nonce="'
                    + self.nonce
                    + '",opaque="'
                    + self.opaque
                    + '"',
                )
            ]
            reponse = Response(status=401, headerlist=headers)
            return reponse

With that in a 401 response Collect may request the user and password and send it using Digest.

[seadowg]

Thanks @qlands. I think we have a fix at #3289. Out of interest: what headers was FormScope sending back when it received the Basic auth before your fix?

[qlands]

Before we fixed it we checked if your request came authenticated and if so proceed to process them using Digest. This because previous versions of Collect maybe had a different authorization schema order (Digest, Basic, etc) as you pointed it out at #3289 . If we did not find the authorization headers then we returned and 401 like:

headers = [
                (
                    "WWW-Authenticate",
                    'Digest realm="'
                    + self.realm
                    + '",qop="auth,auth-int",nonce="'
                    + self.nonce
                    + '",opaque="'
                    + self.opaque
                    + '"',
                )
            ]
            reponse = Response(status=401, headerlist=headers)
            return reponse

We patched the code to not assume Digest if the request came authenticated, but check if it is a Digest authentication and if not (like in this issue) then return a 401 with the Digest headers. This will tell collect to send them again but using Digest.

[seadowg]

@qlands right that makes sense. I've built a debug APK from the fix we've put together in #3289 for you to test if you're able to. I know you've already patched the problem but I'm assuming you'll be seeing extra traffic because of having to return extra 401s.

You'll most likely have to uninstall any previous versions of Collect as this APK is not signed with our release key and remember to uninstall and resinstall from the Play Store after
wards so you get updates 😀.

[qlands]

Confirming that the debug APK works fine with the previous authorization process.

[seadowg]

@qlands thanks so much for giving it a go!

[yanokwa]

Hi @qlands, thanks again for reporting the issue. v1.23.3 will be out shortly with @seadowg’s patch.

I wanted to respond to your comments about our process and how you can help.

We communicate all intentional changes and areas of concern in the pre-releases section of the forum. Betas usually run for a week, but are extended if users report problems. This particular release had a month-long beta and we did note that we'd changed all the network communication code. If you are curious, we describe our testing process in the repo readme and the link to the beta program is also there.

As described at opendatakit.org, breaking changes are governed by the Technical Steering Committees. If you follow those links, it’ll take you to the standard operating procedures and from there, you can find upcoming changes for the ODK Suite listed on the roadmap. These changes are discussed in the public recorded calls that are held every two weeks. Maintainers also regularly discuss breaking changes in features and development categories on the forum.

Is this process as I've described it sufficient for your team to help us catch unintentional changes? Is it in the place you’d expect to find it? If not, please share the changes you'd like to see in the development category on the forum. We’d welcome your feedback and would love to have you participation.

[qlands]

Hit @yanokwa , thank you for the information. We will try to integrate all these into our QA and testing process. I will try to keep an eye on the road map.