Ferdi: Cannot validate URL of Custom Website service

[keunes]

Describe the bug
I enter the url of my personal nextcloud instance, accessible over web, but I get an error message in Ferdi (Custom Website) recipe saying like 'Could not validate server of Custom Website.' (translated from Dutch here) when trying to save the service.

Expected behavior
I'm not sure what it's checking to validate, but if I can open the website in a browser I expect it to accept my url and save the service.

Desktop (please complete the following information):

• OS: Ubuntu 18.04.3
• Version: 5.4.0

[kytwb]

Can you give us the (redacted if needed) URL you're trying to add please?

[vantezzen]

This sound like it is probably a problem in the way the nextcloud recipe implements the URL check.

[keunes]

Can you give us the (redacted if needed) URL you're trying to add please?

https://raspberry.keunes.com/nextcloud/index.php/apps/contacts (this is redacted; base domain doesn't exist)

[vantezzen]

Have you tried using "https://raspberry.keunes.com/nextcloud" and "https://raspberry.keunes.com/"?

[keunes]

"https://raspberry.keunes.com/nextcloud"

Doesn't work either.

"https://raspberry.keunes.com/"

Works, but that's just a very simple static html page.

[kytwb]

Here is how custom website services URLs are validated; can you double check the initial request to the URL you're trying to add is returning a 2xx or 3xx HTTP status code?

[keunes]

Here is how custom website services URLs are validated

I can't code, so I don't know what that does :)

can you double check the initial request to the URL you're trying to add is returning a 2xx or 3xx HTTP status code?

In Firefox, using the Developer > Network tools I get indeed 302 for https://raspberry.keunes.com/nextcloud and then 200 for https://raspberry.keunes.com/nextcloud/index.php/login

[keunes]

I have saved a custom website service for the AntennaPod group I help manage. Whenever I update the address to https://groups.google.com/forum/#!forum/antennapod as the adjusted address, after (seemingly succesfully) saving the settings, closing the settings page, and reopening them, the address is (re)set to https://groups.google.com/forum. This means that I have to take every time an extra step to reach the group messages (apart from that it's also frustrating).

@kytwb / @vantezzen, maybe you can tell if it's/could be related?

[kytwb]

I think it's because of:

https://github.com/getferdi/ferdi/blob/335b5898bbfa85486d7cd0f2cea1b527490ba2ca/src/components/settings/services/EditServiceForm.js#L172

The call to normalizeUrl might be the culprit.

Edit: Apparently normalizeUrl doesn't strip hash by default, so maybe a false lead.
Edit 2: I see we're using an outdated version of normalizeUrl:

ferdi % cat package.json | grep normalize
    "normalize-url": "^1.9.1",

Current version is v5.0.0 so the default might have changed. Probably a good lead in the end.

[kytwb]

Upgrading normalize-url produce the following result:

Before save: https://groups.google.com/forum/#!forum/antennapod
After save: https://groups.google.com/forum#!forum/antennapod

[kytwb]

Adding removeTrailingSlash: false to https://github.com/getferdi/ferdi/blob/6ae97425b0c0da181a9ec26e52535ceadd1f4e46/src/models/Service.js#L149 and https://github.com/getferdi/ferdi/blob/335b5898bbfa85486d7cd0f2cea1b527490ba2ca/src/components/settings/services/EditServiceForm.js#L172

seems to fix it.

[kytwb]

@keunes I ran raspberry.keunes.com/nextcloud/index.php/apps/contacts through the old version of normalize-url and it returns the right value (I suspected it may return something like raspberry.keunes.com/nextcloud/index.phpapps/contacts). I'm not sure what the problem is/was with that URL; were you using a proxy to access this URL in Firefox?

[kytwb]

Seems similar to https://github.com/meetfranz/franz/issues/1681, https://github.com/meetfranz/franz/issues/1517 and https://github.com/meetfranz/franz/issues/1479.

@keunes you said earlier:

base domain doesn't exist

Can you tell me what it looks like? We may want to stop normalizing URL for Custom Website services?

[keunes]

Hi @kytwb
Thannks for looking into this!

Adding removeTrailingSlash: false seems to fix it.

Happy to hear :) Is that something I can do on my local installation (Ubuntu) to try it out?

Can you tell me what it looks like?

Not sure if that's what you're asking, but if you want to know the real url I can create a separate, private issue (?)

We may want to stop normalizing URL for Custom Website services?

I don't know what magic is performed exactly, but the issue with my Nextcloud address also exists with the relatively basic url https://raspberry.keunes.com/nextcloud So I'm not sure stopping normalisation would help in this case?

[kytwb]

Not sure if that's what you're asking, but if you want to know the real url I can create a separate, private issue (?)

I think you can share it here as it's related to the issue. I need to know what is the value of raspberry.keunes.com you are really using to be able to make sense of it as you said the base domain doesn't exist; if the problem is in the base domain, raspberry.keunes.com/nextcloud will not work either.

[keunes]

I think you can share it here as it's related to the issue

I don't want to publicly reveal the domain ;) I thought it was possible to create a private issue (hence the proposal to create a new one). Maybe I can provide you the details 'in person' instead (mail, Matrix, ...)?

if the problem is in the base domain, raspberry.keunes.com/nextcloud will not work either

Sorry, keunes.com is a fake domain, so is the subdomain I pasted here. However, as noted:

"https://raspberry.keunes.com/" Works, but that's just a very simple static html page.

keunes.com would be a completely different thing, hosted by a professional provider (not a RPi)

[kytwb]

Feel free to reach out directly to [email protected] for the data you want to share privately.

[kytwb]

What happened with @keunes URL is that the fetch in the recipe URL validation fails with ERR_TOO_MANY_REDIRECTS. Removing this line fixes it but I don't know why it was done like this in the first place. I think we should just check if the status code starts with _2_ and maybe tell fetch to follow redirects (checking if the status code starts with _3_ still doesn't let us know if the URL in the Location header of the response will respond with a 2xx, in the case of a redirect). WDYT @vantezzen?

[vantezzen]

We could also not follow redirects and simply check if the status code doesn't start with a 4, i.e. there is not a user error.

[kytwb]

@vantezzen actually fetch follows redirect by default; I got mislead by that 3xx check. Checking if the status is 2xx is enough imo. Easier than handling 3xx, 4xx, 5xx. I have submitted a pull request to update the recipe.

[vantezzen]

We may actually want to allow 3xx and 5xx codes too:

• 300 will sometimes get returned from the server with a list of different files you can go to. A user might actually want to have this page in Ferdi and is wondering why Ferdi won't let him add it.
• 500 or 504 could get temporarily returned if the server currently has a problem (e.g. sometimes Google or GitHub return a 5xx code for some unknown reason). In that case we can suspect that this problem will hopefully be fixed in the near future and that we can still add the website nonetheless.

That is the reason why I suggested only blocking 4xx codes as those are the only codes with which we can be pretty sure that the error actually is from an invalid URL and not something else.
Of course, 3xx and 5xx also come from invalid URLs but I think we should let the user inspect what the server actually returned (i.e. let them add the URL) to see if that might be the wrong URL or not.

[kytwb]

@vantezzen good point, I will update my pull request accordingly!

[vantezzen]

So is this issue resolved as getferdi/recipes#47 is merged?

[kytwb]

@vantezzen Yes! 🙏
Thanks for your patience @keunes, things should work as expected with your Nextcloud/Google Groups custom service in the next release.

[keunes]

Thanks both for your help!

[kytwb]

Shipped in v5.4.4-beta.2.

[pztrn]

Hey guys, are you sure it is fixed? Got same error on 5.4.4-beta.2 when trying my nextcloud instance. No 4xx errors, only 302 when redirecting to login page.

[kytwb]

@pztrn Can you please share the URL so we can reproduce? Thank you.

[pztrn]

@pztrn Can you please share the URL so we can reproduce? Thank you.

Sent to your mail address.

[kytwb]

Could reproduce, will take a better look at it later 🙏

[kytwb]

@pztrn The error I get is: ERR_TOO_MANY_REDIRECT (see developers console). I think this is an issue with Nextcloud (see Google search).

[pztrn]

Well, I saw no such error in Firefox, Safari and Google Chrome, only in ferdi. Also there is a difference in requests that might affect ferdi's behavior. For example, this is Chrome:

And this is ferdi:

So the whole point is CORS which is prohibited on my Nextcloud instance for a reason (and according to official Nextcloud nginx configuration).

[kytwb]

@pztrn I went a bit fast in re-closing, my bad.
Thanks for the precious additional details 🙏

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. Please check if this issue is still relevant and please close it if it's not. This will make sure that our open issues are actually of use and reduce the list of obsolete issues. Thank you for your contributions.

[Mic92]

I simply work-around this issue by creating my own nextcloud recipe: https://github.com/Mic92/franz-recipe-nextcloud

[renard]

Hi,

I also have the same issue with thruk and Ferdi 5.5.0.

It uses 2 kind of authentications: either the basic www-auth or a login form.
In both situations a 401 Unauthorized is return.
Ferdi should not forbid the service creation. Maybe a force checkbox is the solution here.

Here are the HTTP details:

GET /thruk/cgi-bin/login.cgi?thruk/ HTTP/1.1
Host: XXX
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36 Ferdi/5.5.0 (Electron 8.2.3)
Accept: */*
Sec-Fetch-Site: cross-site
Sec-Fetch-Mode: cors
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB
Cookie: thruk_test=%2A%2A%2A%2A
X-Forwarded-Proto: https
X-Forwarded-For: XXX

And the response:

HTTP/1.1 401 Unauthorized
Date: Thu, 28 May 2020 15:09:33 GMT
Server: Apache/2.4.25 (Debian)
Cache-Control: no-store
Expires: 0
Set-Cookie: thruk_test=%2A%2A%2A%2A; path=/thruk/
Vary: Cookie
Content-Length: 7298
Content-Type: text/html; charset=utf-8

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. Please check if this issue is still relevant and please close it if it's not. This will make sure that our open issues are actually of use and reduce the list of obsolete issues. Thank you for your contributions.

[keunes]

@kytwb D'you think this might be something for Hacktoberfest?