Acme.sh: Verify error:Invalid response from [domain]

[wion]

Hi Neil,

Something seems to have changed since the last time I renewed certs. This time around I'm getting an error.

I run this command:

acme.sh --renew -d domain.tld -d www.domain.tld --force

But then it errors after the "Standalone mode server" line:

...
[Tue May 30 18:17:17 UTC 2017] The new-authz request is ok.
[Tue May 30 18:17:17 UTC 2017] Verifying:domain.tld
[Tue May 30 18:17:17 UTC 2017] Standalone mode server
[Tue May 30 18:17:22 UTC 2017] domain.tld:Verify error:Invalid response from http://domain.tld/.well-known/acme-challenge/qEp9FiogrSkAOM3TYzfhDDKo1J_6abK8FQ5qbtaQY9w: 
GET / HTTP/1.1
User-Agent: acme.sh client: https://github.com/Neilpang/acme.sh
Host: localhost:14927
Accept: */*

I am trying to troubleshoot it with the web host too, but they're not finding the issue. At first they thought it was because of my http --> https redirect rules, but when these are commented out in .htaccess, the error still happens.

I've looked at --debug but I'm not knowledgeable enough with this kind of thing to know if there's anything there or not.

Any suggestions?

[FernandoMiguel]

if you already have a http server running using port 80, you cant run standalone, unless you use pre/post hook to stop it

[Neilpang]

why not paste the --debug log or --debug 2 log ?
How can I guess ?

[wion]

Yes, sorry, Neil. I guess that would help.

So, you mean what I get in output after running this?

acme.sh --renew -d domain.tld -d www.domain.tld --force --debug

I just want to make sure I'm using it right. That gives me a lot of output, including the contents of my websites' 404 template, and the usage instructions of Ncat. ¯\_(ツ)_/¯

[wion]

While I'm waiting on confirmation about using --debug correctly, I'd like to ask about another point that is unclear to me.

I used the "Standalone mode" commands for multiple domains when first setting up the certificates a while back and then the renewal commands indicated above when renewing certs and I never had a problem before. I created a doc for the process, which worked last time I renewed certs...
https://github.com/content-strategy-forum/csf-docs/blob/master/Administrators/WebFaction/creating-ssl-certs-on-webfaction.md

But the website is an Apache server, and I maintain certs for multiple domains on the server. Should I be using the Apache mode commands now, or does it matter?

[Neilpang]

@wion

The standalone mode conflicts with webroot/or apache mode.

see:
https://github.com/Neilpang/acme.sh/wiki/How-to-issue-a-cert#2-standalone-mode

Yes, give me the outpu of --debug 2

BTW, when you use --renew command, you jut need to give one single -d parameter, with the first main domain.

acme.sh --renew  -d example.com

acme.sh knows how many domains in the cert need to renew.

[wion]

Thanks, Neil. I have to step away for a while but I will follow up later.

[wion]

Ok, Neil. It's long. I changed the account username and domain, but otherwise it's all --debug 2 output.

$ acme.sh --renew -d domain.tld --force --debug 2
[Thu Jun  1 07:14:34 UTC 2017] Lets find script dir.
...
Output clipped for sanity.

[wion]

@FernandoMiguel

Thanks. But I'm not sure what you mean. I run acme.sh on the web host's server (Apache). The port is something different than 80. But I have wondered if setting everything up as Standalone was a mistake, but it never seemed to be an issue before. I've only started getting this error now on this latest attempt to renew.

[FernandoMiguel]

@wion you mean you run apache on a different port other than 80/443?
or that you are trying to run LE on a different port?
LE can only use the standard ports for validation

[Neilpang]

@wion
I can only tell you that: if you already have apache server running on the server, please use webroot mode or apache mode.

The standalone mode should not work at all, because it needs to listen at 80 port, which is already used by your apache server.

[wion]

@FernandoMiguel

All I can tell you is that I followed this instruction from start to finish about a year ago to setup certs on my web host (WebFaction) using acme.sh and it worked:

https://github.com/content-strategy-forum/csf-docs/blob/master/Administrators/WebFaction/creating-ssl-certs-on-webfaction.md

And I successfully updated the certs using the same tutorial the last time I renewed them.

So between now and 80 days ago, something has changed somewhere, whether at the web host end or acme.sh.

@Neilpang

Okay, thanks. It seems what you and Fernando are telling me is I need to reconfigure the way I have acme.sh setup, and subsequently rewrite/edit my tutorial with respect to webroot or Apache mode.

Can you confirm... Do I just delete all existing certificates in the _/certificates_ directory, and then run new commands? Or must I delete the existing install of acme.sh entirely and start from scratch?

[wion]

@Neilpang

On your README page, under the Apache mode section, it says:

If you are running a web server, Apache or Nginx, it is recommended to use the Webroot mode.

Particularly, if you are running an Apache server, you should use Apache mode instead. This mode doesn't write any files to your web root folder.

Those two lines, particularly the parts I put in bold, are contradictory. If I'm using an Apache server (which I'm setup with on WebFaction), which mode should I use — webroot or apache?

[Neilpang]

The answer is: either.

You can use either way.

If you use apache mode, you must be root, but it's not required for webroot mode.

[Neilpang]

Can you confirm... Do I just delete all existing certificates in the /certificates directory, and then run new commands?

Yes, correct. you can just delete the domain folder.

[wion]

Thanks Neil.

WebFaction suggests I use their API to "work around it", which I wouldn't know the first thing about. Or to use "the other" script, LetsEncrypt-WebFaction...
https://github.com/will-in-wi/letsencrypt-webfaction

I'll give it a shot using your other modes. If I still run into trouble, I'll try that other script, since it does seem to be designed specifically for my web host.

[bmmeijers]

I am running in the same issue. I think there is a problem with the composition of the URL that is used.

To me it seems that ": " is added to one of the URLs and then it does not work. The file however does exist inside the webroot ("ECXhqL9PzbAD2Zam6ituHZBv7uCjjY6Lb0lIuyhHOto" without ": " at the end.

See below:

[do jun 15 10:24:54 CEST 2017] response='{"type":"http-01","status":"invalid","error":{"type":"urn:acme:error:unauthorized","detail":"Invalid response from http://[...]/.well-known/acme-challenge/ECXhqL9PzbAD2Zam6ituHZBv7uCjjY6Lb0lIuyhHOto: \"\u003chtml\u003e\r\n\u003chead\u003e\u003ctitle\u003e404 Not Found\u003c/title\u003e\u003c/head\u003e\r\n\u003cbody bgcolor=\"white\"\u003e\r\n\u003ccenter\u003e\u003ch1\u003e404 Not Found\u003c/h1\u003e\u003c/center\u003e\r\n\u003chr\u003e\u003ccenter\u003e\"","status": 403},"uri":"https://acme-v01.api.letsencrypt.org/acme/challenge/eprzcpCGI12F8X-N4YJw5MBizQxFIK0BBCw8cXA9sYU/1344421236","token":"ECXhqL9PzbAD2Zam6ituHZBv7uCjjY6Lb0lIuyhHOto","keyAuthorization":"ECXhqL9PzbAD2Zam6ituHZBv7uCjjY6Lb0lIuyhHOto.7pHcZaRMkgpQ0bSX94-cZfVHKZWjtFeHEaKjjBrwYRo","validationRecord":[{"url":"http://[...]/.well-known/acme-challenge/ECXhqL9PzbAD2Zam6ituHZBv7uCjjY6Lb0lIuyhHOto","hostname":"[...]","port":"80","addressesResolved":["37.252.121.235","2a02:2770:7:0:21a:4aff:fe7b:4ab5"],"addressUsed":"2a02:2770:7:0:21a:4aff:fe7b:4ab5","addressesTried":[]}]}'
[do jun 15 10:24:54 CEST 2017] error='"error":{"type":"urn:acme:error:unauthorized","detail":"Invalid response from http://[...]/.well-known/acme-challenge/ECXhqL9PzbAD2Zam6ituHZBv7uCjjY6Lb0lIuyhHOto: '
[do jun 15 10:24:54 CEST 2017] errordetail='Invalid response from http://[...]/.well-known/acme-challenge/ECXhqL9PzbAD2Zam6ituHZBv7uCjjY6Lb0lIuyhHOto: '
[do jun 15 10:24:54 CEST 2017] [...]:Verify error:Invalid response from http://[...]/.well-known/acme-challenge/ECXhqL9PzbAD2Zam6ituHZBv7uCjjY6Lb0lIuyhHOto: 
[do jun 15 10:24:54 CEST 2017] Debug: get token url.

[Neilpang]

@bmmeijers
Why not tell us more details:

1. what is your OS
2. what is your webserver
3. what mode do you use.
4. Paste the full log

[bmmeijers]

1. Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.73-2+deb7u3 x86_64 GNU/Linux
2. nginx version: nginx/1.2.1
3. webroot mode (it has worked for quite some time very well!)
4. i am currently rate limited, what's the best way to get a log file? This?

acme.sh --renew -d domain --force --debug 3 --log /tmp/debug.log

[Neilpang]

@bmmeijers
Your nginx website conf file must be changed from last time your issued the cert.
Please paste the nginx website conf file content.

And please upgrade to the latest version first, then you can also try with the --nginx mode.

acme.sh --upgrade
acme.sh --issue -d example.com  --nginx  --test --debug 2

[bmmeijers]

Ok, herewith the log file: log.txt

See around line 204, the URL contains ": " at the end, so:
"http://rambonnetgroep.nl/.well-known/acme-challenge/WjtpQaDK5_TRzEAgyUegUMnbC8sdNVWP6ax-_esbDmQ: "

That gives a 404 (not strange, the file is not there)!

But if I go to (so manually remove ": " from the end):

http://rambonnetgroep.nl/.well-known/acme-challenge/WjtpQaDK5_TRzEAgyUegUMnbC8sdNVWP6ax-_esbDmQ

It is there... Does this mean that acme.sh gives a wrong URL to let's encrypt?

Note, that this set up has been running for >6 months without problems before.

It broke this night, when the cron job that is scheduled gave me the same error message. I do have another domain setup and for this domain the problem does not occur (actually it was renewed by the cron job.

[Neilpang]

@baiyangliu
No, it's just an error message. the url is good.

[bmmeijers]

It's not, because of the 404 reported the URL has to be incorrect. Question is: where is the URL composed?

[Neilpang]

@bmmeijers
It's just the error message returned from the CA. And we never give any url to the CA, the CA can make the correct url by itself to visit.

I checked the log, the url is good.

[bmmeijers]

How does this error message then comes into existence:

"""
original='{
"type": "http-01",
"status": "invalid",
"error": {
"type": "urn:acme:error:unauthorized",
"detail": "Invalid response from http://rambonnetgroep.nl/.well-known/acme-challenge/WjtpQaDK5_TRzEAgyUegUMnbC8sdNVWP6ax-_esbDmQ: \"\u003chtml\u003e\r\n\u003chead\u003e\u003ctitle\u003e404 Not Found\u003c/title\u003e\u003c/head\u003e\r\n\u003cbody bgcolor=\"white\"\u003e\r\n\u003ccenter\u003e\u003ch1\u003e404 Not Found\u003c/h1\u003e\u003c/center\u003e\r\n\u003chr\u003e\u003ccenter\u003e\"",
"status": 403
},
"uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/cgY7Y2bW9i4GjXQ6js8ugqL8ZWd1hEQIGxVc6E5aUSA/1345669940",
"token": "WjtpQaDK5_TRzEAgyUegUMnbC8sdNVWP6ax-_esbDmQ",
"keyAuthorization": "WjtpQaDK5_TRzEAgyUegUMnbC8sdNVWP6ax-_esbDmQ.7pHcZaRMkgpQ0bSX94-cZfVHKZWjtFeHEaKjjBrwYRo",
"validationRecord": [
{
"url": "http://rambonnetgroep.nl/.well-known/acme-challenge/WjtpQaDK5_TRzEAgyUegUMnbC8sdNVWP6ax-_esbDmQ",
"hostname": "rambonnetgroep.nl",
"port": "80",
"addressesResolved": [
"37.252.121.235",
"2a02:2770:7:0:21a:4aff:fe7b:4ab5"
],
"addressUsed": "2a02:2770:7:0:21a:4aff:fe7b:4ab5",
"addressesTried": []
}
]
}'
"""
Can you explain?

[Neilpang]

@bmmeijers
It's just json encoded message from the CA.
Please read the message carefully. you will understand.

[bmmeijers]

Do you see anything else that then goes (obviously) wrong?

[Neilpang]

@bmmeijers
No, I don't see any error.

I just visited the url:

curl  http://rambonnetgroep.nl/.well-known/acme-challenge/WjtpQaDK5_TRzEAgyUegUMnbC8sdNVWP6ax-_esbDmQ

It's not 404 error.

Can you please try with --nginx mode ?

[Neilpang]

@bmmeijers
I know the reason now.

your domain have both ipv4 and ipv6 resolved. The CA visited the ipv6 address, it's 404 error.

curl  -6  http://rambonnetgroep.nl/.well-known/acme-challenge/WjtpQaDK5_TRzEAgyUegUMnbC8sdNVWP6ax-_esbDmQ

This is a policy change of the CA, they prefer ipv6 than ipv4.

[Neilpang]

@bmmeijers

"validationRecord": [
    {
      "url": "http://rambonnetgroep.nl/.well-known/acme-challenge/WjtpQaDK5_TRzEAgyUegUMnbC8sdNVWP6ax-_esbDmQ",
      "hostname": "rambonnetgroep.nl",
      "port": "80",
      "addressesResolved": [
        "37.252.121.235",
        "2a02:2770:7:0:21a:4aff:fe7b:4ab5"
      ],
      "addressUsed": "2a02:2770:7:0:21a:4aff:fe7b:4ab5",
      "addressesTried": []
    }
  ]

If your website is not accessable at the ipv6 address, you can remove the ipv6 record of the domain.

[bmmeijers]

Thanks for finding that out!

It does make sense that it then works for the other domain, as this other domain does not have a DNS entry for IPv6.

I adapted my nginx configuration (to also listen on ipv6) and now it works!

[wion]

Just a note that my situation was probably different. I have it working now, following Neil's recommendation to delete old certs, and create new ones using webroot mode.

Unfollowing now.

[toxi22]

I confirm. The verification is done first by IPv6, and then by IPv4, if IPv6 is specified in the domain settings. I faced the same problem when I had to order a new certificate.