Documentserver: nodeJS - [AMQP] Error: connect ECONNREFUSED 127.0.0.1:5672

[zubanst]

Here is my configuration: Server1 ubuntu server 18.04 with ownCloud 10.4.1.3 and onlyoffice connector 6.0.1, Server2 ubuntu server 18.04 with onlyoffice document-server 6.0.1 initially installed from the repository described here https://helpcenter.onlyoffice.com/installation/docs-community-install-ubuntu.aspx . and migrated to the latest version. The document server works fine, a test in a browser on the local network with https:/// is redirected tohttps:///welcome/ saying Document server is running. Client side the server is correctly configured, no error saving the configuration. Try open a document in the ownloud client says error - server cannot be connected and results in the following error server side:
[2020-10-21T06:25:05.565] [WARN] nodeJS - update cluster with 1 workers
[2020-10-21T06:25:05.600] [WARN] nodeJS - worker 4737 started.
[2020-10-21T06:25:05.602] [WARN] nodeJS - update cluster with 1 workers
[2020-10-21T10:36:30.914] [WARN] nodeJS - update cluster with 1 workers
[2020-10-21T10:36:30.925] [WARN] nodeJS - worker 2123 started.
[2020-10-21T10:36:30.945] [WARN] nodeJS - update cluster with 1 workers
[2020-10-21T10:36:32.131] [ERROR] nodeJS - [AMQP] Error: connect ECONNREFUSED 127.0.0.1:5672
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)
[2020-10-21T10:36:33.136] [ERROR] nodeJS - [AMQP] Error: connect ECONNREFUSED 127.0.0.1:5672
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)
[2020-10-21T10:54:50.166] [WARN] nodeJS - update cluster with 1 workers
[2020-10-21T10:54:50.178] [WARN] nodeJS - worker 3055 started.
[2020-10-21T10:54:50.179] [WARN] nodeJS - update cluster with 1 workers
[2020-10-21T11:05:04.222] [WARN] nodeJS - update cluster with 1 workers
[2020-10-21T11:05:04.236] [WARN] nodeJS - worker 10699 started.
[2020-10-21T11:05:04.237] [WARN] nodeJS - update cluster with 1 workers
[2020-10-29T12:07:32.453] [WARN] nodeJS - update cluster with 1 workers
[2020-10-29T12:07:32.461] [WARN] nodeJS - worker 2091 started.
[2020-10-29T12:07:32.471] [WARN] nodeJS - update cluster with 1 workers
[2020-10-29T12:07:33.490] [ERROR] nodeJS - [AMQP] Error: connect ECONNREFUSED 127.0.0.1:5672
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)
[2020-10-29T12:10:58.779] [WARN] nodeJS - update cluster with 1 workers
[2020-10-29T12:10:58.816] [WARN] nodeJS - worker 13778 started.
[2020-10-29T12:10:58.820] [WARN] nodeJS - update cluster with 1 workers

sudo supervisorctl status all
ds:converter RUNNING pid 13646, uptime 0:30:13
ds:docservice RUNNING pid 13645, uptime 0:30:13
ds:metrics RUNNING pid 13643, uptime 0:30:13
ds:spellchecker RUNNING pid 13644, uptime 0:30:13

nc -v localhost 5672
Connection to localhost 5672 port [tcp/amqp] succeeded!

I think this is a bug server side.

[ShockwaveNN]

[2020-10-21T10:36:32.131] [ERROR] nodeJS - [AMQP] Error: connect ECONNREFUSED 127.0.0.1:5672

This is not right, which version of RabbitMQ are you using?

[zubanst]

I use the one installed from the bundle "RabbitMQ","3.6.10"

[ShockwaveNN]

Version seems OK, we got 3.6.10-1ubuntu0.3 in our default Docker image

Not sure that you mean by from the bundle

Could you show contents of /etc/onlyoffice/documentserver/local.json and /etc/onlyoffice/documentserver/default.json, are change anything?

We got similar error earlier https://github.com/ONLYOFFICE/DocumentServer/issues/674 but never found the reason

[zubanst]

From the bundle, I mean from the repository described in https://helpcenter.onlyoffice.com/installation/docs-community-install-ubuntu.aspx
Here are the 2 files. No changes.
json.tar.gz

[ShockwaveNN]

Everything seems fine in configs

Do you using some kind firewall? If you using default ubuntu firewall could you check it by ufw disable and restart all services via supervisorctl restart all

Also could you check that

rabbitmqctl list_connections

show some connections like this in our default Docker image

[zubanst]

No firewall, and rabbitmqctl list_connections give me this.

Listing connections
guest 127.0.0.1 43750 running
guest 127.0.0.1 43752 running
guest 127.0.0.1 43754 running

Additionally, at shutdown, RabbitMQ messaging server is stuck till timeout before real shutdown.

[ShockwaveNN]

@zubanst Hm, maybe something wrong with RabbitMQ itself? Something interesting at /var/log/rabbitmq/?

[zubanst]

Well, here is the startup log; all seems fine for me:

=INFO REPORT==== 2-Nov-2020::14:13:31 ===
Starting RabbitMQ 3.6.10 on Erlang 20.2.2
Copyright (C) 2007-2017 Pivotal Software, Inc.
Licensed under the MPL.  See http://www.rabbitmq.com/

=INFO REPORT==== 2-Nov-2020::14:13:31 ===
node           : rabbit@onlyoffice
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config (not found)
cookie hash    : cuAG0CmupG9TsAAeMlWFgA==
log            : /var/log/rabbitmq/[email protected]
sasl log       : /var/log/rabbitmq/[email protected]
database dir   : /var/lib/rabbitmq/mnesia/rabbit@onlyoffice

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
Memory limit set to 797MB of 1993MB total.

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
Enabling free disk space monitoring

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
Disk free limit set to 50MB

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
Limiting to approx 65436 file handles (58890 sockets)

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
FHC read buffering:  OFF
FHC write buffering: ON

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
Waiting for Mnesia tables for 30000 ms, 9 retries left

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
Waiting for Mnesia tables for 30000 ms, 9 retries left

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
Priority queues enabled, real BQ is rabbit_variable_queue

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
Starting rabbit_node_monitor

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
started TCP Listener on [::]:5672

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
Server startup complete; 0 plugins started.

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
accepting AMQP connection <0.311.0> (127.0.0.1:47548 -> 127.0.0.1:5672)

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
connection <0.311.0> (127.0.0.1:47548 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
accepting AMQP connection <0.332.0> (127.0.0.1:47550 -> 127.0.0.1:5672)

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
connection <0.332.0> (127.0.0.1:47550 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'

=INFO REPORT==== 2-Nov-2020::14:13:33 ===
accepting AMQP connection <0.360.0> (127.0.0.1:47552 -> 127.0.0.1:5672)

=INFO REPORT==== 2-Nov-2020::14:13:33 ===
connection <0.360.0> (127.0.0.1:47552 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'

[ShockwaveNN]

Log seems fine

I'm almost out of ideas,

Could you show output of rabbitmqctl list_queues, so there is no strange queues not related to DocumentServer?

[zubanst]

Timeout: 60.0 seconds ...
Listing queues for vhost / ...
name messages
amq.gen-1-UQ0iggePbqxCrIfgpTCw 0
ds.convertdead 0
ds.converttask 0
ds.convertresponse 0
ds.delayed 0

I did a fresh install on Ubuntu server 20.04 and updated owncloud client to 6.1.1, same result. It simply doesn't work. Has anybody succeeded to have such a working setup?

[ShockwaveNN]

I did a fresh install on Ubuntu server 20.04 and updated owncloud client to 6.1.1, same result.

This is very strange

Could you describe which exactly version of Ubuntu 20.04 are you using? Do you install it via official image on bare-bone server, or did you use some kind of cloud version from Amazon, Google Cloud, DigitialOcean?

I want to reproduce your problem via myself

[zubanst]

It is a VMware virtual machine. If you want, I can export the virtual machine and share it over on my cloud. I installed the latest Ubuntu distribution https://releases.ubuntu.com/20.04.1/ubuntu-20.04.1-live-server-amd64.iso and followed this guide: https://helpcenter.onlyoffice.com/installation/docs-community-install-ubuntu.aspx

[ShockwaveNN]

Yeah, of you export vm image - this would be great

On Sat, Nov 7, 2020, 11:31 zubanst [e-mail] wrote:

It is a VMware virtual machine. If you want, I can export the virtual
machine and share it over on my cloud. I installed the latest Ubuntu
distribution
https://releases.ubuntu.com/20.04.1/ubuntu-20.04.1-live-server-amd64.iso
and followed this guide:
https://helpcenter.onlyoffice.com/installation/docs-community-install-ubuntu.aspx

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ONLYOFFICE/DocumentServer/issues/1045#issuecomment-723418263,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFDG3CKR2D3BAQZAIH5LOLSOUAXNANCNFSM4TDVXH3Q
.

[zubanst]

Send me an email address so I could share the link.

[ShockwaveNN]

[email protected]

On Sat, Nov 7, 2020, 19:33 zubanst [e-mail] wrote:

Send me an email address so I could share the link.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ONLYOFFICE/DocumentServer/issues/1045#issuecomment-723466213,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAFDG3EMTJPNT4HRPWZ2I5LSOVZFRANCNFSM4TDVXH3Q
.

[ShockwaveNN]

@zubanst I've recieved you VM, never have any expirience with VM Sphere, so need some time to figure out how to run it and is it possible to run it using VirtualBox

Post any results here as soon as get any

[ShockwaveNN]

Ok, I was able setup your vm instance using VirtualBox and login, but never able to setup any network, even local, so I got troubles even reproduce your issue

So only idea or advice I have right now - could you setup some cloud provider like AWS or DigitalOcean with default instruction and see if this work (and I'm pretty sure this should work) and compare configs with yours

Or use docker

[zubanst]

I don't use AWS nor DigitalOcean so it is pointless I setup such. I will try a docker, though I doubt though it would work. Again, have you any knowledge of someone having set up a Ubuntu/Debian environment working with Owncloud client?

[zubanst]

I'll convert the machine to virtualbox myself and let you know when it's ready for download.

[ShockwaveNN]

Again, have you any knowledge of someone having set up a Ubuntu/Debian environment working with Owncloud client

I'm not a owncloud guy, but there is a plenty a people have setup (you can take a look here, https://github.com/ONLYOFFICE/onlyoffice-owncloud a lot of closed issues, and I think more have no issues)

But again, I don't think your error have something to do with owncloud, since rabbitmq is vital part of DocumentServer itself

[zubanst]

I figured out what is wrong with your Virtualbox. You need to configure bridged interface for the VM network and your vbox host need to connect to your network infrastructure through wired connection. Wireless wont do.

[ShockwaveNN]

Yeah, I set up network settings like this:

But still no luck

Maybe something wrong with DNS, I tryed to changed to google via settings up DNS param to 8.8.8.8 in /etc/systemd/resolved.conf, and restarting service systemd-resolved restart but seems this is not enought

[ShockwaveNN]

@zubanst Also my colleague give me some additional info about situation in which this can be happen

By any chance - didn't you try to install some other edition of DocumentServer on this server?
I see that onlyoffice-documentserver is installed, but didn't you try to install onlyoffice-documentserver-de before, remove it and install onlyoffice-documentserver after that by any chance?

[zubanst]

No, this was a fresh install strictly following the guide I mentioned, published on the official website. I'm still working to get you a vbox machine with a similar install (I had other emergencies since).

[zubanst]

I just shared a vbox machine with you. Check your email. Same username, same password. No need to install the certificate, I already installed a self signed one.

[ShockwaveNN]

@zubanst Yeah, thanks I've downloaded it, and will take a look as soon as I have time for it, I'll reply here if got any new info

[ShockwaveNN]

Ok, thanks for new version I was able to setup it in VirtualBox

Soo
I done those steps

1. Stopped all services via sudo supervisorctl stop all
2. Cleared docservice log file via editing it with sudo vim /var/log/onlyoffice/documentserver/docservice/err.log
3. Started all services via sudo supervisorctl start all
4. Open test example and create new file got those:
   
   Look at logs in /var/log/onlyoffice/documentserver/docservice/out.log and this seems related to self-signed cert

[2020-11-17T08:59:26.690] [ERROR] nodeJS - postData error: docId = 192.168.1.238https___192.168.1.237_example_files_192.168.1.238_new_20_1_.docx1605603566028;url = https://192.168.1.237/example/track?filename=new%20(1).docx&useraddress=192.168.1.238;data = {"key":"192.168.1.238https___192.168.1.237_example_files_192.168.1.238_new_20_1_.docx1605603566028","status":1,"users":["uid-1"],"actions":[{"type":1,"userid":"uid-1"}]}
Error: self signed certificate
    at TLSSocket.onConnectSecure (_tls_wrap.js:1058:34)
    at TLSSocket.emit (events.js:198:13)
    at TLSSocket._finishInit (_tls_wrap.js:636:8)

Seems this is related to self-signed cert, I'm not very good at that, but I know that rabbitmq error is my interset now, so I temoraly disable self-signed certs via restoring default non-ssl config:

sudo cp /etc/onlyoffice/documentserver/nginx/ds.conf /etc/onlyoffice/documentserver/nginx/ds.conf.bak.ssl
sudo cp /etc/onlyoffice/documentserver/nginx/ds.conf.bak /etc/onlyoffice/documentserver/nginx/ds.conf
sudo service nginx start

And after that I can open test example via http://192.168.1.237/ and everyhing fine for me

No errors for connection to rabbitmq or any errors at all at cat /var/log/onlyoffice/documentserver/docservice/out.log

[zubanst]

Unfortunately it cannot be used in the cloud with plain http. How can I do the same test? How to build and where to put this example html so I could use the document server 'offline' without the Onwcloud app in my own LAN?

[ShockwaveNN]

@zubanst To be clear - this issue was created not about problems with https and self-signed certificate, you shared VM for me with problem about RabbitMQ connection

[2020-10-21T10:36:32.131] [ERROR] nodeJS - [AMQP] Error: connect ECONNREFUSED 127.0.0.1:5672
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)

To be able to help with this error I should be able to reproduce it, but even with your VM image I cannot see this error.

But problem with self-signed cert it's all other problem, which is not critical for me right now (for example you can use use not self-signed cert, but let's encrypt free cert)

[zubanst]

You sent some screenshots saying the document server I sent you in vbox works. How do I build and where I put this example html so I could use the document server 'offline' and do the same test as you with my VMware document server?

[ShockwaveNN]

@zubanst this test example is included in latest release of DocumentServer, you can start it by executing supervisorctl start all and access by http://local_ds_ip/example or see details of it on http://local_ds_ip/welcome

[zubanst]

Issue solved. Used port redirection and my true website ssl certificate instead of the self-signed one. Should someone be interested by more details, contact me.