DietPi Software | Firefox Sync Server upgrade/new install fails

[kinoushe]

ADMIN EDIT

Solution:

apt install libmariadb-dev
cd /opt/firefox-sync
make build
systemctl start firefox-sync

---

Creating a bug report/issue

Required Information

• DietPi version |
  G_DIETPI_VERSION_CORE=6
G_DIETPI_VERSION_SUB=32
G_DIETPI_VERSION_RC=2
G_GITBRANCH='master'
G_GITOWNER='MichaIng'
• Distro version | buster
• Kernel version | Linux 5.4.51-v7+ #1333 SMP Mon Aug 10 16:45:19 BST 2020 armv7l GNU/Linux
• SBC model | RPi 3b+ and RPi 4
• Power supply used | Official power supply
• SDcard used | Samsung EVO plus 64GB

Additional Information (if applicable)

• Software title | Firefox Sync Server
• Initially installed and working well in v6.31 release, trying to upgrade (RPi 3b+) or fresh install (RPi 4) in v6.32.2 now fails

Steps to reproduce

1. Update method: dietpi-software reinstall 177 (as per https://dietpi.com/phpbb/viewtopic.php?p=24713#p24713)

Expected behaviour

• Upgrade from syncserver 1.8.0 to syncserver 1.9.0

Actual behaviour

• [FAILED] DietPi-Services | ● firefox-sync.service

Sep 11 19:17:28 dietFeeds systemd[905]: firefox-sync.service: Failed to execute command: No such file or directory
Sep 11 19:17:28 dietFeeds systemd[905]: firefox-sync.service: Failed at step EXEC spawning /opt/firefox-sync/local/bin/gunicorn: No such file or directory

Extra details

• Install/reinstall errors:

Using cached mysqlclient-1.4.6.tar.gz (85 kB)
    ERROR: Command errored out with exit status 1:
     command: /opt/firefox-sync/local/bin/python2 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-9B10eb/mysqlclient/setup.py'"'"'; __file__='"'"'/tmp/pip-install-9B10eb/mysqlclient/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-LDhCqa
         cwd: /tmp/pip-install-9B10eb/mysqlclient/
    Complete output (12 lines):
    sh: 1: mysql_config: not found
    sh: 1: mariadb_config: not found
    sh: 1: mysql_config: not found
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-9B10eb/mysqlclient/setup.py", line 16, in <module>
        metadata, options = get_config()
      File "setup_posix.py", line 61, in get_config
        libs = mysql_config("libs")
      File "setup_posix.py", line 29, in mysql_config
        raise EnvironmentError("%s not found" % (_mysql_config_path,))
    EnvironmentError: mysql_config not found
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

[MichaIng]

Many thanks for your report. So basically the build failed. Ah I was able to replicate even on x86_64. Let's see what changed there.

Ah they now build MySQL support directly, was missing in build instructions as well: https://github.com/mozilla-services/syncserver/issues/235

The problem is we use it with SQLite (MySQL/MariaDB seems to be quite a large overhead for a home/family instance at least). The following solves the issue, but I'm not happy with it: apt install libmariadb-dev

Let me see if there is a way to disable MySQL for the build to skip those dependencies.

[MichaIng]

Okay, I found what has changed the last days: https://github.com/mozilla-services/syncserver/issues/235#issuecomment-691665317
I'll hence add the libmariadb-dev dependency for now, but probably it is reasonable to migrate to MariaDB backend all together then, since half of it is already installed now.

@kinoushe
How large is your database and how many users use your instance?

ls -l /mnt/dietpi_userdata/firefox-sync/FF-Sync-DB.db

[MichaIng]

Fixed here (together with some overdue uninstall alignments): https://github.com/MichaIng/DietPi/commit/486a072d8b6867b8205a5056b90873641cf85eef
Changelog: https://github.com/MichaIng/DietPi/commit/3c4031b964c0fbb887b1595d48aec348af140dfb

---

Solution:

apt install libmariadb-dev
cd /opt/firefox-sync
make build
systemctl start firefox-sync

[kinoushe]

Thanks so much for your thorough investigation and quick solution! It appears to be working correctly again.

With regards to the database, I'm using with just a few clients (maybe 5 max?), and the size:
-rw-r--r-- 1 ffsync ffsync 6295552 Sep 14 09:44 /mnt/dietpi_userdata/firefox-sync/FF-Sync-DB.db

I'm also running FreshRSS on the same device which I think already uses MariaDB, so migrating to it is probably a good idea?

There were also some warnings/errors during the build which I accidentally closed, so I've reinstalled with dietpi-software reinstall 177 to copy them just in case any are important:

DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.

...

WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', BadStatusLine('No status line received - the server has closed the connection',))': /simple/grpcio/

...

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

requests 2.20.0 requires idna<2.8,>=2.5, but you'll have idna 2.8 which is incompatible.
requests 2.20.0 requires urllib3<1.25,>=1.21.1, but you'll have urllib3 1.25.2 which is incompatible.
gevent 1.4.0 requires greenlet>=0.4.14; platform_python_implementation == "CPython", but you'll have greenlet 0.4.13 which is incompatible.
tokenserver 1.5.11 requires configparser==3.7.4, but you'll have configparser 3.5.0 which is incompatible.
tokenserver 1.5.11 requires cornice==3.5.1, but you'll have cornice 0.16.2 which is incompatible.
tokenserver 1.5.11 requires gunicorn==19.10.0, but you'll have gunicorn 19.6.0 which is incompatible.
tokenserver 1.5.11 requires mozsvc==0.10, but you'll have mozsvc 0.9 which is incompatible.
tokenserver 1.5.11 requires pyramid==1.10.4, but you'll have pyramid 1.5.3 which is incompatible.
tokenserver 1.5.11 requires requests==2.22.0, but you'll have requests 2.20.0 which is incompatible.
tokenserver 1.5.11 requires WebOb==1.8.5, but you'll have webob 1.4.1 which is incompatible.
google-api-core 1.22.2 requires futures>=3.2.0; python_version < "3.2", but you'll have futures 3.0.0 which is incompatible.

And from systemctl status firefox-sync:

Sep 14 09:51:25 dietFeeds gunicorn[15007]: [2020-09-14 09:51:25 +0000] [15389] [INFO] Booting worker with pid: 15389
Sep 14 09:51:28 dietFeeds gunicorn[15007]: /opt/firefox-sync/local/local/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.2) or chardet (3.0.4) doesn't match a supported vers
ion!
Sep 14 09:51:28 dietFeeds gunicorn[15007]:   RequestsDependencyWarning)
Sep 14 09:51:55 dietFeeds gunicorn[15007]: [2020-09-14 09:51:55 +0000] [15007] [CRITICAL] WORKER TIMEOUT (pid:15389)
Sep 14 09:51:56 dietFeeds gunicorn[15007]: [2020-09-14 09:51:56 +0000] [15457] [INFO] Booting worker with pid: 15457
Sep 14 09:51:58 dietFeeds gunicorn[15007]: /opt/firefox-sync/local/local/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.2) or chardet (3.0.4) doesn't match a supported vers
ion!
Sep 14 09:51:58 dietFeeds gunicorn[15007]:   RequestsDependencyWarning)

[MichaIng]

Jep, I recognised those errors as well, safe to ignore: https://github.com/mozilla-services/syncserver/issues/238
The second dependency warning here: https://github.com/mozilla-services/syncserver/issues/233
Basically all due to an underlying switch to pypa for dependency handling which as well caused the need to install the MySQL/MariaDB headers now.

I can't remember seeing this broken connection and worker timeout, however if your server otherwise works fine with your clients, I'd ignore it as well for now. Give it some week or two until those dependency warnings and probably some other related issues have been fixed (see the linked issues above), then run a dietpi-software reinstall 177 to see if the noisy startup is sorted.

[kinoushe]

I've just installed Beta v6.33.0 #3802, and successfully reinstalled firefox-sync using dietpi-software reinstall 177 without any errors - fantastic!

However the service status reports an error about Can't open run-syncstorage-rs.sh:

Oct 04 09:42:15 dietFeeds systemd[1]: Started Firefox Sync Server (DietPi).
Oct 04 09:43:27 dietFeeds gunicorn[658]: [2020-10-04 09:43:27 +0000] [658] [INFO] Starting gunicorn 19.6.0
Oct 04 09:43:27 dietFeeds gunicorn[658]: [2020-10-04 09:43:27 +0000] [658] [INFO] Listening at: http://192.168.10.139:5000 (658)
Oct 04 09:43:27 dietFeeds gunicorn[658]: [2020-10-04 09:43:27 +0000] [658] [INFO] Using worker: sync
Oct 04 09:43:27 dietFeeds gunicorn[658]: [2020-10-04 09:43:27 +0000] [1183] [INFO] Booting worker with pid: 1183
Oct 04 09:43:30 dietFeeds gunicorn[658]: /opt/firefox-sync/local/local/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.2) or chardet (3.0.4) doesn't match a supported versio
n!
Oct 04 09:43:30 dietFeeds gunicorn[658]:   RequestsDependencyWarning)
Oct 04 09:43:34 dietFeeds gunicorn[658]: sh: 0: Can't open run-syncstorage-rs.sh

Syncing also doesn't appear to be working in Firefox, partial log from about:sync-logs:

1601769429034   Services.Common.TokenServerClient   DEBUG   Beginning OAuth token exchange: http://192.168.10.139:5000/token/1.0/sync/1.5
1601769429034   Services.Common.RESTRequest DEBUG   GET request to http://192.168.10.139:5000/token/1.0/sync/1.5
1601769429045   Services.Common.RESTRequest DEBUG   GET http://192.168.10.139:5000/token/1.0/sync/1.5 200
1601769429045   Services.Common.TokenServerClient   DEBUG   Got token response: 200
1601769429045   Services.Common.TokenServerClient   DEBUG   Successful token response
1601769429046   Sync.Status DEBUG   Status.login: success.login => success.login
1601769429046   Sync.Status DEBUG   Status.service: success.status_ok => success.status_ok
1601769429046   Sync.BrowserIDManager   DEBUG   _findCluster returning http://192.168.10.139:5000/storage/1.5/1/
1601769429046   Sync.BrowserIDManager   DEBUG   Cluster value = http://192.168.10.139:5000/storage/1.5/1/
1601769429046   Sync.Status DEBUG   Status.login: success.login => error.login.reason.network
1601769429046   Sync.Status DEBUG   Status.service: success.status_ok => error.login.failed
1601769429046   Sync.ErrorHandler   ERROR   Sync encountered a login error
1601769429047   Sync.SyncScheduler  DEBUG   Clearing sync triggers and the global score.
1601769429047   Sync.SyncScheduler  DEBUG   Next sync in 3600000 ms. (why=schedule)
1601769429049   FirefoxAccounts TRACE   not checking freshness of profile as it remains recent
1601769429049   Sync.Service    DEBUG   Exception calling WrappedLock: Error: Login failed: error.login.reason.network(resource://services-sync/service.js:1043:15) JS Stack trace: [email protected]:1043:15
1601769429050   Sync.Service    DEBUG   Not syncing: login returned false.

[MichaIng]

Okay, the Python-based Firefox Sync Server is history. They changed it to simply clone and run the Rust-based server now: https://github.com/mozilla-services/syncserver/issues/246
I thought this will be a new repository but now it has become sort of a migration, sadly breaking all installers in integrations radically, aside of ours e.g. the Docker image: https://github.com/mozilla-services/syncserver/commit/d9253211837c94c93ea285c4894646590dfdc2ab

Here is the hint that this kinda patchwork migration means we cannot really offer a reliable install as it will change until the complete migration has been done: https://github.com/mozilla-services/syncserver#note

Not sure if staying on an older commit will work, until all this has settled: https://github.com/mozilla-services/syncserver/issues/246#issuecomment-703229482

[MichaIng]

Just to see, I'm trying to get it running:

cd /tmp
wget https://github.com/mozilla-services/syncstorage-rs/archive/master.tar.gz
tar xf master.tar.gz
mv syncstorage-rs-master /opt/firefox-sync/syncstorage-rs
usermod -d /opt/firefox-sync ffsync
chown -R ffsync /opt/firefox-sync
sudo -u ffsync dash -c "$(curl -sSf https://sh.rustup.rs)" rustup-init.sh -y --no-modify-path
apt install libssl-dev pkg-config cmake libmariadb-dev-compat libcurl4-openssl-dev
G_CONFIG_INJECT 'Environment=' 'Environment="PATH=/opt/firefox-sync/.cargo/bin:/usr/local/bin:/usr/bin:/bin"' /etc/systemd/system/firefox-sync.service '\[Service\]'

... not sure why the service still does not start up with above, failing with sh: 0: Can't open run-syncstorage-rs.sh
Via sudo -su ffsync, exporting above PATH, running the script works fine and it downloads and compiles a whole bunch of additional storage server dependencies. That is probably something we should do directly within the installer.

Running Rust applications is kinda complicated as system service. By default Rust is installed in the users home directory, including all binaries, toolchains and components. So the PATH variable must be extended to allow calling cargo like the new script does. We could install cargo (including rustc) via APT package from Debian repository, but it will be a bid outdated and we get inconsistent results on different distros. Toolchains and components are still installed to the users home dir, so the issue remains that ffsync requires write access to the install dir to install all those. Probably this can be reverted when the install has finished once.

Okay that works to pre-build the storage server:

cd /opt/firefox-sync/syncstorage-rs
cargo build`

Also found a solution for startup issue, it's only because it assumes to be in the install directory, since it, cd fails. Not very robust but at least easy to work around. This is the current systemd unit:

2020-10-04 15:16:43 root@VM-Buster:/opt/firefox-sync# systemctl cat firefox-sync
# /etc/systemd/system/firefox-sync.service
[Unit]
Description=Firefox Sync Server (DietPi)
Documentation=https://github.com/mozilla-services/syncserver
Wants=network-online.target
After=network-online.target dietpi-boot.service

[Service]
User=ffsync
WorkingDirectory=/opt/firefox-sync
Environment="PATH=/opt/firefox-sync/.cargo/bin:/usr/local/bin:/usr/bin:/bin"
ExecStart=/opt/firefox-sync/local/bin/gunicorn --paste /mnt/dietpi_userdata/firefox-sync/syncserver.ini

# Hardening
ProtectSystem=strict
ProtectHome=true
PrivateDevices=true
ProtectKernelTunables=true
ProtectControlGroups=true
ReadWritePaths=/mnt/dietpi_userdata/firefox-sync /tmp /opt/firefox-sync

[Install]
WantedBy=multi-user.target

Next issue:

Oct 04 15:14:57 VM-Buster gunicorn[4087]: thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ApiError { inner:
Oct 04 15:14:57 VM-Buster gunicorn[4087]: Invalid SYNC_DATABASE_URL: sqlite:////mnt/dietpi_userdata/firefox-sync/FF-Sync-DB.db, status: 500, metric_label: None }', src/main.rs:51:64
Oct 04 15:14:57 VM-Buster gunicorn[4087]: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Oct 04 15:14:57 VM-Buster gunicorn[4087]: Panic in Arbiter thread.

...

---

Seems like it does not support SQLite anymore, so we are finally forced to switch to MariaDB 😉. Yaaa, actually I do not see a point to implement all this yet, as long as it's half-baked patchwork like now. Either the previous commit still works or we have to wait until all is finished and migrated.

[MichaIng]

Let's see if this works: #3807

G_DEV_BRANCH ffsync
dietpi-software reinstall 177

[MichaIng]

@kinoushe
Do you find time to apply this change to your /boot/dietpi/dietpi-software and reinstall ffsync to see if it works? The above two commands are a shortcut.

[kinoushe]

Thanks again @MichaIng for your thorough investigation and sorting out of this mess!

I've been away but just running the above shortcuts to reinstall using the ffsync branch now (thanks for this, I wasn't quite sure how to apply #3807 originally).

All done, and no errors during reinstall or from systemctl status firefox-sync. Syncing also is working again, so I think this resolves the problem 👍

[MichaIng]

Great, many thanks for testing. Let's hope that staying on this older commit for a while does not cause issues later. I'll keep an eye on this and migrate as fast as it seems to be a stable setup process.

[MichaIng]

The move to Rust-based tokenserver has been btw reverted. So current master would work again. Since this can happen a second time and since the current master now matches exactly our hardcoded commit, we'll leave it like that and update the commit when there is any save change.

From the bitwarden_rs server integration (#3724) we might be able to learn a bit about Rust-software in general, or the other way round. I mark this as closed but keep an eye on the development of ffsync and its migration to Rust.