Spksrc: http://packages.synocommunity.com down ?

[techbliss]

Hello i get no response from http://packages.synocommunity.com via DSM
and website https://synocommunity.com/ seems to have cloudfare problems, and it have been like this for some days now.
Anyone knows when the site will be back up.

[sstraus]

same problem here

[JoveYu]

try to use https://packages.synocommunity.com

[cytec]

@Dr-Bean @ymartin59 seems to be server related...
seems to be on/off a lot lately: https://github.com/SynoCommunity/spksrc/issues/3718

[acolomb]

I can confirm the "502 Bad Gateway" message from Cloudflare, for both packages.synocommunity.com as well as synocommunity.com. Using HTTPS in both cases. Seems like #3718 wasn't fixed after all...

[ymartin59]

Today repository is back online - I had to restart service, but CPU usage is still high.

[jucor]

Thanks @ymartin59 for restoring access -- very helpful :)

[jucor]

Woops, down again, 502 Bad Gateway, same as @acolomb above.

[ymartin59]

It really depends on system load... Not a DoS but it looks like there are more and more DSM connected to our repository, and probably packages history in repository has impact - I will do so clean up too.

[Qrbaker]

@ymartin59 Anything we can do to help? If the repo is on a cloud service, I would donate to spin up some more instances. Additionally, perhaps the CDN edge servers could cache packages?

[ymartin59]

@Qrbaker Thanks for proposal. Current system is an old linux running on 3 cores at OVH hosted by founder @Diaoul
redis server should act as cache, so that CloudFlare too, but top requests "check to know if an installed package has updates" do not take benefits of any of theses as URI parameters contain timezone, dsm minor version... so from my point of view, relevant option is to improve caching in python/flask https://github.com/SynoCommunity/spkrepo/issues/16

[Diaoul]

Infrastructure wise, the cost astronomical go put everything in the cloud.
Mainly because we serve 90+GB or packages every day.
The proper solution would be some private S3 hosting and public cloud for
the app only with proper caching.
CDN cannot cache POST requests and unfortunately this is how some NAS still
query the package server.

Le lun. 10 juin 2019 à 09:16, Yves Martin [e-mail] a
écrit :

@Qrbaker https://github.com/Qrbaker Thanks for proposal. Current system
is an old linux running on 3 cores at OVH hosted by founder @Diaoul
https://github.com/Diaoul
redis server should act as cache, so that CloudFlare too, but top requests
"check to know if an installed package has updates" do not take benefits of
any of theses as URI parameters contain timezone, dsm minor version... so
from my point of view, relevant option is to improve caching in
python/flask SynoCommunity/spkrepo#16
https://github.com/SynoCommunity/spkrepo/issues/16

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/SynoCommunity/spksrc/issues/3719?email_source=notifications&email_token=AACN55DFDMBBVEBDQD6IEL3PZX5V7A5CNFSM4HT5524KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXJDYJY#issuecomment-500317223,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACN55FZWF5RNFGXGODI5PLPZX5V7ANCNFSM4HT5524A
.

[Qrbaker]

@ymartin59 I use OVH as well, I'm happy to loan some time to help.

@Diaoul long shot, but could there be an option for distributed (i.e. torrent) packages? At the least, for people with transmission installed or a personal torrent client, they could get the spk and manually install as a fallback.

[BenjV]

Most likely the problem is not the downloaden of packages, but the constant pounding of every Synology Nas in the World to check if one of their packages needs an update.

[ghost]

I just bought a new synology and finally got to the point where I add synocommunity only to be blocked by this lovely issue

[chickenandpork]

@benjv “constant pounding”
@diaoul “CDN cannot cache POST”

Nginx caches POST requests:

proxy_cache_methods POST;

http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_methods

It may be enough to layer on a quick reverse-proxy with the caching of POSTs turned on to focus the load of the backend towards new and unique requests.

Nginx revproxy can be some in a very small alpine-based docker that would be light enough to run on eachother’s VPS or a small cloud offering. Effectively community-sponsored CDN that does POST.

This would also allow a fan-out of more proxies than backend to scale in a broad, shallow direction. Interested parties may be able to contribute materially by, for example, individually supporting a single proxy instance in a small-resources AWS or GCP or Azure image that points to the upstream and shards load.

[ymartin59]

@chickenandpork Alan, I am really not convince your proposal will help.

There is already:

• CloudFlare as CDN
• Nginx as reverse-proxy in front of uwsgi python code Flask
• and redis-server as Flask object-mapper cache over PostgreSQL

So again, please review https://github.com/SynoCommunity/spkrepo/issues/16 and any PR proposal is welcome, as I do not known these Python stacks and frameworks enough to implement it quickly.

[chickenandpork]

Hey Yves; you're correct -- even though "Nginx as reverse-proxy" is there, we can't assume that POST is cached; that said, the data in SynoCommunity/spkrepo#16 shows why my suggestion doesn't affect it, and why I should not try to add to a conversation while on commuting between jobs.

I'll armchair-quarterback another idea over on spkrepo#16 (and sorry -- you're quite gracious to answer everything)

[ymartin59]

@chickenandpork Be sure I do not answer everything - considering volume I receive from GitHub in my mailbox... I have to consider severity, urgency and priority for anything requiring my attention and time from community, my job and the remaining of my real life...

[kekukui]

maybe use zeronet or IPFS for P2P distribution?

• https://zeronet.io/
  https://github.com/HelloZeroNet/ZeroNet

• https://ipfs.io
  https://github.com/ipfs

[valdearg]

Any idea when it might be up? I could do with getting the mediainfo package again.

[CyberBlaed]

Any idea when it might be up? I could do with getting the mediainfo package again.

Still down for me, Http error 400 when I try, that said, manually download the SPK from the website and install as a manual package.. seems fine that way.. as a work around for now :)

[ymartin59]

Looks like server requires regular reboot (every ~100 days) probably because of some leaks in kernel...

[s1r14na]

synocomunity.com down again?

[Diaoul]

Indeed

[Diaoul]

Rebooted, should be ok now.

[DKeppi]

Sadly down again, could not download any of the packages!

[hgy59]

Still the same, may be internet in Paris is overheated?

Pic above shows status of https://synocommunity.com/packages.
Yesterday is was working and for one of my diskstations I had not yet replaced http by https for the repo https://packages.synocommunity.com to work.

[Diaoul]

Reboot in progress

[valdearg]

What's the cause of the errors? Seems to happen a few times.

What are the requirements for the site, need me to host it or something?

[Diaoul]

We have a huge load on the server. Caching is also sub-optimal. The whole backend would need a refactor.

[ymartin59]

A short term option may be to drop old packages versions... It seems to have effect.

[noplanman]

And it's down again :disappointed:

[publicarray]

@noplanman We are working a more permanent solution

If you feel adventurous you can try to use http://syno-repo.publicarray.com it proxies to packages.synocommunity.com , The json responses from our API are more efficiently cached but the binary downloads still point to packages.synocommunity.com since it's hardcoded in the API response. Having a few users on that domain would help me determine if there cache issues before the changes go live.

[Diaoul]

Rebooted again :disappointed:

[noplanman]

@publicarray Ok, have changed the repo source to yours, for science :wink:
@Diaoul What is the problem exactly, of the server dying so often?

[Diaoul]

I have not investigated the cause of this. The server is quite old and so is the software. Most probably the increase of users on the poorly optimized low end server has taken a toll on it :smile:

[noplanman]

I see, the classic then :joy_cat:

[ymartin59]

In my opinion, spkrepo code cache is not optimal, it should discard some request parameters in key when value does not depend of that parameters.
Many concurrent requests consume almost all CPUs, so answers get longer to come, nginx and/or cloudflare considers system as out-of-order, and so 502 are emitted for a safety period...
I have already tried to tune nginx and wsgi servers to accept delay on requests, queueing them if possible, but CPUs are lacking.
Then spkrepo probably suffers of a memory leak and the more it has processed requests, the more CPU is consumed... I tried to set parameters to restart wsgi after thousand of requests. End of 2019, a server restart was required once a month... now it is already every two days, or even more frequently.

[ymartin59]

Even if admin interface, I have to re-submit my requests up to 5 times before "catching a slot"...

[ymartin59]

spkrepo holds 257 version of packages, and thanks to generic packages (x64, aarch64, armv7), number of builds are expected to decrease... but it still requires regular manual cleanup to discard old versions and reduce data processing.

[Diaoul]

Although the SQL model of the database is great to avoid inconsistencies, it causes very heavy requests on the database with huge JOINs thus taking a long time. This coupled with improper caching results in excessive load.
To compute a response we need to pull all the versions of a package, filter on architecture, take DSM version into account and determine with those constraint (and some more) what is the "latest" version.
Looking back on the design, I think I wouldn't mind something simpler on the database side with more application code for consistency.
Ideas welcome.

[nkqiaolin]

Sorry to raise : seems http://packages.synocommunity.com is down again? @Diaoul

[Diaoul]

See https://github.com/SynoCommunity/spksrc/issues/3981
Migration should start as soon as I find some time, most probably during the weekend if all goes well.

[nkqiaolin]

That's cool! Thanks so much~~~

[thehoff]

@Diaoul happy to help if I can. located in AU and could help out supporting the servers under your guidance if that helps at all. Monitoring, admin etc. Just hit me up if you want/need help.

[matrixn]

Looks like today in London http://packages.synocommunity.com/ is down again... do we have a mirror for this repo? I would like to host one if is possible...

[thbl]

is it down again ?

[matrixn]

yeah...still down... at least from London i can't access it

[publicarray]

@matrixn @thehoff While we appreciate the offer, the current architecture does not lend itself well for mirroring. If have suggestions on how this can be done have a look at https://github.com/SynoCommunity/spkrepo.
This time around the outage is due to a server migration and not to an overloaded server. See #3981 Also once the server is migrated Fastly.com has kindly offered to be our CDN.

[carloscae]

Hi there. Are the SPKs available somewhere so I can manually download/install?
I came into a stupid situation of migrating my volumes and need to get one package there quite urgently. :(

(Hass.io in this case)

Thanks.

[publicarray]

@carloscae https://github.com/publicarray/spksrc/releases/download/hassio-20191224/hassio_x64-6.1_20191224-1.spk

[baya44]

Any update about the migration ? I need the git spk for my ds1817+ after changing my volume last week.. There is any link to find it ?

[publicarray]

Yes see #3981 all further updates regarding the migration will be announced on that issue.
@baya44 I just build git for you: https://github.com/publicarray/spksrc/releases/tag/git-2.24.1 for the ds1817+ you need the git_x64-6.1_2.24.1-12.spk

[jbega]

Hi.
I have been trying to install the fuse libraries for several days but the synocomunity website is down. How could I install them on my 218+?
Thank you

[hgy59]

@jbega in #3981 you find the current status. Migration to the new server is still in progress.

Do you mean you need synocli-disk for the fuse libraries?

[jbega]

So, is there no other way to install fuse libraries?

[hgy59]

on my DS218+ there is already libfuse (and fusermount) installed

lrwxrwxrwx 1 root root     16 May 26 11:39 /usr/lib/libfuse.so -> libfuse.so.2.9.4
lrwxrwxrwx 1 root root     16 May 26 11:39 /usr/lib/libfuse.so.2 -> libfuse.so.2.9.4
-rwxr-xr-x 1 root root 245792 May  7 02:40 /usr/lib/libfuse.so.2.9.4

And this are the SynoCommunity Packages containing libfuse and fusermount (currently version 2.9.9)

• synocli-disk
• synocli-net (contains sshfs that installs libfuse)
• sshfs (contained in synocli-net, no dedicated package anymore)
• itools (includes ifuse that depends on libfuse)

[hgy59]

@jbega please open a new issue for further discussions regarding fuse libraries.

[jabarel]

When will be up... i need transmission for 718+

[brucey531]

Does anyone have the Synology package for Mosquito so I can install manually until Syno community is back.

Thanks

[eRaz3r]

Have a big Problem too. I need the Syncthing-File for my 212+ because it is broken after updating. Are there any news, what is the problem with SynoCommunity about? I can't klick on my architecture to dowlad the .spk manually.

[acolomb]

@eRaz3r I wish I could be of help with an updated testing build from PR #4027 which should fix your issue. However the DS212+ has an 88f6282 architecture and should be compatible with the armv5 build, which is not included there.

Possibly armv5 is listed as an unsupported architecture for the package? I can't check myself right now but maybe these hints will enable you to find a solution.

Totally unrelated to the offline package server though.