Spksrc: PSA: Server Migration

[publicarray]

The website and package registry will be offline temporarily as we migrate the server

When?

Starting 30th of April and we plan to have the migration completed within 7 days⌛️
Migration is ongoing, current ETA is May 24th⌛️

Why?

The server has been running on an old OVH instance type and OVH is deprecating the instance we where using. @Diaoul, @ymartin59 and many more @SynoCommunity/developers and contributors are running the project in their spare time, so we naturally left it to the last minute 😅, please be patient as we migrate the site.

Huge thanks go to @Diaoul and @ymartin59 for keeping it updated and running for so long. (And restarting when it when it became overloaded 🔥)

Current Status (2020-05-28 00:17 UTC)

• [x] Backup
  
  
  
  • [x] SPKs
  
  
  • [x] Database
  
  
  • [x] Configuration
  
  
• [x] Repository code refresh: spkrepo
  
  
  
  • [x] Update all dependencies and fix failing code
  
  
  • [x] Use poetry for packaging
  
  
  • [x] Make the test suite pass
  
  
  • [x] Use pytest as test runner
  
  
  • [ ] Make PyPI repository
  
  
  • [x] Use pre-commit hooks for linting
  
  
  • [x] [black](https://github.com/psf/black)
  
  
  • [x] [flake8](https://flake8.pycqa.org/en/latest/)
  
  
  • [x] [isort](https://github.com/timothycrosley/isort)
  
  
  • [x] My own poetry export
  
  
  • [x] Create GitHub Actions CI
  
  
  • [x] Run linting
  
  
  • [x] Run tests
  
  
  • [ ] Build artifacts
  
  
  • [ ] Release on PyPI
  
  
  • [x] Create Docker image
  
  
  • [x] Use Docker Hub as a CI step
  
  
• [x] Buy a new server
• [x] Configure the new server
• [x] Create a complete Docker stack based on configuration backup
• [ ] Publish the Docker compose file
• [x] Restore Database and SPKs
• [x] Deploy the stack on the server
• [x] Test everything is OK
• [x] Route traffic to the new server
  
  
  
  • [x] Packages
  
  
  • [x] Website
  
  
  • [x] API
  
  
• [x] Monitor server performance

[MrSparklle]

Does migration finished? I still have issues to download packages from Synocommunity.

[publicarray]

Not yet but @Diaoul in on the job. If all goes well, by the end of the weekend we’ll be up and running. But don’t hold me to it. The site is returning stale items from Fastly CDN so some popular devices/packages might still work. The DNS setting changed and traffic is no longer routed through fastly

[thehoff]

long term if you want/need another admin to help support this project let me know. happy to share my details in private.

[raymondtrueman]

Any updates regarding the migration? @Diaoul

[Diaoul]

I've updated the issue with current status and will try to keep that updated. I think I can finish the docker stack tonight and initiate SPK transfer to the server this evening.

I can only work on this during late evenings (11PM-4AM) so please be patient :hourglass_flowing_sand:

[Diaoul]

Just updated the status. Tests ongoing. Looking good so far.

[matrixn]

I am still unable to access the package repo .. not sure how @Diaoul was able to test and report a success :))
I've even tried different dsn's like google ones and 1.1.1.1 so far no luck to get them working on Virgin Broadband in London....

[hgy59]

@matrixn please read the provided status in the "Current Status List". The traffic is not yet routed to the new server.

[DcMcIx]

Hello, Thanks for the great job you're doing to make our NAS's more user friendly
In the mean time I was able to compile (with your framework) the synocli-monitor and it just worked.
sure the package center is more simple to use, can't wait to add it again in my Addon center :)

[Stanley008]

When will be the server online again? Thx

[ljstronge]

Looking forward to this being back on. I was going to set up my full media stack today (Sabnzb etc) but noticed this was offline.

Keep up the good work and let us know if there is anything we can help with!

[Diaoul]

Packages are now available but the website is still down.

[ljstronge]

Thanks. I can add the Package Source but get failed to download on any app. Appreciate your work.

[psergiu]

Same here ... package source can be added, list of packages visible in Package Center but installs fail with: Failed to download "packagename".

/var/log/messages:

2020-05-26T17:38:07-05:00 NAS-DS1019 synoscgi_SYNO.Core.Package.Installation_1_install[16402]: pkgserver.cpp:698 Failed to curl perfor
m, code=3, err=Error, url=/tcl/4/tcl.v4.f4458%5Bapollolake-avoton-braswell-broadwell-broadwellnk-bromolow-cedarview-denverton-dockerx64-grantley-kvmx64-x86-x86_64%5D.spk?arch=apollolake&build=24922, mode=install

Update:
You can download https://packages.synocommunity.com//tcl/4/tcl.v4.f4458%5Bapollolake-avoton-braswell-broadwell-broadwellnk-bromolow-cedarview-denverton-dockerx64-grantley-kvmx64-x86-x86_64%5D.spk just fine - the error shows up with the added ?arch=apollolake&build=24922

Thanks !

[Diaoul]

I've investigated this for hours now and I can't find a reason why we can download manually but not with the package center. Maybe it's just a DNS caching issue. I will continue tomorrow.

[szyb]

Hi @Diaoul ,
I look into the json response from the server and it looks like "link" node does not have full URL with protocol and server name.
i.e. for sslh package (DS718+, build 25423) there is:
"Link": "/sslh/3/sslh.v3.f4458%5Bapollolake-avoton-braswell-broadwell-broadwellnk-bromolow-cedarview-denverton-grantley-kvmx64-x86-x86_64%5D.spk?arch=apollolake&build=25423",

while for the same package (but for DS413) there was before (I get it from my website cache):
"Link": "https://packages.synocommunity.com/sslh/3/sslh.v3.f4458%5Bqoriq%5D.spk?arch=qoriq&build=25423",

Hope that help

[publicarray]

@Diaoul Thanks @szyb I think you found the crux of the issue.

Shame that Synology doesn't support relative links.

[Diaoul]

Thanks, I didn't touch that part so I don't know why this is now relative
links...
Will try to fix this.

[Vf58]

@Psergiu,

Thank you for the info. I download manualy Transmission packages : with :
https://packages.synocommunity.com//transmission/16/transmission.v16.f4458%5Bapollolake-avoton-braswell-broadwell-broadwellnk-bromolow-cedarview-denverton-dockerx64-grantley-kvmx64-x86-x86_64%5D.spk?arch=avoton&build=24922

@Diaoul,

Thank you for your job and good luck,

Philippe

[ljstronge]

How can I download packages manually (Nzbget & CouchPotato) in the interim? Thanks

[szyb]

@ljstronge until the fix is ready you may use https://search.synopackage.com, but first setup your model & version, find the package in the synocommunity source and click download icon. Just added and deployed such possibility as many people asking for that.

[prometheus247]

It is not possible to change a manual downloaded package back to a repository source, without re-installing, right?
But from settings migration perspective, it could be an easy copy & paste of the @appstore folder - nor?

[Safihre]

Just give @Diaoul a day or 2 more days, guys 🏎🏁

[ljstronge]

Just give @Diaoul a day or 2 more days, guys 🏎🏁

Yep. Absolutely. Apologies for being inpatient!

[Diaoul]

I addressed the issue regarding relative links being sent instead of absolute ones.
Packages services should be completely restored, still working on the website and API.

[Diaoul]

Website is alive, API should be too. I've updated the status.

[publicarray]

Fastly now redirects all requests to HTTPS 🎊
@Diaoul What happened to the Fastly logo and sponsor message?

[Diaoul]

It was not part of the backup as it has been done after April 30th.

Le jeu. 28 mai 2020 à 10:55, Sebastian Schmidt [e-mail] a
écrit :

Fastly now redirects all requests to HTTPS 🎊
@Diaoul https://github.com/Diaoul What happened to the Fastly logo and
sponsor message?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/SynoCommunity/spksrc/issues/3981#issuecomment-635212055,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACN55FUR6ZFQMED4KJVILDRTYRJTANCNFSM4MTVRSCQ
.

[seaquake]

Thanks so much!

[ardenyang]

Website is alive,Thanks so much!

[ymartin59]

@Diaoul Great. My login seems to fail and I can no longer access to admin pages. May you please investigate? It may be an issue with my account only.
Please also create a system Linux account "yma" and register my public key for it?

[hgy59]

@Diaoul what is the schedule to update/fix the web page?

• Fastly logo and sponsor message
• Updated Easy Install section with screen shots
• all links to manually download packages are missing

[Diaoul]

Any code that is not part of spkrepo has been lost. The templates are the
ones from April 30th. I don't have an ETA on those fixes, probably next
week but ymartin has access so maybe sooner.

[hgy59]

@Diaoul thanks for your effort on spkrepo! I was not aware of links to manually download packages are not provided by spkrepo.

[th0ma7]

@Diaoul thnx for your work on this.
I was wondering (for my own geek curiosity), how things are in comparison with the previous setup?
My understanding is that it much more responsive and now running on an updated platform?

[ymartin59]

Notice publish is broken: https://github.com/SynoCommunity/spkrepo/issues/47