Spksrc: Automate new upstream version availability detection and processing

[ymartin59]

Here is first specification of my idea. Python sounds me a good choice for implementation.

Expected behavior

• Without package location as argument, loop on packages, either cross, native or spk
• Parse Makefile to get version and project download location
• Test if a newer version is available according to "supported" download sources:
  
  
  
  • For a git or subversion repository, list tags ordered by dates
  
  
  • For a file storage, either ftp listing, html download page, parse content to list available files and target URI
  
  
  • Maybe some specific support will be required for common code hosting platforms
  
  
• Update package version in Makefile and increase revision
• Try to download archive in distrib and report error in FAILED-version-update.log
• Generate digests from archive
• Any previous steps may fail, for instance test algorithm does not support source or failed to compare versions, and if so will generate a FAILED-version-update.log with details
• Run make arch-88f6281 first and by default to fail fast and report compilation issue. Generate FAILED-arch-99f6281.log file or OK-arch-99f6281.log
• Generate/update PLIST or mark it as modified for manual review
• If script has all-archs switch turn on, loop over all archs and produce each FAILED/OK log file

How to use

When over, browsing files marked as new or modified in git working copy is straight forward.

With luck and probably for most minor updates, removing log files and review before commit may be the only remaining job

[GuillaumeSmaha]

I made a POC to do this, I will share it when I will obtain a good result. Currently, I just checks the link availibility.

@ymartin59 What do you think about try to ping on possible new file by incrementing the version ? It can be lesser specific than parsing html page.

Also parsing Makefile can be complex when there is specific call like boost which use substr (https://github.com/SynoCommunity/spksrc/blob/master/cross/boost/Makefile#L4).
Currently, substr is the only special case.

[GuillaumeSmaha]

You can find the POC here :
https://gist.github.com/GuillaumeSmaha/5545f6e48b5d3ce35399d6b3727c4a9c#file-check_package_availability-sh

The result on master :

cross/domoticz -> Not Available: svn://svn.code.sf.net/p/domoticz/code/trunk
cross/iksemel -> Not Available: http://iksemel.googlecode.com/files/iksemel-1.4.tar.gz
cross/imagemagick -> Not Available: http://www.imagemagick.org/download/ImageMagick-6.9.5-10.tar.xz
cross/jappix -> Not Available: http://download.jappix.org/1.0.1/jappix-1.0.1-one.zip
cross/libcap2 -> Not Available: http://ftp.sunet.se/pub/Linux/kernel.org/linux/libs/security/linux-privs/libcap2/libcap-2.22.tar.gz
cross/libev -> Not Available: http://dist.schmorp.de/libev/libev-4.22.tar.gz
cross/memcached -> Not Available: http://memcached.googlecode.com/files/memcached-1.4.15.tar.gz
cross/mpd -> Not Available: http://downloads.sourceforge.net/project/musicpd/mpd/0.16.7/mpd-0.16.7.tar.bz2
cross/phpmemcachedadmin -> Not Available: http://phpmemcacheadmin.googlecode.com/files/phpMemcachedAdmin-1.2.2-r262.tar.gz
cross/pyload -> Not Available: https://github.com/pyload/pyload/releases/download/v0.4.9/pyload-src-v0.4.9.zip
cross/serf -> Not Available: https://serf.googlecode.com/files/serf-1.2.0.tar.bz2
cross/squidclamav -> Not Available: http://downloads.sourceforge.net/project/squidclamav/squidclamav/6.10/squidclamav-6.10.tar.gz
cross/sslh -> Not Available: http://www.rutschle.net/tech/sslh-v1.17.tar.gz
cross/zsh -> Not Available: http://www.zsh.org/pub/zsh-5.2.tar.xz

[GuillaumeSmaha]

Another issue to fix : Sometimes few packages like native/mono or cross/rtmpdump doesn't have a SPK_VERS variable but in case of mono, there are SPK_VERS_MAJOR and SPK_VERS_MINOR and for rtmpdump, this is a git repo as source.
It will be better to define all time SPK_VERS even if the var is just equal to the HASH for git or to the concatenation of version variables :

PKG_VERS = $(PKG_VERS_MAJOR).$(PKG_VERS_MINOR).$(PKG_VERS_PATCH)

[GuillaumeSmaha]

Here the first result of my script : http://pastebin.com/azZyyn2Z
Script : https://gist.github.com/GuillaumeSmaha/5545f6e48b5d3ce35399d6b3727c4a9c#file-search_package_update-sh

• Get new tag for GIT and SVN from the current hash/revision
• For hosting file, check if parent directory is a files list, then get the new versions using filename
  --> Else if parent directory is a html page, then search for any version in the page (may be a news about the new versions) and try the URL by replacing current version by the found versions in the filename
  --> Else Not Implemented

[ymartin59]

@GuillaumeSmaha Great job but from my point of view such code would have been more robust and easier to maintain if written with Python, typically when error handling is concerned.

[GuillaumeSmaha]

@ymartin59 Yes, python is more robust. I planed to stop the script at this point. The aim was to check if it was possible to have good results. And the next dev needs to be more specific.

[GuillaumeSmaha]

I created a repository with the python script which returns the same result as the bash script :
https://github.com/GuillaumeSmaha/spksrc-check-update

• Better makefile parsing with pyparsing
• Simply better with python :D

(svn dependency required the PR https://github.com/dsoprea/PySvn/pull/85 to work)

[GuillaumeSmaha]

Yeah ! First step is finished !
The script can now check the current version or get the new version for all package except these :

cross/dobby : Old github upload API. And the package no more used
cross/domoticz :  Server down
cross/ejabberd :  Server down
cross/gc :  Error to fix : Script can't download page content http://hboehm.info/gc/gc_source/
cross/jappix :  Server down
cross/libcap2 :  Server down
cross/mpd : mpd is not found in the project https://sourceforge.net/projects/musicpd/files/
cross/nzbget-testing : Hard to detect because the package uses `PKG_VERS` and `PKG_REV`
cross/pear : No version in package
cross/pyload : File is not available. Need to update and check patches.
cross/unzip : Error to connect to the ftp server.

I also create the PR #2708 to add a new var DOWNLOAD_PAGE which defines the page with the files list.
For example : On http://www.sqlite.org/, there is not a download link to a file, and the list of files is located on page http://www.sqlite.org/download.html

Updated :
cross/gc is now fixed using an User-Agent in the HTTP request

Updated 2:
cross/nzbget-testing can be solved by removing PKG_REV and adding revision in PKG_VERS
See patch nzbget-testing.patch.txt

[ymartin59]

@GuillaumeSmaha You really did a great job. Do you mind if I request your repository to be hosted by SynoCommunity organization for issues and pull requests ? I really think it makes sense

May I ask you for a small improvement: follow DEPENDS and BUILD_DEPENDS when fetching a dedicated package ? In that case, running with --package=spk/ffmpeg would report all included cross/ and native/ libraries required by the tool ?

[GuillaumeSmaha]

@ymartin59 I miss your message and no time the last months to do it.
But come back to continue.
For the build part, it could be interesting to check if we could create a PR and use travis (for example) to validate the build

[ymartin59]

You're right for building. A ticket specifically concerns Travis support: #2990

[Safihre]

@GuillaumeSmaha This looks great, I see there's 2 branches, what does the feature/add_build do?

[hgy59]

I don't like automatic updates. An automatism for finding broken downloads and to force SPK_VERS updates is fine.
But a package update should only be done if it is worth to, and this should be carefully selected. Each package version should be handled like a LTS version (since there is not way to download an older version anymore after update in package center).

• Consider update/migrate handling in spk
• Consider patched files will change
• Consider new or removed dependecies

For broken downloads I recently found very different reasons:

• repository does not exist anymore (dnscrypt 1.9.5) repository is reused for version 2.x and 1.9.5 is gone
• repository moved from svn/google-code/custom to github (jappix, phpMemcachedAdmin)
• older versions were moved in github repository to different location (imagemagick)
• used version not available for download anymore (squidclamav)

[GuillaumeSmaha]

The aim of the branch add_build is to update Makefile and build package, but this part is not already done.
Currently, I can detect and found new version for package.

[GuillaumeSmaha]

I impress myself when I reviewed the code of add_build branch. I totally forgot the improvement done.
Summary:

• Check new package with option for pre-release, major version: Done
• Update Makefile: Done (need to update digests file)
• Build package (TODO)

@ymartin59 I saw your post: Yes, I agree to move it to SynoCommunity organization

[noplanman]

This looks pretty, what's the status?

[hgy59]

What about implementing this as a bot?
Like dependabot that looks for known vulnerabilities in dependent python modules, this could be implemented as a bot, that notifies about packages, for which updated sources are available.
The first benefit would be the notification, but as dependabot, it could provide a PR with the update. And for a PR we will have the github build checks too.

_And I am dreaming of automated package installation and functional testing with a virtual DSM environment, implemented as github action..._