Server: 🛠 Nextcloud update experience for development instances

[jancborchardt]

So on every update to a new major dev version I get on a whole lot of apps "the files of these apps were not properly replaced", and then "These incompatible apps will be disabled". And I have to manually go through updating them.

Now I used to have a script to just do updates automatically: https://github.com/jancborchardt/nextcloud-scripts/blob/master/nextcloud-update.sh
But it doesn’t work so simply anymore since most apps need to be built. And most apps use some slightly different method which makes it completely confusing. Here just an excerpt:

• activity: does not have development setup info in readme ;) (it’s git pull, just works) @nickvergessen
• bookmarks: doesn’t have dev setup in the readme either (it’s also git pull, just works)
• calendar: clean package-lock.json, git pull, make
• deck: clean js/package-lock.json, git pull, make build-js
• files_pdfviewer: git pull, just works
• firstrunwizard: clean js/firstrunwizard.js & js/firstrunwizard.js.map, git pull, make dev-setup
• logreader: clean build/* & package-lock.json, git pull, make
• mail: clean package-lock.json, git pull, make dev-setup
• notes: git pull, install gulp, gulp
• notifications: clean js/notifications.js & js/notifications.js.map, git pull, make dev-setup
• recommendations: clean js/main.js & js/main.js.map, git pull, npm run dev
• social: git pull, make dev-setup
• talk/spreed: git pull, make dev-setup
• twofactor_totp: does not have development setup info in readme ;) @ChristophWurst
• twofactor_u2f: clean package-lock.json, git pull, composer install, npm install, npm run build
• viewer: git pull, make dev-setup

How difficult is it to have one simple standard for all apps:

• git pull, make (make dev-setup is also fine, but then let’s standardize on that)

Or even:

• git pull, which then does make automatically with some post-git-pull-hook-magic?

cc @rullzer @MorrisJobke @skjnldsv @ChristophWurst @nickvergessen @juliushaertl – am I missing something here? As someone who is involved in many different apps, every update feels like a chore.

[ChristophWurst]

If apps follow the documented convention you simply run krankerl up for installing dependencies & building. No configuration required. So far I have not found an app where this wouldn't work.

[nickvergessen]

Not sure where you got the git checkout -- ... From. Make dev-setup should take care of that. Also for all vue based apps I did so far stufd is commited, so git pull works and is enough, but its more pain full on prs then due to rebasing.

I agree that it's a pain. Which is also why my dev setup sets the enabled to false for all apps by default and i then enable what i need this time.

I think we should have one way to do things, or at least as few as possible. Maybe we can agree on:

• It works on git pull or
• It works on git pull && make

You can still use krankerl, but it has the disadvantage that you need to install yet another tool. So maybe we can hide that with make and make make install it if its not found?

[nickvergessen]

git pull, which then does make automatically with some post-git-pull-hook-magic

I would dislike that, as it might cause too much of a hazzle while working together as you automatically rebuild all the time, althiugh it might not be needed because js wasnt touched, etc.

[skjnldsv]

Yep, usually I just git checkout master, (git stash if anything changed locally and I don't care), make dev-setup, make build-js.

I feel like the process is clean enough. Having pending changes on your local repo and not being able to pull is a tad irrelevant, this is how git work and if you want a clean setup, you can still use git clean or git reset --hard :woman_shrugging: :wink:
But I agree, we need to unify the make commands (which we're slowly doing while the apps migrate to vue) :)

On a side note, the apps that gets disabled on every update is really annoying, especially since we're supposed to have integrated a way to enable apps even if they're not supported. We should find a better way of doing so I think! :thinking:

[jancborchardt]

If apps follow the documented convention you simply run krankerl up for installing dependencies & building. No configuration required. So far I have not found an app where this wouldn't work.

@ChristophWurst Sure – however then do we agree on making this a standard? So far I haven’t seen krankerl referred to in any app readme.

---

Not sure where you got the git checkout -- ... From. Make dev-setup should take care of that. Also for all vue based apps I did so far stufd is commited, so git pull works and is enough, but its more pain full on prs then due to rebasing.

@nickvergessen yeah the git checkout -- could also simply be a git stash. It’s about the files that a build leaves behind or modifies, which makes git pull get blocked sometimes and say "your local files would be overwritten etc" if these files changed in the repository meanwhile.

I think we should have one way to do things, or at least as few as possible. Maybe we can agree on:

* It works on `git pull` or
* It works on `git pull && make`

Totally agree – this is exactly what I wrote in the original post. :)

You can still use krankerl, but it has the disadvantage that you need to install yet another tool. So maybe we can hide that with make and make make install it if its not found?

Would tend to agree there too. Especially as krankerl is not yet used in _any_ of the apps, adding yet another tool on top, especially a Nextcloud-specific one only makes it more confusing.

---

Yep, usually I just git checkout master, (git stash if anything changed locally and I don't care), make dev-setup, make build-js.

I feel like the process is clean enough.

@skjnldsv that’s you as someone who codes on Nextcloud _daily_. ;)

The thing here is I’m talking about clean local repositories, simply an earlier version build, where there’s build file residue – there shouldn’t be the necessity to stash anything.
And then additionally, why a complex command like make dev-setup if make could be possible as shorthand? :)

I know for experienced devs, especially people who work on Nextcloud all day every day this doesn’t seem like much of a hurdle – but it is. :)

[nickvergessen]

I know for experienced devs, especially people who work on Nextcloud all day every day this doesn’t seem like much of a hurdle – but it is. :)

I know you didn't because you replied to it, but I feel my comment is being ignored. I totally thing it's a big hurdle which is why I keep shipping build files, so git pull just works.

[jancborchardt]

@nickvergessen sorry – I actually agree with everything you said, which is why I :+1:’d your comment. :heart: :) (I separated the replies to Christoph, you and John with a line, which was probably not visible – the last part was mostly as reply to @skjnldsv’s comment ;)

[MorrisJobke]

I fully get the reason for this. We are facing multiple problems here:

• either assets are checked in and git pull just works (it has multiple drawbacks nevertheless: more conflicts on PRs, harder to backport fixes, checked in assets even if they are technically not needed)
• a Makefile that is 99% the same for every repo and thus needs to be updated if there is a critical change that every app should respect (like deleting some files that should not be shipped) - there the idea came up to unify this information inside the krankerl tool that knows what files need to be removed, what are typical build steps, how to sign an app, how to package it, where to upload the app and how to publish it in the our appstore - all in one tool - sure it's one more tool, but otherwise we duplicate those steps in Makefiles and this makes it harder for many maintainers to be up to date

To solve the initial issue that @jancborchardt had a git pull solves that because then the apps can usually be enabled already (even if the JS part is not known fully, but at least the screen you posted will not happen):

Run git pull in all app directories:

find apps* -name .git -exec git --git-dir={} --work-tree={}/.. pull -p \;

Checkout a branch in all app directories (for testing stable15 for example):

find apps* -name .git -exec git --git-dir={} --work-tree={}/.. checkout stable15 \;      

[jancborchardt]

Upgraded @MorrisJobke’s command:

find apps* -maxdepth 2 -name .git -exec sh -c 'cd {}/../ && printf "\n\033[1m${PWD##*/}\033[0m\n" && git checkout master && git pull --quiet && git --no-pager log -3 --pretty=format:"%h %Cblue%ar%x09%an %Creset%s" && printf "\n" && git branch --merged master | grep -v "master$" | xargs --no-run-if-empty git branch -d && git fetch --prune --quiet && cd ..' \;

It automatically:

• goes through all apps which are not shipped via server
• shows the app name in bold and uses whitespace for separation
• changes to master and pulls quietly
• shows the 3 most recent commits for context
• removes branches merged into master
• … could even do the build steps if they are consistent for the apps (like make)

Basically a 1-command version of the app update part of the nextcloud-update.sh script.

Is that command something we could potentially include e.g. in the /build folder as a shell script so that people can simply do ./build/update-apps.sh? I can do a pull request … :)

[MorrisJobke]

@jancborchardt Really nice 👍 🚀