Server: Install app via command line - occ app:install

[r2evans]

The occ list of commands allows enabling apps that are already installed, but not installing them. In a growing use-case, nextcloud can be provisioned on docker or a similar containerization/virtualization setup. With automated setups, it would be very useful to be able to install known apps automatically, such as "Calendar" and "Contacts", neither of which is (easily) installable without the GUI.

There might be a couple of levels of supporting this functionality:

• Allow "any" app that is installable from the nextcloud app-store;
• Only allow automated install of "Approved Nextcloud apps" (such as the list of supported apps).

I think the latter is almost certainly safer, and in concert with your recent categorization of app levels.

Get a bounty if you fix this issue, or help raise the bounty if you want someone else to fix it:

[MariusBluem]

We could use the id's here: occ app:install news ... maybe we could introduce this together with the new appstore - but I am not really sure about the use case 😁

cc @BernhardPosselt

[r2evans]

Alright, more thoughts on use-case, with the underlying assumption that it is instantiated with something like docker.

When a new instance of nextcloud is spun up, it has only the default server-included apps installed and enabled. Though I can enable "user_ldap" on the command line, in order to install another app like "calendar" or "contacts", I need to log in and work through the GUI. This works fine for "straight-forward" setups, but security, consistency, and robust setups benefit from automation. If this only happens at the first install, then that's it.

When you upgrade a docker instance, it typically goes something like, for instance upgrading from 9.0.53-1 to 9.0.53-2:

# this just downloads the new image, nothing done yet
$ docker pull nextcloud:9.0.53-2
$ docker stop nextcloud
# these two commands remove all files except persistent "volumes";
# anything not in one of these "volumes" is completely discarded
$ docker rm nextcloud
# this is the actual upgrade
$ docker run [options] nextcloud:9.0.53-2 --name nextcloud

(This is often automated with docker-compose.)

Docker supports persistent volumes. From their definition:

• Volumes are initialized when a container is created. If the container’s base image contains data at the specified mount point, that existing data is copied into the new volume upon volume initialization.
• Changes to a data volume will not be included when you update an image.

Two common uses for these "volumes" is for user data (such as the data/ directory of nextcloud) and logs. Some docker images have suggested including the apps/ directory as a volume; two side-effects from this: (1) when you upgrade, all of your previous apps are there still, meaning you may be able to php occ app:enable programmatically (I know they are disabled on upgrade for a reason); and (2) _none_ of the apps are upgraded, whether authorized server apps or third-party. The former seems good, but the latter easily causes problems.

Two alternatives to GUI-installation come to mind:

1. Store apps other than "official nextcloud apps" (shipped with the server) in a second apps/ directory, in a manner that is supportable (such as within the webroot, https://github.com/nextcloud/server/issues/130). Make apps2/ a volume.
2. Enable (re-)installation of non-official apps on the command line. This resolved both the problem on nextcloud upgrades as well as on the initial install.

The first option benefits from keeping the intended version installed. The latter helps on initial install and ensures new spun-up instances have the most recent and approved version installed.

There is a pressing argument _against_ storing apps in a persistent volume: lack of persistence provides assurances in security and performance. When not-persistent, even if a container is breached and modified files left in place, when the container is reset (with stop, rm, run) the breach is reversed. By putting apps2/ in a persistent volume, we negate this advantage of non-persistence of code/executables.

This suggests alternative two as the preferred solution: by allowing a fresh install every time an image is spun-up, we don't run the risk of modified code persisting between otherwise-pristine images.

[r2evans]

@MariusBluem, any thoughts or feedback on my use-case?

[DevCybran]

I second this request for the very same reason @r2evans explained in the use case. An automated installation and upgrade of verified apps in non-persistent storage provides significant security benefits, as well as an easy way to automatically deploy nextcloud with additional features and without requiring user interaction.

[r2evans]

An additional note: currently when I restart the docker instance and go to (re)install the calendar app, it is marked as already-enabled, even though it is not installed in the apps directory. This is not itself causing a problem, but it does indicate an inconsistency between the database and installation. (I then disable then re-enable to install it.)

[r2evans]

@LukasReschke, you seem to be the go-to person for app management (among many components of nextcloud) within the server. Do you have any thoughts on this? Other than providing code and a PR (and petitioning for feature-popularity votes), what would make this security feature more enticing?

(By "added security", I'm referring to lowering the entry cost and increasing the default security posture of a dockerized nextcloud.)

[linuxrrze]

Just wanted to support this request for occ app:install:

I'm preparing a presentation and live demo (docker based) of nextcloud for an upcoming Linux user group event, and I'd like to supply everyone with his/her own nextcloud environment (with a big variety of apps enabled an pre-configured).

[mdik]

How is it possible that this issue only has 5 :+1: ?

[ChessSpider]

I made it six 👍

[enoch85]

@linuxrrze Have you checked out the Nextcloud VM? https://www.techandme.se/nextcloud-vm/ & https://github.com/nextcloud/vm

[Xeboc]

From what @enoch85 shared:

wget -q https://github.com/nextcloud/calendar/releases/download/v1.5.3/calendar.tar.gz -P /var/www/nextcloud/apps
tar -zxf /var/www/nextcloud/apps/calendar.tar.gz -C /var/www/nextcloud/apps
rm /var/www/nextcloud/apps/calendar.tar.gz
sudo -u www-data php /var/www/nextcloud/occ app:enable calendar

[r2evans]

This answers the question of "how do I programmatically install the calendar app into my nextcloud instance". What I envision is a programmatic way, iterating over each desired app:

• given the app name/id, find the most recent version (given the current instance of NC)
  
  
  
  • the brute-force way is to get https://apps.nextcloud.com/apps/spreed and parse the HTML for the <div class="app-download"> (there's more to it), and manually parse out the textual NC versions in the table and extract the highest-version found. This is ugly and not robust to HTML changes.
  
  
  • more efficient: what is the API for the app-store? All I've found is the API for app developers to publish/deploy
  
  
• download it
• untar it
• occ app:enable it (per the script mentioned above)

Pseudo (bash-like) code (using a fake API):

NCVER=$(php occ config:system:get version) # 11.0.3.2
NCVER="${NCVER%%.*}"                       # 11
NCAPPS="/var/www/nextcloud/apps"
NCOCC="/var/www/nextcloud/occ"
INSTALLSTRING="contacts,calendar,spreed"   # or perhaps ${@} if cmdline-driven
IFS="," read -ra INSTALLAPPS <<< "${INSTALLSTRING}"
for app in "${INSTALLAPPS[@]}" ; do
  TMPFILE=$(mktemp)
  DLURL=$(curl -d "ncver=${NCVER},id=${app}" https://apps.nextcloud.com/API/1.0/getNewestUrl/)
  # FIXME: should get and verify its sha256sum
  curl -O "${TMPFILE}" "${DLURL}" \
    && tar -C "${NCAPPSDIR}" -zxf "${TMPFILE}" \
    && php "${NCOCC}" app:enable "${app}"
  [ -f "${TMPFILE}" ] && rm -f "${TMPFILE}"
done

Lacking an actual API to determine the most recent (and NC-version applicable) download URL, this script should actually work (albeit needing more error-checking, etc). And if I can figure out how to robustly do that API hit, I'll probably use a script like this. (Caveat: I wrote this without testing, mostly for concept than actual execution. Don't try this at home!)

But the elegant, consistent, and long-term maintainable answer is that it should be implemented as occ app:install <appid>.

@enoch85: many of your scripts seem to handle underlying OS dependencies (packages, snaps, etc). Disregard those; meaning, think about apps that are either handled completely in the PHP .tar.gz file, or assume that the external dependencies are already installed.
Question: where will the basic process of "download, verify, untar, occ app:install" fail?

[mdik]

Maybe it is worth mentioning that there is an api that returns a json from which the download url and signatures of all apps could be extracted. The curl looks something like

curl https://apps.nextcloud.com/api/v1/platform/11.0.3/apps.json -H 'If-None-Match: "1-1-2016-06-17 23:08:58.042321+00:00"' > nextcloud-appstore.json

So it seems to be rather low-hanging fruit to me, although I'd need to get paid in order to have the time to do it properly :-/

[enoch85]

@mdik https://www.bountysource.com/issues/38017770-install-app-via-command-line-occ-app-install

[r2evans]

That's a useful download link, thank you @mdik. Is there a queryable API against it? Though the 100kb (unconfirmed, my phone isn't allowing me to see size) download is far from horrific, it's a bit inefficient to pull the whole list for a handful of apps.

As far as the "need to get paid in order to have the time to do it properly", does "properly" mean submitting a full featured PR to NC to implement occ app:install <appid>?

[enoch85]

submitting a full featured PR to NC to implement occ app:install ?

Yes.

[mdik]

Links to Nextcloud Appstore sources and API documentation (where my snippet is from as well):
https://github.com/nextcloud/appstore
https://nextcloudappstore.readthedocs.io/en/latest/

[r2evans]

Yeah, I looked through that before, but I have only found topics like "get all apps", "delete app", "publish app", etc. Nothing about "search within apps by keyword". The API as I understand it is predominantly for developers creating apps, not for finding specific/matching apps. (Perhaps the impetus to search for apps via API has never been explored or justified.) Thanks, I'll work with the whole list.

[BernhardPosselt]

Just to chime in: the API was built with simplicity and caching in mind. You can't properly cache a fully fledged keyword search. Another benefit apart from good caching is that in order to implement the API yourself you can simply drop 2-4 JSON files on a webserver and have it work like the actual store would.

However you are losing yourself in the details: first of all downloading and filtering the JSON using a semver lib is easy as pie and it makes absolutely 0 difference for you if the actual JSON file is 10kb or 1Mb since (yes I'm generalizing ;)).

Apart from that the actual JSON blob already exists on your system under data/appdata_something/appstore and you should be able to just update that copy so you can just go through that.

PS: No, I won't implement this feature for you :)

[r2evans]

I had not realized that the API was built for caching. The are two types of systems i had considered: cache the whole app list (such as apt-get); and, for much larger ecosystems of apps, query-as-you-go (such as Google Play or the Apple App Store). I jumped on the second, I should have considered the first. In that context, no API is required. Thanks!

To your point 2: nope, not lost in the details. There have never been concerns over parsing or computational complexity.

Your point 3: interesting point, but I would rather not make the assumption that the version stored in the VM image is the most recent. It is possible, even likely, that newer versions of apps exist, and just as likely that the previous versions in the old cache would no longer be available (e.g., significant security fixes). When dealing with images, it's quite possible that the image could be months old, and using a cache that old is certainly problematic. Even if NC automatically updates it periodically (does it?), that update will not happen on image instantiation before app-installation would occur.

Point 4: were you referring to not implementing "searchable API"? (In which case I agree, it was the intent I didn't know.) Or were you confirming that NC either has not prioritized or does not agree with this use-case? (Not being snarky, I know there are a gazillion bugs/features/PRs to go through ... though I think my use-case is pretty solid :-)

[enoch85]

Regarding what I shared from the VM, this is how I get the latest version at all times: curl -s https://api.github.com/repos/nextcloud/calendar/releases/latest | grep "tag_name" | cut -d\" -f4 | sed -e "s|v||g" Using Github API as you can see.

All the version stuff is in this file: https://github.com/nextcloud/vm/blob/master/lib.sh

Nextcloud version is done differently;
NCREPO="https://download.nextcloud.com/server/releases"
curl -s -m 900 $NCREPO/ | sed --silent 's/.*href="nextcloud-\([^"]\+\).zip.asc".*/\1/p' | sort --version-sort | tail -1

[BernhardPosselt]

Your point 3: interesting point, but I would rather not make the assumption that the version stored in the VM image is the most recent. It is possible, even likely, that newer versions of apps exist, and just as likely that the previous versions in the old cache would no longer be available (e.g., significant security fixes). When dealing with images, it's quite possible that the image could be months old, and using a cache that old is certainly problematic. Even if NC automatically updates it periodically (does it?), that update will not happen on image instantiation before app-installation would occur.

Your install command (if implemented) should ask if the cached local version is still the latest version (etags are supported). You need internet access anyways to download the files. It's likely already implemented, you need to read through the code.

Point 4: were you referring to not implementing "searchable API"? (In which case I agree, it was the intent I didn't know.) Or were you confirming that NC either has not prioritized or does not agree with this use-case? (Not being snarky, I know there are a gazillion bugs/features/PRs to go through ... though I think my use-case is pretty solid :-)

You download the json and search it locally. Or you create a service that downloads the json and makes a search available over REST. Your choice :)

[enoch85]

@MariusBluem Also, use case would be for me in the Nextcloud VM, installing apps via bash scripts.

[sualko]

I added occ app:install in #5644. Please review and test the pr.

[denics]

Thanks for this, very useful! what about installing apps from a git repository?

[BernhardPosselt]

@denics not possible since git repo does not contain the shipped code but requires built tools in most cases