Vm: Script Full Text Search

[enoch85]

Script installation of Full Text Search as requested here: https://github.com/daita/fullnextsearch/issues/1

https://github.com/nextcloud/fulltextsearch/wiki/Basic-Installation

UPDATE
Script is here: https://github.com/nextcloud/vm/blob/improvments/apps/fulltextsearch.sh

[enoch85]

Install Elastic Search:

check_command apt-get install openjdk-8-jre
check_command curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.1.2.deb
check_command dpkg -i elasticsearch-6.1.2.deb
check_command /etc/init.d/elasticsearch start

# Install plugin
if [ -d /usr/share/elasticsearch ]
then
    cd /usr/share/elasticsearch
    check_command bin/elasticsearch-plugin install ingest-attachment
fi

occ_command "config:app:set --value '1' fullnextsearch app_navigation"
occ_command "fulltextsearch:index"

Install part Elastic Search servlet

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.**x**.list
sudo apt-get update && sudo apt-get install elasticsearch

[enoch85]

https://gist.github.com/ondrej-kvasnovsky/9363975 (Install steps)

Basic installation: https://github.com/nextcloud/fulltextsearch/wiki/Basic-Installation

ReadonlyREST https://github.com/beshu-tech/readonlyrest-docs/blob/master/elasticsearch.md

https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html

https://www.elastic.co/guide/en/elasticsearch/plugins/current/plugin-management-custom-url.html

[enoch85]

@Ark74 Would you mind do some researching, or maybe even try to make a script our of this? I'm happy to assist.

[Ark74]

I've been wanting to work this out, but haven't got the time yet.
I was thinking about starting on removing Solr so old instances could implement this new engine too.
Next week i'll make time to take a look on this.
Cheers!

[EmilyLove26]

Can't wait!

Also, make sure you make a script to uninstall nextant & solr. I already know how, but other end users may not know how.

[enoch85]

I've started a PR here: https://github.com/nextcloud/vm/pull/446

[Ark74]

Wow! Looks like you nailed. Great work.

I'll help on the testing then.
Cheers!

[enoch85]

@Ark74 There are still a few things left until perfection. We need to secure it with SSL, and also check the SHA against the package that is in ../fulltextsearch-files. We also need a proper YML file file passwords and username that needs to be put in the GUI automatically: https://github.com/nextcloud/vm/pull/446#issuecomment-359592078

Can you do the finishing touches?

[enoch85]

@Ark74 Can you please test this and add the finishing details before 13.0.0 is released? Would be awesome! It's live in this branch: https://github.com/nextcloud/vm/blob/nc13-appinstall/apps/fulltextsearch.sh

[Ark74]

Yeah, i'll be taking a look at it today.
I'll be posting after lunch :+1:

[Ark74]

How do you get NC 13, since I see you use occ app:install wich is not supported on 12?
With the 12.0.5 update I have no option to upgrade to 13 anymore. :-/

Edit: Never mind: https://github.com/nextcloud/server/releases :laughing:

[enoch85]

How did it go? Did you manage to fix the remaining issues?

[enoch85]

Just upgrade to the latest RC according to the README, then download this script manually and run it: https://github.com/nextcloud/vm/blob/nc13-appinstall/apps/fulltextsearch.sh

[enoch85]

@Ark74 Ping 127.0.0.1 :)

[enoch85]

@daita Maybe you can help out with the finish details? https://github.com/nextcloud/vm/blob/master/apps/fulltextsearch.sh

[daita]

What do you need ?

[enoch85]

I want to pre-generate user and password and fill in everything in the GUI with occ so that the user gets a complete setup, ready to use.

https://github.com/nextcloud/vm/blob/master/apps/fulltextsearch.sh#L110
https://github.com/nextcloud/vm/blob/master/apps/fulltextsearch.sh#L129

[daita]

a command will come with the next release (0.6.0) of Fts:

./occ fulltextsearch:check --json

that return the current setup of the app.
I will make an equivalent command to write the setup from the CLI.

Edit: The configuration will be done by sending a JSON.

[enoch85]

@daita Sounds like a plan.

[daita]

https://github.com/nextcloud/fulltextsearch/commit/c48e818326aedd272b2aad6851d689e9f194eebf
https://github.com/nextcloud/fulltextsearch_elasticsearch/commit/2feccd9d70bb103f757b8d9e4ab69c6f1879d9a9
https://github.com/nextcloud/files_fulltextsearch/commit/720746dd51abe78e4222f32d10d7aa766ee42f03
https://github.com/nextcloud/bookmarks_fulltextsearch/commit/3abc0b27442a5ec267ed53625d5b0956dedcbef9

Full text search 0.6.x will introduce a command for each app:

 ./occ fulltextsearch:configure <json>
 ./occ fulltextsearch_elasticsearch:configure <json>
 ./occ files_fulltextsearch:configure <json>
 ./occ bookmarks_fulltextsearch:configure <json>

Working example:

maxence@stealth ~/sites/nextcloud/server $ ./occ files_fulltextsearch:configure {\"files_local\":\"0\"}
{
    "files_local": "0",
    "files_external": "0",
    "files_encrypted": "0",
    "files_federated": "0",
    "files_size": "20",
    "files_pdf": "0",
    "files_office": "1",
    "files_image": "0",
    "files_audio": "0"
}
maxence@stealth ~/sites/nextcloud/server $ ./occ files_fulltextsearch:configure {\"files_local\":\"1\"}
{
    "files_local": "1",
    "files_external": "0",
    "files_encrypted": "0",
    "files_federated": "0",
    "files_size": "20",
    "files_pdf": "0",
    "files_office": "1",
    "files_image": "0",
    "files_audio": "0"
}

Note that you can send a full JSON containing all parameters for your configuration.
Also, list of the current configuration can be obtained with:

maxence@stealth ~/sites/nextcloud/server $ ./occ fulltextsearch:check --json
{
    "fulltextsearch": {
        "version": "0.5.1",
        "config": {
            "search_platform": "OCA\\FullTextSearch_ElasticSearch\\Platform\\ElasticSearchPlatform",
            "index_chunk": "20",
            "app_navigation": "1",
            "provider_indexed": "{\"bookmarks\":\"1\",\"files\":\"1\"}"
        }
    },
    "platform": {
        "elastic_search": {
            "class": "OCA\\FullTextSearch_ElasticSearch\\Platform\\ElasticSearchPlatform",
            "version": "0.5.1",
            "config": {
                "elastic_host": "http:\/\/username:********@localhost:9200",
                "elastic_index": "my_index"
            }
        }
    },
    "providers": {
        "bookmarks": {
            "version": "0.2.0",
            "config": {
                "bookmarks_ttl": "5"
            }
        },
        "files": {
            "version": "0.5.0",
            "config": {
                "files_local": "1",
                "files_external": "0",
                "files_encrypted": "0",
                "files_federated": "0",
                "files_size": "20",
                "files_pdf": "0",
                "files_office": "1",
                "files_image": "0",
                "files_audio": "0"
            }
        }
    }
}

[enoch85]

@daita Thanks! Do you think you could add a minimal configuration in the scripts for Full Text Search to work as expected?

[daita]

You're welcome. However I think I set enough default values. The platform needs to be installed and selected by the end user (or by your script)

[enoch85]

@daita Could you please test the script, understand the logic and add what's missing for a complete setup? It's run in Ubuntu 16.04, and as a suggestion I think that you should test it with the Nextcloud VM, there you have a full environment.

In Nextant we set all the needed stuff in the GUI with occ commands, like the localhost address, port and so on. What I would want is to do the same in Full Text Search. The goal here is that the user runs the script and then the app is ready to use out of the box.

Right now it installs everything but doesn't set any values in the GUI.

[daita]

@enoch85 : i will add some test, this is on my roadmap, but not for 0.6.0.
With the version from github, you should be able to set everything using occ commands.

[enoch85]

@daita I saw that 0.6.0 was released, does this mean that you could implement this in the scripts?

[daita]

@enoch85 I don't think it is a good idea I start working on your scripts. I can however add some command (like the ./occ fulltextsearch:check --json and ./occ fulltextsearch:configure) but I will let you implement this into your installation script.

[enoch85]

@daita OK, please tell me what needs to be configured for FTS to work.

And yes, please add those commands so that FTS can be setup from CLI.

[daita]

This would do the trick:

./occ fulltextsearch:configure {\"search_platform\":\"OCA\\\\FullTextSearch_ElasticSearch\\\\Platform\\\\ElasticSearchPlatform\"}
./occ fulltextsearch_elasticsearch:configure {\"elastic_host\":\"http://username:password@localhost:9200\"\,\"elastic_index\":\"my_index\"}

[enoch85]

@daita Great! Thanks!

username:password, is that the username and password for the Nextcloud user, or is it something else?

[enoch85]

@daita Please check if https://github.com/nextcloud/vm/commit/32577a44fd2a8e40c44f072b069c10921ffbbb31 would be enough.

[daita]

It is the username and password in case of the need of authentication on the elasticsearch servlet.

I will check that later today

[enoch85]

I will check that later today

@daita Have you had the time to check yet?

[morph027]

Hi! ;)

https://github.com/nextcloud/vm/commit/32577a44fd2a8e40c44f072b069c10921ffbbb31 looks good to me.

When ingest plugin is installed, we might want to enable document scanning:

occ_command files_fulltextsearch:configure {\"files_pdf\":\"1\"\,\"files_office\":\"1\"}

[morph027]

Also, when installation is finished, we need to start first time index occ_command fulltextsearch:index

[daita]

@morph027 : in fact the ingest plugin is needed even for simple text files.

I think both enhancement from @morph027 are nice. Edit the configuration to add the pdf and office and then start the first index.

[enoch85]

OK, so I added @morph027's suggestions. Does that look OK to you @daita?

Also, what about the username and password?

[daita]

I though I already answered this, the username and password are used to ident people on the elasticsearch servlet using one of the plugins available. Useful if people can send request to the servlet or if people want to index multiple cloud on the same servlet.

It looks OK, but I haven't tested it

[morph027]

In the script i don't see that a username+password is set for the ES instance. Also, if i got the ReadOnlyREST settings right, access is restricted to localhost only, therefore you don't need username+password for the addon.

Maybe the script should restrict ES to localhost in general:

sed -i 's,^network.host: 0.0.0.0,network.host: 127.0.0.1,' /etc/elasticsearch/elasticsearch.yml

[enoch85]

@daita

It is the username and password in case of the need of authentication on the elasticsearch servlet.
I will check that later today

So I thought you would check, that's why I asked. :)

@morph027
Can you please add your suggestions to the improvements branch? I want to merge this before 13.0.1 is released.

[morph027]

Ah, nevermind, just had a quick look into a fresh Elastic installation, already is set to localhost only. So nothing to change there.

[enoch85]

@morph027 Could you please try the script from the improvements branch and see if it works? I'm still not sure if I should remove username:password, or if we have to generate one?

[Ark74]

Redis is also for localhost usage only, but for some reason the install script set password for it too.
Maybe if we follow that path.
Also, one of new features of this release over nextant is multiple NC instances over the ES server, being on the same server than NC or even a dedicated server for ES.

If try to catch up with you guys, but so far i'm still looking at https://github.com/nextcloud/vm/issues/473

[enoch85]

@Ark74 Security even if it's not needed. At least it looks good ;)

No, but with Redis I just followed the security advice. If we can use a password then why not. It's just like a few lines of code and all the variables are already in place. But if it's reeeeaallly not needed, then skip it. :)

[Ark74]

I believe it's exactly the same thing.
If we can use username and password to increase the security why not?
Since one ES (like redis) can be used for different servers.

Edit: On a second thought, maybe we can skip it 'til becomes a feature request, and the request(er) is willing to invest some time/resources on it.

[morph027]

I've tried and it works. But i agree, if we can simply add security layer, let's do it.

But if we want to expose ES to the network, we should do it proper. I'm not sure if we should use the ReadonlyREST thing, we are already having Apache on the box (do we?), so we proxy ES and can re-use certificates for TLS, simply add .htpasswd access and so on.

[enoch85]

@morph027 Sounds like a great idea.

I'm in the middle of testing the improvements branch now, noticed I needed some changes. DId you test master?

[morph027]

Nope, improvement branch. Should i add some apache configuration to ES? Or do you want to merge without password and localhost first?

[enoch85]

@morph027 Do whatever you feel is best. I trust you. :)

I want to merge this when it's complete. I'm planning to merge the improvements branch when I release 13.0.1 which is pretty soon I think. So yes, add passwords and everything at once.

Would love to change all occ commands to occ_command as well, that's on my whishlist.

[daita]

2 cents: If I were you, I would add the readonlyREST plugin to elasticsearch.

[enoch85]

@daita Please check the script. It's already added here: https://github.com/nextcloud/vm/blob/improvments/apps/fulltextsearch.sh#L93L107

[daita]

ok, do you set a password into /etc/elasticsearch/readonlyrest.yml ?

[enoch85]

@daita No, that's why I asked about the (username:)passwords before. I'm totally new to this and would appreciate your guidance in how to configure everything so that the VM users can benefit from your apps.

[daita]

Have a look here about the file you need to create in /etc/elasticsearch/. It contains the name of the index and credentials. You use those information during the setup of fulltextsearch_elasticsearch

[enoch85]

Added a $VAR for the password. Will try to fix a YML as well.

[enoch85]

Added the YML again.

[enoch85]

It's implemented now, but needs to be improved. I'll open another issue for that.