Photos: Object detection

[pixelyo]

I would love to see object detection added to photos. It could add tags to a photo and make all photos searchable.

[skjnldsv]

Hi! There is a way for it but it's a bit tricky :)
It would be super nice to have this integrated as an app
https://nayarweb.com/blog/2019/automated-tagging-of-images-on-nextcloud-with-tensorflow-for-poets-2/

[marcelklehr]

A simpler approach might be to have admins set up tensorflow serving with a pre-trained resnet model and build a simple app that shoots prediction requests at the HTTP endpoint in a background task.

[skjnldsv]

@marcelklehr possible in pure php?

[marcelklehr]

That would be awesome, of course. There's php-ml, but it doesn't allow loading pre-trained models, so every instance would have to train their own classifier, which is not that cool, IMO. Ideally there'd be a library that can load tensorflow models for prediction in php.

[da3dsoul]

This is cool, but would need to come with a huge red flag warning. Nextcloud screams its views on security from atop the tallest mountains of the internet, and sending user data to a service for machine learning systems violates that hard. If you have the whole thing locally processed, and the trained set only uses encrypted, unreversible data, then sure, but it seems unlikely as far as I am familiar with the concept.

If I'm totally wrong here, then tell me, as I'm no expert, but it seemed like something that at least one person would worry about.

[marcelklehr]

sending user data to a service for machine learning systems violates that hard

The goal would be to never send user data to a third-party for classification, but have classification take place on the same server as nextcloud.

[marcelklehr]

Just found rubixml.com, which might be interesting.

[sattitan19]

looking forward for this feature

[andrewdalpino]

Just found rubixml.com, which might be interesting.

We're currently working on GPU acceleration in Rubix ML with the Tensor extension so this feature might be practical to implement in PHP soon with a convnet.

Rubix is currently being used in the Mail app for spam filtering

https://github.com/nextcloud/mail/blob/master/lib/Service/Classification/ImportanceClassifier.php

Feel free to join our Telegram channel here and discuss implementation https://t.me/RubixML

[rekcodocker]

I had this feature on my NAS. It took a week to index 10000 photos, but it works when I am asleep. The results are absolutely awesome.
Just saying that (1) local processing is possible and (2) processing time is not an issue for me.

[marcelklehr]

@andrewdalpino Being able to efficiently run a convolutional net would certainly be a big step. Another step would be the ability to load models trained with other frameworks, which makes it far easier to use existing pre-trained image classification models, instead of having to download, manage and train an entire image dataset by hand.

[marcelklehr]

Perhaps an alternative to doing ML directly in PHP would be to create an app that exposes an API for tensorflow serving, which runs tensorflow models in a docker container. Other apps could then interface with that API to install and use trained tensorflow models. This would only require installing a docker container, treating ML akin to a database or redis.

@ChristophWurst What do you think about this?

[ChristophWurst]

Calling into external applications is very well possible, it just comes with the downside that this doesn't work out of the box, needs configuration and has the danger of potential version mismatches as you can't enforce a certain version of TF from your Nextcloud app.

[marcelklehr]

Alright, so I'm currently experimenting with building an app that's calling out from PHP to MobileNet in tensorflow in node.js, which seems to work. The question is what to do with the labels then. Should system tags be used for this?

[marcelklehr]

I've published a working POC of an image classfier app on github: https://github.com/marcelklehr/recognize

The caveat is, you can't train it, but for normal users, this should be a big improvement.

Feedback and contributions welcome! :)