Streetcomplete: Framapic shutdown for undefined amount of time

[ENT8R]

Framapic (https://framapic.org) shows the following image on every page you visit:

Right now it is not possible at all to access already uploaded pictures and a user told me in a note that the upload of new images does not work too...
Should we switch to yet another provider (would be the third if I remember right) or do we have to think about a completely different solution?

[rugk]

:cry:

No, no, no…

OSM should really implement this.

In the meantime, I would suggest:

• Make that whole thing as modular as possible. (maybe so you can also access different APIs/…)
• Add two or three providers, so the app automatically falls back, if one is not available.

This is the fail-safest option, I could think of.

[westnordost]

Sigh, own solution then. There seems to be no reliability on external services.

A simple upload script which also connects each photo with a note (id) and a cron(like) script which deletes all photos attached to closed/redacted notes or starts deleting the oldest if a predefined max size limit is reached.
Probably something needs to be done to make the cron not run too long

Am 18. August 2018 14:17:47 MESZ schrieb ENT8R [e-mail]:

Framapic (https://framapic.org) shows the following image on every page
you visit:

Right now it is not possible at all to access already uploaded pictures
and a user told me in a
note that the upload of
new images does not work too...
Should we switch to yet another provider (would be the third if I
remember right) or do we have to think about a completely different
solution?

[rugk]

Just be aware that you need to deal with all those trouble, in case you are self-hosting these images:

• you need to verify only pictures are uploaded (otherwise XSS attacks or so may be possible – depending on what else runs on the domain), or use a separate domain or so
• you need to deal with copyright claims/whatever illegal photos or so that may be uploaded
• you need to have all this disk space (do we even have an estimation of what is expected?)
• …

[deuzeffe]

Regardless the suspension of Framapic service, instead of delete pictures, may be you should keep them in dir de.westnordost.streetcomplete/files/Pictures or, at least, move them in the common dir. Pictures/Streetcomplete after synchro.

[rugk]

@deuzeffe I guess you are talking about the client, i.e. the app. In such a case, please open a new issue, explain your use case and why you think this is useful. Thanks.

[deuzeffe]

@rugk You're right, of course!

[deuzeffe]

As Framapic was developed during the de-google-ify Internet campaign from Framasoft, and free software based (Lutim : https://framagit.org/luc/lutim), may be you (or other volunteers) could install "somewhere on the osm network" you own instance of Framapic-like service. Just an idea.

[westnordost]

you need to deal with copyright claims/whatever illegal photos or so that may be uploaded

Users can only upload pictures by taking a photo, how can there be a copyright on that?

I imagine it like this:

1. upload picture(s) to westnordost. Call returns URL of where to find the picture once it is activated. The uploaded picture is saved in a tmp directory which is not readable
2. post the note to OSM -> get note id
3. post the note id to westnordost. The script will query the note from OSM, parse all linked picture URLs (only from non-anonymous users) and move the pics from tmp to their normal directory

This is to ensure that the service is bound to OSM notes and cannot be misused easily as some kind of photo storage.

[rugk]

Users can only upload pictures by taking a photo

You cannot technically limit it to this. Even if you use an API key only in the app, this can be reverse-engineered.

But using this and your proposed note "verification" algorithm sound liee a great idea.

[exploide]

Neat idea! So the only way to abuse this is to place a publicly visible note at OSM containing the link to it. And when this is abusive content, the note will pretty quickly be closed, leading to the removal of the picture short after...

I cannot guarantee that I find time to implement this, but whoever might start with this may be interested to have the following points cleared beforehand (I take a few assumptions based on the development of the oneway-api):

• you want it to be easily deployable without dependencies, favorably in PHP ( irks :S )
  
  
  
  • i.e. it is not known if something like a python3 CGI script with a MySQL connector is runnable
  
  
• you don't configure document roots, e.g. need the served scripts in the top directory
• you are using Apache, so at least restricting access via .htaccess works
• association between picture and note ID should be stored in a MySQL database

Would be good if you could comment on these points and clarify if my assumptions are accurate or if something is different.

[westnordost]

PHP is not favored at all, it just runs anywhere and everybody seems to be able to code in PHP. I think Python would work, Perl works, also Ruby. But easily deployable holds.

[rugk]

the note will pretty quickly be closed, leading to the removal of the picture short after...

Maybe not do so directly after closing, as a note may also be reopened, due to accidental closing etc.

[westnordost]

True, but moderated notes can be distinguished from closed notes.

[exploide]

Yes, the thing with PHP is it runs everywhere but makes no fun to program (at least for me). The only problem with Python is, that it can be hard to deploy and has no MySQL support built-in. Depends. If you would like to find out if Python has a chance in your scenario, try to deploy and run the following Python3 CGI script and see if it tells you its version and some info about available MySQL connectors.

#!/usr/bin/env python3
import sys
print("Content-Type: text/plain")
print()
print(sys.version)
try:
    import PyMySQL
except:
    print("no PyMySQL")
try:
    import MySQLdb
except:
    print("no MySQLdb")
try:
    import mysql.connector
except:
    print("no mysql.connector")
try:
    import sqlalchemy
except:
    print("no sqlalchemy")

If this won't work or no connector is available, then PHP it is :D

[westnordost]

Python seems to be not installed. I can ask my webhoster if it would be possible to install/enable it. Still, there is Ruby and Perl. At least the former is pretty, isn't it? (Never used it)

[rugk]

Come on, let's not argue what language is better. They all have their advantages and disavantages, and you can choose what suit's the use case best.
And you can write good and bad code with any language.

[westnordost]

Okay, Python is definitely not available, I asked the hoster

[exploide]

Yes, I think Ruby is pretty, but I can't write in Ruby :D

However, then PHP is the simplest and most appropriate choice I think.

I started implementing the photo service. Don't expect it to be finished within the next few days. Hope that's fine.

[westnordost]

(v8 is a wish, if it is not ready, it is not ready, but it would be cool. It won't be due until _at least_ 2 weeks though anyway)

[exploide]

I almost finished the upload script. Activate and cleanup script to go.

I think v8 is a reasonable target.

I'll be back ^^

[ENT8R]

I created some statistics so that you know how much load you might expect:

Lutim & Framapic

| Lutim | Framapic |
| ------------- |-------------|
| | |

[westnordost]

There were only 500 photos in total? That's about 100MB only!

[ENT8R]

No, the plots show the amount of photos per month. So in total there were more than 2000 photos

[westnordost]

Ah okay. Well, I got several GB, so it seems that there will be no problems.

[Quix0r]

Or switch to @Nextcloud ? The APIs there are easy to implement and there are already libraries for many common and uncommon languages.

[exploide]

I'm back ^^

First version of the photo upload service is done so far. Get it here:
https://github.com/exploide/sc-photo-service

Take a look, give it a try, let me know what you think. If the one or other person does a review, that would be appreciated :)

Things I deliberately omitted for now:

• upload size limit, because maybe you are just fine with max upload file size from global php.ini
• overall data size limit because it sounds like the expected usage might be acceptable

But if you want one or both things to be implemented, I can take a look.

(@Quix0r I guess that would not solve the abuse problem, since everyone could upload to that Nextcloud instance. Something like an API key could be extracted from SC apk)

[Quix0r]

@exploide yes, I later came to same thought. :-/

[Quix0r]

If OSM could provide oauth2, then you can authenticate OSM users easily. At least not everbody can upload.

[Quix0r]

Oh, they do have OpenID (for users) ...

[rugk]

If OSM could provide oauth2, then you can authenticate OSM users easily.

And they do have OAuth, at least: https://wiki.openstreetmap.org/wiki/OAuth

[westnordost]

By the way, this will be the last thing I'll do for v8, if someone would like to write the client-side code to talk with exploide's API, feel free to. Though, I plan to release the first beta of v8 in about a week or two.

[westnordost]

I think I am going to postpone this to v9. There is no reason to delay v8 when the feature I am currently working on is complete. But when v8 is out quicker, v9 will be as well, so...