Apps-android-commons: Avoid "Commons link should be well-formed" warning

Created on 2 Dec 2019  路  30Comments  路  Source: commons-app/apps-android-commons

When uploading a picture via Nearby, the Wikidata item's image field is filled via a sanitized version of the picture's title (notice the underscores):
Screenshot from 2019-12-02 20-12-11

It works, but actually, Wikidata prefers the non-sanitized version, so it complains:

This statement has some issues.
Commons link constraint.
Commons link should be well-formed.

Screenshot from 2019-12-02 20-19-51

This can be fixed by editing and writing the title instead, like this (notice the lack of underscores):
Screenshot from 2019-12-02 20-12-52

Which solves the problem:
Screenshot from 2019-12-02 20-13-10

Your task:

  • Find where in the code the Wikidata item's image value is set.
  • Set it to the real image title. I am pretty sure we already have that title, and we don't need dirty tricks (such as replacing underscore with spaces... there might be other differences).
assigned enhancement good first issue
Source
picture nicolas-raoul

Most helpful comment

@gouri-panda seeing as how it has been 2 months with no activity I say go for it, try and replicate the issue first because I don't know if it exists

picture macgills on 17 Mar 2020
👍2

All 30 comments

Hi @nicolas-raoul, I would like to work on this.

harshika-kashyap picture harshika-kashyap on 18 Jan 2020

@harshika-arya Yes, sure. The issue is yours. Feel free to ping here if you get stuck anywhere. :)

maskaravivek picture maskaravivek on 18 Jan 2020

@maskaravivek Is this issue still open? Can I work on this issue?

gouri-panda picture gouri-panda on 17 Mar 2020

@gouri-panda seeing as how it has been 2 months with no activity I say go for it, try and replicate the issue first because I don't know if it exists

macgills picture macgills on 17 Mar 2020
👍2

@macgills After Debugging and dig into source code I found that it happens on the uploadContribution method in the UplaodService class. In the uploadContribution method after uploadResult success, it returns the file name with underscores but I can't find where it sets as the string value as @nicolas-raoul said.
EDIT - I also checked The URL from the log but it sets the file name without underscore.

gouri-panda picture gouri-panda on 18 Mar 2020 
.subscribe(uploadResult -> {
                    Timber.d("Stash upload response 2 is %s", uploadResult.toString());

                    notificationManager.cancel(notificationTag, NOTIFICATION_UPLOAD_IN_PROGRESS);

                    String resultStatus = uploadResult.getResult();
                    if (!resultStatus.equals("Success")) {
                        Timber.d("Contribution upload failed. Wikidata entity won't be edited");
                        showFailedNotification(contribution);
                    } else {
                        String canonicalFilename = "File:" + uploadResult.getFilename();
                        Timber.d("Contribution upload success. Initiating Wikidata edit for entity id %s",
                                contribution.getWikiDataEntityId());
                        wikidataEditService.createClaimWithLogging(contribution.getWikiDataEntityId(), canonicalFilename);
                        contribution.setFilename(canonicalFilename);
                        contribution.setImageUrl(uploadResult.getImageinfo().getOriginalUrl());
                        contribution.setState(Contribution.STATE_COMPLETED);
                        contribution.setDateUploaded(CommonsDateUtil.getIso8601DateFormatShort()
                                .parse(uploadResult.getImageinfo().getTimestamp()));
                        compositeDisposable.add(contributionDao
                                .save(contribution)
                                .subscribeOn(ioThreadScheduler)
                                .observeOn(mainThreadScheduler)
                                .subscribe());
                    }
                }

This is what you are talking about?

Try debugging UploadClient.uploadFileFromStash I think, we need to worry about the data that gets uploaded so you can ignore the Contribution object getting set with details to be saved to the DB.

I am afraid I have no idea right now how anything gets uploaded and what fields are what. The 17 fields of Contribution are very unfriendly.

macgills picture macgills on 18 Mar 2020
👍1

@macgills I think when we upload the image we sent the correct image title formate(i.e without underscores) then after image uploaded successfully the Database sends the UploadResult. In the Upload result, the image title contains underscores and then the title stores Local Database.

contribution.setFilename(canonicalFilename);
                        contribution.setImageUrl(uploadResult.getImageinfo().getOriginalUrl());
                        contribution.setState(Contribution.STATE_COMPLETED);
                        contribution.setDateUploaded(CommonsDateUtil.getIso8601DateFormatShort()
                                .parse(uploadResult.getImageinfo().getTimestamp()));
                        contributionDao.save(contribution);
gouri-panda picture gouri-panda on 19 Mar 2020

We can't be sending the correct thing, Nicolas' screenshots are from the website not the app so for this ticket any LocalDatabase should be completely irrelevant but I think the contribution database is what is used to queue uploads but here it has state contribution.setState(Contribution.STATE_COMPLETED); so this is probably the last operation?

@ashishkumar468 may know more than me, I think there are a lot of endpoints involved

macgills picture macgills on 19 Mar 2020
👍1

AFAIR, we intentionally do nothing of this sort. But, I do see this happening, but at least to me its looks like we are sending the correct format file name, but this api seems to return underscored file name even though we pass the file name in the non-underscored format.
Attaching logs from my observation
200 https://commons.wikimedia.org/w/api.php?format=json&formatversion=2&errorformat=plaintext&action=upload&stash=1&ignorewarnings=1 (21987ms, unknown-length body) 2020-03-19 18:06:26.312 10702-16158/fr.free.nrw.commons D/UploadService: Stash upload response 1 is fr.free.nrw.commons.upload.UploadResult@6b3515a 2020-03-19 18:06:31.230 10702-16158/fr.free.nrw.commons D/UploadService: making sure of uniqueness of name: Manarola in Italy.jpg 2020-03-19 18:06:51.576 10702-16158/fr.free.nrw.commons D/OkHttp: --> GET https://commons.wikimedia.org/w/api.php?action=query&format=json&formatversion=2&titles=File%3AManarola%20in%20Italy.jpg 2020-03-19 18:06:51.645 10702-16158/fr.free.nrw.commons D/CookieManager: Domain:commons.wikimedia.org 2020-03-19 18:06:52.520 10702-16158/fr.free.nrw.commons D/OkHttp: <-- 200 https://commons.wikimedia.org/w/api.php?action=query&format=json&formatversion=2&titles=File%3AManarola%20in%20Italy.jpg (942ms, unknown-length body) 2020-03-19 18:07:07.172 10702-16158/fr.free.nrw.commons D/OkHttp: --> GET https://commons.wikimedia.org/w/api.php?format=json&formatversion=2&errorformat=plaintext&action=query&meta=tokens&type=csrf 2020-03-19 18:07:07.186 10702-16158/fr.free.nrw.commons D/CookieManager: Domain:commons.wikimedia.org 2020-03-19 18:07:07.703 10702-16158/fr.free.nrw.commons D/OkHttp: <-- 200 https://commons.wikimedia.org/w/api.php?format=json&formatversion=2&errorformat=plaintext&action=query&meta=tokens&type=csrf (529ms, unknown-length body) 2020-03-19 18:07:07.764 10702-16158/fr.free.nrw.commons D/OkHttp: --> POST https://commons.wikimedia.org/w/api.php?format=json&formatversion=2&errorformat=plaintext&action=upload&ignorewarnings=1 (838-byte body) 2020-03-19 18:07:07.771 10702-16158/fr.free.nrw.commons D/CookieManager: Domain:commons.wikimedia.org 2020-03-19 18:07:10.470 10702-16158/fr.free.nrw.commons D/OkHttp: <-- 200 https://commons.wikimedia.org/w/api.php?format=json&formatversion=2&errorformat=plaintext&action=upload&ignorewarnings=1 (2705ms, unknown-length body) 2020-03-19 18:07:13.042 10702-16158/fr.free.nrw.commons D/UploadService: Stash upload response 2 is fr.free.nrw.commons.upload.UploadResult@5e3217b 2020-03-19 18:09:00.003 10702-10712/? W/art: Debugger attempted to resume all threads without having suspended them all before. 2020-03-19 18:09:00.008 10702-16158/? D/UploadService: Contribution upload success. Initiating Wikidata edit for entity id null 2020-03-19 18:09:00.008 10702-10712/? I/art: Debugger is no longer active 2020-03-19 18:09:00.009 10702-10712/? I/art: Starting a blocking GC Instrumentation 2020-03-19 18:09:00.014 10702-10712/? W/art: Suspending all threads took: 5.221ms 2020-03-19 18:09:00.015 10702-16158/? D/WikidataEditService: Skipping creation of claim as Wikidata entity ID is null 2020-03-19 18:09:00.142 10702-10702/? D/ContributionsFragment: Fetching thumbnail for File:Manarola_in_Italy.jpg

It would be wrong to blame the api's without proper debugging though.

ashishkumar468 picture ashishkumar468 on 19 Mar 2020
👍1

https://commons.wikimedia.org/w/api.php?action=help&modules=upload lets keep this handy

macgills picture macgills on 19 Mar 2020
👍1

We first upload the picture to Commons, _and then_ inject into Wikidata a reference to that newly created picture.

Commons transforms ":" into "-", and possibly other sanitations. So we need to rely on the name that Commons gives us back.

Example:

  1. We upload "Director at premi猫re of Batman:Reborn" using the Commons API
  2. The Commons API returns to me a response saying: OK, I created your file at "Director_at_premi猫re_of_Batman-Reborn" (notice the spaces replaced with underscores and the colon replaced with a dash)
  3. We call the Wikidata API to set the image item of the Batman:Reborn item to "Director_at_premi猫re_of_Batman-Reborn"
  4. Wikidata understands, but tells us that we should have written "Director at premi猫re of Batman-Reborn" instead (notice the colon replaced with a dash).

So, the next step is to analyze the the response we get at step 2 and see whether it contains what Wikidata wants at step 4.

nicolas-raoul picture nicolas-raoul on 19 Mar 2020
👍1

@nicolas-raoul Are you saying we should check the response we get from API should be equal to the image title we uploaded with underscores.

gouri-panda picture gouri-panda on 19 Mar 2020

@gouri-panda Please take a picture of any common object in your house, upload it via the app with a name containing spaces and ":", while recording adb logcat. Then please post the Commons API response here. Thanks! :-)

nicolas-raoul picture nicolas-raoul on 19 Mar 2020

Image title = Beautiful house :

Logs

020-03-19 19:48:07.412 1001-1076/fr.free.nrw.commons.beta D/NearbyPlaces: 7 results at radius: 2.617924
2020-03-19 19:48:07.415 1001-1076/fr.free.nrw.commons.beta V/OkHttp: --> GET https://query.wikidata.org/sparql?query=SELECT%0A%20%20%20%20%20%28SAMPLE%28%3Flocation%29%20as%20%3Flocation%29%0A%20%20%20%20%20%3Fitem%0A%20%20%20%20%20%28SAMPLE%28COALESCE%28%3FitemLabelPreferredLanguage%2C%20%3FitemLabelAnyLanguage%29%29%20as%20%3Flabel%29%0A%20%20%20%20%20%28SAMPLE%28%3FclassId%29%20as%20%3Fclass%29%0A%20%20%20%20%20%28SAMPLE%28COALESCE%28%3FclassLabelPreferredLanguage%2C%20%3FclassLabelAnyLanguage%2C%20%22%3F%22%29%29%20as%20%3FclassLabel%29%0A%20%20%20%20%20%28SAMPLE%28COALESCE%28%3Ficon0%2C%20%3Ficon1%29%29%20as%20%3Ficon%29%0A%20%20%20%20%20%3FwikipediaArticle%0A%20%20%20%20%20%3FcommonsArticle%0A%20%20%20%20%20%28SAMPLE%28%3FcommonsCategory%29%20as%20%3FcommonsCategory%29%0A%20%20%20%20%20%28SAMPLE%28%3Fpic%29%20as%20%3Fpic%29%0A%20%20%20%20%20%28SAMPLE%28%3Fdestroyed%29%20as%20%3Fdestroyed%29%0A%20%20%20WHERE%20%7B%0A%20%20%20%20%20%23%20Around%20given%20location...%0A%20%20%20%20%20SERVICE%20wikibase%3Aaround%20%7B%0A%20%20%20%20%20%20%20%3Fitem%20wdt%3AP625%20%3Flocation.%0A%20%20%20%20%20%20%20bd%3AserviceParam%20wikibase%3Acenter%20%22Point%28-121.4145%2039.2011%29%22%5E%5Egeo%3AwktLiteral.%0A%20%20%20%20%20%20%20bd%3AserviceParam%20wikibase%3Aradius%20%224.24%22%20.%20%23%20Radius%20in%20kilometers.%0A%20%20%20%20%20%7D%0A%0A%20%20%20%20%20%23%20Get%20the%20label%20in%20the%20preferred%20language%20of%20the%20user%2C%20or%20any%20other%20language%20if%20no%20label%20is%20available%20in%20that%20language.%0A%20%20%20%20%20OPTIONAL%20%7B%3Fitem%20rdfs%3Alabel%20%3FitemLabelPreferredLanguage.%20FILTER%20%28lang%28%3FitemLabelPreferredLanguage%29%20%3D%20%22en%22%29%7D%0A%20%20%20%20%20OPTIONAL%20%7B%3Fitem%20rdfs%3Alabel%20%3FitemLabelAnyLanguage%7D%0A%0A%20%20%20%20%20%23%20Get%20Commons%20category%20%28P373%29%0A%20%20%20%20%20OPTIONAL%20%7B%20%3Fitem%20wdt%3AP373%20%3FcommonsCategory.%20%7D%0A%0A%20%20%20%20%20%23%20Get%20%28P18%29%0A%20%20%20%20%20OPTIONAL%20%7B%20%3Fitem%20wdt%3AP18%20%3Fpic.%20%7D%0A%0A%20%20%20%20%20%23%20Get%20%28P576%29%0A%20%20%20%20%20OPTIONAL%20%7B%20%3Fitem%20wdt%3AP576%20%3Fdestroyed.%20%7D%0A%0A%20%20%20%20%20%23%20Get%20the%20class%20label%20in%20the%20preferred%20language%20of%20the%20user%2C%20or%20any%20other%20language%20if%20no%20label%20is%20available%20in%20that%20language.%0A%20%20%20%20%20OPTIONAL%20%7B%0A%20%20%20%20%20%20%20%3Fitem%20p%3AP31%2Fps%3AP31%20%3FclassId.%0A%20%20%20%20%20%20%20OPTIONAL%20%7B%3FclassId%20rdfs%3Alabel%20%3FclassLabelPreferredLanguage.%20FILTER%20%28lang%28%3FclassLabelPreferredLanguage%29%20%3D%20%22en%22%29%7D%0A%20%20%20%20%20%20%20OPTIONAL%20%7B%3FclassId%20rdfs%3Alabel%20%3FclassLabelAnyLanguage%7D%0A%0A%20%20%20%20%20%20%20OPTIONAL%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%3FwikipediaArticle%20%20%20schema%3Aabout%20%3Fitem%20%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20schema%3AisPartOf%20%3Chttps%3A%2F%2Fen.wikipedia.org%2F%3E%20.%0A%20%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20OPTIONAL%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%3FwikipediaArticle%20%20%20schema%3Aabout%20%3Fitem%20%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20schema%3AisPartOf%20%3Chttps%3A%2F%2Fen.wikipedia.org%2F%3E%20.%0A%20%20%20%20%20%20%20%20%20%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22en%22%20%7D%0A%20%20%20%20%20%20%20%20%20%7D%0A%0A%20%20%20%20%20%20%20%20%20OPTIONAL%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%3FcommonsArticle%20%20%20schema%3Aabout%20%3Fitem%20%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20schema%3AisPartOf%20%3Chttps%3A%2F%2Fcommons.wikimedia.org%2F%3E%20.%0A%20%20%20%20%20%20%20%20%20%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22en%22%20%7D%0A%20%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%7D%0A%20%20%20%7D%0A%20%20%20GROUP%20BY%20%3Fitem%20%3FwikipediaArticle%20%3Fcommon
2020-03-19 19:48:07.416 1001-1076/fr.free.nrw.commons.beta V/OkHttp: sArticle%0A&format=json
2020-03-19 19:48:07.416 1001-1076/fr.free.nrw.commons.beta V/OkHttp: --> END GET
2020-03-19 19:48:08.513 1001-1059/fr.free.nrw.commons.beta I/rw.commons.bet: Explicit concurrent copying GC freed 341937(18MB) AllocSpace objects, 15(324KB) LOS objects, 49% free, 6079KB/11MB, paused 668us total 51.390ms
2020-03-19 19:48:08.630 1001-1071/fr.free.nrw.commons.beta D/OkHttp: <-- 200 https://commons.wikimedia.beta.wmflabs.org/w/api.php?format=json&formatversion=2&errorformat=plaintext&action=upload&ignorewarnings=1 (2190ms, unknown-length body)
2020-03-19 19:48:08.664 1001-1071/fr.free.nrw.commons.beta D/UploadService: Stash upload response 2 is fr.free.nrw.commons.upload.UploadResult@56c538
2020-03-19 19:48:08.667 1001-1071/fr.free.nrw.commons.beta D/UploadService: Contribution upload success. Initiating Wikidata edit for entity id Q49048561
2020-03-19 19:48:08.667 1001-1071/fr.free.nrw.commons.beta D/WikidataEditService: Upload successful with wiki data entity id as Q49048561
2020-03-19 19:48:08.667 1001-1071/fr.free.nrw.commons.beta D/WikidataEditService: Attempting to edit Wikidata property Q49048561
2020-03-19 19:48:08.668 1001-1071/fr.free.nrw.commons.beta D/WikidataEditService: Wikidata property name is "Beautiful_house_-.jpg"
2020-03-19 19:48:08.668 1001-1071/fr.free.nrw.commons.beta D/WikidataEditService: Entity id is Q49048561 and property value is "Beautiful_house_-.jpg"
2020-03-19 19:48:08.676 1001-1093/fr.free.nrw.commons.beta D/OkHttp: --> GET https://www.wikidata.org/w/api.php?format=json&formatversion=2&errorformat=plaintext&action=query&meta=tokens&type=csrf
2020-03-19 19:48:08.677 1001-1093/fr.free.nrw.commons.beta D/CookieManager: Domain:www.wikidata.org
2020-03-19 19:48:09.025 1001-1076/fr.free.nrw.commons.beta V/OkHttp: <-- 200 https://query.wikidata.org/sparql?query=SELECT%0A%20%20%20%20%20%28SAMPLE%28%3Flocation%29%20as%20%3Flocation%29%0A%20%20%20%20%20%3Fitem%0A%20%20%20%20%20%28SAMPLE%28COALESCE%28%3FitemLabelPreferredLanguage%2C%20%3FitemLabelAnyLanguage%29%29%20as%20%3Flabel%29%0A%20%20%20%20%20%28SAMPLE%28%3FclassId%29%20as%20%3Fclass%29%0A%20%20%20%20%20%28SAMPLE%28COALESCE%28%3FclassLabelPreferredLanguage%2C%20%3FclassLabelAnyLanguage%2C%20%22%3F%22%29%29%20as%20%3FclassLabel%29%0A%20%20%20%20%20%28SAMPLE%28COALESCE%28%3Ficon0%2C%20%3Ficon1%29%29%20as%20%3Ficon%29%0A%20%20%20%20%20%3FwikipediaArticle%0A%20%20%20%20%20%3FcommonsArticle%0A%20%20%20%20%20%28SAMPLE%28%3FcommonsCategory%29%20as%20%3FcommonsCategory%29%0A%20%20%20%20%20%28SAMPLE%28%3Fpic%29%20as%20%3Fpic%29%0A%20%20%20%20%20%28SAMPLE%28%3Fdestroyed%29%20as%20%3Fdestroyed%29%0A%20%20%20WHERE%20%7B%0A%20%20%20%20%20%23%20Around%20given%20location...%0A%20%20%20%20%20SERVICE%20wikibase%3Aaround%20%7B%0A%20%20%20%20%20%20%20%3Fitem%20wdt%3AP625%20%3Flocation.%0A%20%20%20%20%20%20%20bd%3AserviceParam%20wikibase%3Acenter%20%22Point%28-121.4145%2039.2011%29%22%5E%5Egeo%3AwktLiteral.%0A%20%20%20%20%20%20%20bd%3AserviceParam%20wikibase%3Aradius%20%224.24%22%20.%20%23%20Radius%20in%20kilometers.%0A%20%20%20%20%20%7D%0A%0A%20%20%20%20%20%23%20Get%20the%20label%20in%20the%20preferred%20language%20of%20the%20user%2C%20or%20any%20other%20language%20if%20no%20label%20is%20available%20in%20that%20language.%0A%20%20%20%20%20OPTIONAL%20%7B%3Fitem%20rdfs%3Alabel%20%3FitemLabelPreferredLanguage.%20FILTER%20%28lang%28%3FitemLabelPreferredLanguage%29%20%3D%20%22en%22%29%7D%0A%20%20%20%20%20OPTIONAL%20%7B%3Fitem%20rdfs%3Alabel%20%3FitemLabelAnyLanguage%7D%0A%0A%20%20%20%20%20%23%20Get%20Commons%20category%20%28P373%29%0A%20%20%20%20%20OPTIONAL%20%7B%20%3Fitem%20wdt%3AP373%20%3FcommonsCategory.%20%7D%0A%0A%20%20%20%20%20%23%20Get%20%28P18%29%0A%20%20%20%20%20OPTIONAL%20%7B%20%3Fitem%20wdt%3AP18%20%3Fpic.%20%7D%0A%0A%20%20%20%20%20%23%20Get%20%28P576%29%0A%20%20%20%20%20OPTIONAL%20%7B%20%3Fitem%20wdt%3AP576%20%3Fdestroyed.%20%7D%0A%0A%20%20%20%20%20%23%20Get%20the%20class%20label%20in%20the%20preferred%20language%20of%20the%20user%2C%20or%20any%20other%20language%20if%20no%20label%20is%20available%20in%20that%20language.%0A%20%20%20%20%20OPTIONAL%20%7B%0A%20%20%20%20%20%20%20%3Fitem%20p%3AP31%2Fps%3AP31%20%3FclassId.%0A%20%20%20%20%20%20%20OPTIONAL%20%7B%3FclassId%20rdfs%3Alabel%20%3FclassLabelPreferredLanguage.%20FILTER%20%28lang%28%3FclassLabelPreferredLanguage%29%20%3D%20%22en%22%29%7D%0A%20%20%20%20%20%20%20OPTIONAL%20%7B%3FclassId%20rdfs%3Alabel%20%3FclassLabelAnyLanguage%7D%0A%0A%20%20%20%20%20%20%20OPTIONAL%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%3FwikipediaArticle%20%20%20schema%3Aabout%20%3Fitem%20%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20schema%3AisPartOf%20%3Chttps%3A%2F%2Fen.wikipedia.org%2F%3E%20.%0A%20%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%20%20OPTIONAL%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%3FwikipediaArticle%20%20%20schema%3Aabout%20%3Fitem%20%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20schema%3AisPartOf%20%3Chttps%3A%2F%2Fen.wikipedia.org%2F%3E%20.%0A%20%20%20%20%20%20%20%20%20%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22en%22%20%7D%0A%20%20%20%20%20%20%20%20%20%7D%0A%0A%20%20%20%20%20%20%20%20%20OPTIONAL%20%7B%0A%20%20%20%20%20%20%20%20%20%20%20%3FcommonsArticle%20%20%20schema%3Aabout%20%3Fitem%20%3B%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20schema%3AisPartOf%20%3Chttps%3A%2F%2Fcommons.wikimedia.org%2F%3E%20.%0A%20%20%20%20%20%20%20%20%20%20%20SERVICE%20wikibase%3Alabel%20%7B%20bd%3AserviceParam%20wikibase%3Alanguage%20%22en%22%20%7D%0A%20%20%20%20%20%20%20%20%20%7D%0A%20%20%20%20%20%7D%0A%20%20%20%7D%0A%20%20%20GROUP%20BY%20%3Fitem%20%3FwikipediaArticle%20%3Fcommon
2020-03-19 19:48:09.025 1001-1076/fr.free.nrw.commons.beta V/OkHttp: sArticle%0A&format=json (1609ms)
2020-03-19 19:48:09.025 1001-1076/fr.free.nrw.commons.beta V/OkHttp: server: nginx/1.13.6
2020-03-19 19:48:09.025 1001-1076/fr.free.nrw.commons.beta V/OkHttp: date: Thu, 19 Mar 2020 14:18:08 GMT
2020-03-19 19:48:09.025 1001-1076/fr.free.nrw.commons.beta V/OkHttp: content-type: application/sparql-results+json;charset=utf-8
2020-03-19 19:48:09.026 1001-1076/fr.free.nrw.commons.beta V/OkHttp: x-first-solution-millis: 16
2020-03-19 19:48:09.027 1001-1076/fr.free.nrw.commons.beta V/OkHttp: x-served-by: wdqs1005
gouri-panda picture gouri-panda on 19 Mar 2020 
    /**
     * Edits the wikidata entity by adding the P18 property to it.
     * Adding the P18 edit requires calling the wikidata API to create a claim against the entity
     *
     * @param wikidataEntityId
     * @param fileName
     */
    @SuppressLint("CheckResult")
    private void editWikidataProperty(String wikidataEntityId, String fileName) {
        Timber.d("Upload successful with wiki data entity id as %s", wikidataEntityId);
        Timber.d("Attempting to edit Wikidata property %s", wikidataEntityId);

        String propertyValue = getFileName(fileName);

        Timber.d("Entity id is %s and property value is %s", wikidataEntityId, propertyValue);
        wikidataClient.createClaim(wikidataEntityId, propertyValue)
                .flatMap(revisionId -> {
                    if (revisionId != -1) {
                        return wikidataClient.addEditTag(revisionId, COMMONS_APP_TAG, COMMONS_APP_EDIT_REASON);
                    }
                    throw new RuntimeException("Unable to edit wikidata item");
                })
                .subscribeOn(Schedulers.io())
                .observeOn(AndroidSchedulers.mainThread())
                .subscribe(revisionId -> handleClaimResult(wikidataEntityId, String.valueOf(revisionId)), throwable -> {
                    Timber.e(throwable, "Error occurred while making claim");
                    ViewUtil.showLongToast(context, context.getString(R.string.wikidata_edit_failure));
                });
    }

So we just alter getFileName to replace underscore with space?

    /**
     * Formats and returns the filename as accepted by the wiki base API
     * https://www.mediawiki.org/wiki/Wikibase/API#wbcreateclaim
     *
     * @param fileName
     * @return
     */
    private String getFileName(String fileName) {
        fileName = String.format("\"%s\"", fileName.replace("File:", ""));
        Timber.d("Wikidata property name is %s", fileName);
        return fileName;
    }

Or I could be misundestanding the api

macgills picture macgills on 19 Mar 2020 
 /**
     * Formats and returns the filename as accepted by the wiki base API
     * https://www.mediawiki.org/wiki/Wikibase/API#wbcreateclaim
     *
     * @param fileName
     * @return
     */
    private String getFileName(String fileName) {
        fileName = String.format("\"%s\"", fileName.replace("File:", ""));
        Timber.d("Wikidata property name is %s", fileName);
        return fileName;
    }

We can add another method here to replace underscore with space and then we store into Local Database Contribution Provider.

gouri-panda picture gouri-panda on 19 Mar 2020

You can try modifying that method and see if we get the right end result.
I actually just checked and that method is really annoying.
In UploadService we add the "File:" prefix
String canonicalFilename = "File:" + uploadResult.getFilename();
and then just replace it here before making sure this string has double quotes. That should be changed.

I also find it hard to believe this thing needs to be quoted? seeing as how the other parts are fine in the request just being strings.

Observable<Long> createClaim(String entityId, String value) {
        return getCsrfToken()
                .flatMap(csrfToken -> wikidataInterface.postCreateClaim(toRequestBody(entityId),
                        toRequestBody("value"),
                        toRequestBody("P18"),
                        toRequestBody(value),
                        toRequestBody("en"),
                        toRequestBody(csrfToken)))
                .map(mwPostResponse -> mwPostResponse.getPageinfo().getLastrevid());
    }

I am confused by pretty much all of this.

macgills picture macgills on 19 Mar 2020
👍1

I don't think we would have to handle the underscores manually in our code. If you take a look at the example response at the linked wiki page it has a canonicaltitle key inside imageinfo which seems to be the version of the title that Wikidata is expecting (after slicing the File: part). I haven't confirmed this but I believe that it would be case.

sivaraam picture sivaraam on 19 Mar 2020
👍1

If my theory is correct and I were to map the various API parameters in the example response with Nicolas' example above, this would be the mapping:

  • filename would correspond to the Director_at_premi猫re_of_Batman-Reborn in step 2 (if we remove the File:)
  • canonicaltitle within imageinfo would correspond to the Director at premi猫re of Batman-Reborn in step 4 (if we remove the File:)
sivaraam picture sivaraam on 19 Mar 2020
👍1

@sivaraam I am really confused about this code 

String canonicalFilename = "File:" + uploadResult.getFilename();
                        Timber.d("Contribution upload success. Initiating Wikidata edit for entity id %s",
                                contribution.getWikiDataEntityId());
                        wikidataEditService.createClaimWithLogging(contribution.getWikiDataEntityId(), canonicalFilename);
                        contribution.setFilename(uploadResult.getFilename());

                        contribution.setImageUrl(uploadResult.getImageinfo().getOriginalUrl());
                        contribution.setState(Contribution.STATE_COMPLETED);
                        contribution.setDateUploaded(CommonsDateUtil.getIso8601DateFormatShort()
                                .parse(uploadResult.getImageinfo().getTimestamp()));
                        compositeDisposable.add(contributionDao
                                .save(contribution)
                                .subscribeOn(ioThreadScheduler)
                                .observeOn(mainThreadScheduler)
                                .subscribe());
                    }

In the code the uploadResult method gives the image title with underscore but in the code database it stores without underscore.Can you please look at it?Thanks!
image

gouri-panda picture gouri-panda on 19 Mar 2020

@gouri-panda the database is irrelevant so don't focus on it.

wikidataEditService.createClaimWithLogging(contribution.getWikiDataEntityId(), canonicalFilename); this is the line you need to focus on to fix this ticket.

If ImageInfo has canonicaTitle then it isn't in this wikidata library

public class ImageInfo implements Serializable {
    private int size;
    private int width;
    private int height;
    @Nullable private String source;
    @SerializedName("thumburl") @Nullable private String thumbUrl;
    @SerializedName("thumbwidth") private int thumbWidth;
    @SerializedName("thumbheight") private int thumbHeight;
    @SerializedName("url") @Nullable private String originalUrl;
    @SerializedName("descriptionurl") @Nullable private String descriptionUrl;
    @SerializedName("descriptionshorturl") @Nullable private String descriptionShortUrl;
    @SerializedName("mime") @Nullable private String mimeType;
    @SerializedName("extmetadata")@Nullable private ExtMetadata metadata;
    @Nullable private String user;
    @Nullable private String timestamp;


public class ExtMetadata {
    @SerializedName("DateTime") @Nullable private Values dateTime;
    @SerializedName("ObjectName") @Nullable private Values objectName;
    @SerializedName("CommonsMetadataExtension") @Nullable private Values commonsMetadataExtension;
    @SerializedName("Categories") @Nullable private Values categories;
    @SerializedName("Assessments") @Nullable private Values assessments;
    @SerializedName("GPSLatitude") @Nullable private Values gpsLatitude;
    @SerializedName("GPSLongitude") @Nullable private Values gpsLongitude;
    @SerializedName("ImageDescription") @Nullable private Values imageDescription;
    @SerializedName("DateTimeOriginal") @Nullable private Values dateTimeOriginal;
    @SerializedName("Artist") @Nullable private Values artist;
    @SerializedName("Credit") @Nullable private Values credit;
    @SerializedName("Permission") @Nullable private Values permission;
    @SerializedName("AuthorCount") @Nullable private Values authorCount;
    @SerializedName("LicenseShortName") @Nullable private Values licenseShortName;
    @SerializedName("UsageTerms") @Nullable private Values usageTerms;
    @SerializedName("LicenseUrl") @Nullable private Values licenseUrl;
    @SerializedName("AttributionRequired") @Nullable private Values attributionRequired;
    @SerializedName("Copyrighted") @Nullable private Values copyrighted;
    @SerializedName("Restrictions") @Nullable private Values restrictions;
    @SerializedName("License") @Nullable private Values license;
macgills picture macgills on 20 Mar 2020
👍1

If ImageInfo has canonicaTitle then it isn't in this wikidata library

Yeah, that does seem to be a limitation of the data-client library[1]. I believe we can request them to provide a way to access the canonicaltitle in the response.

sivaraam picture sivaraam on 20 Mar 2020
👍1

@sivaraam Thanks for the research!

@gouri-panda Communicating with upstream is an important part of doing open source. Please make sure you understand the issue and write the enhancement request that you would send to https://github.com/wikimedia/wikimedia-android-data-client . Then we will check it and then you can post it there and wait for their answer. Thanks :-) https://blog.wikimedia.org/2015/10/14/bug-reports-and-feature-requests/ might give you some tips.

nicolas-raoul picture nicolas-raoul on 21 Mar 2020
👍1

The app is using my fork for the data-client as was dbrant was not available for maintaining the original repo.

https://github.com/maskaravivek/wikimedia-android-data-client

PS: We should transfer the ownership of this fork to commons-app.

maskaravivek picture maskaravivek on 22 Mar 2020

@gouri-panda Do you mind if I take up this issue as I am working on #1831 and it will change the implementation of how we set a claim. See WIP PR for more details #3449.

maskaravivek picture maskaravivek on 22 Mar 2020

Sure @maskaravivek :)

gouri-panda picture gouri-panda on 22 Mar 2020
👍1

@maskaravivek a bit off topic here but do we even need the wikimedia data library? Moving it into the app just seems easier as we won't have to worry about publication. I guess the question is how many projects are dependent on it?

macgills picture macgills on 23 Mar 2020

but do we even need the wikimedia data library?

IIRC, it's a fairly recent integration. Here's an issue which you would give you some context about this: #1863 (TL;DR: https://github.com/commons-app/apps-android-commons/issues/1863#issuecomment-418163099)

I guess the question is how many projects are dependent on it?

I guess you'll find an answer for this in the comment mentioned before.

sivaraam picture sivaraam on 23 Mar 2020

@maskaravivek a bit off topic here but do we even need the wikimedia data library? Moving it into the app just seems easier as we won't have to worry about publication. I guess the question is how many projects are dependent on it?

@macgills Yes, there's a lot of history behind why the data-client came into picture and honestly I would prefer to have everything in our main codebase rather than maintain a library that too for model classes which keep changing frequently. That being said let us continue this discussion in a separate issue. :)

maskaravivek picture maskaravivek on 23 Mar 2020

discussion to continue here #3571

macgills picture macgills on 23 Mar 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

misaochan picture misaochan  路  4Comments
madhurgupta10 picture madhurgupta10  路  4Comments
nicolas-raoul picture nicolas-raoul  路  4Comments
madhurgupta10 picture madhurgupta10  路  3Comments
misaochan picture misaochan  路  3Comments