Gallery-dl: Using filters on flickr

[nihelmasell]

Hi, I write you this because I don't understand how to use the filter when trying to download results from a flickr search. I used the Apollo Photo Archive for a model, and obtained the following result:

gallery-dl --list-keywords https://www.flickr.com/photos/projectapolloarchive/

Keywords for directory names:

category
flickr
subcategory
user
user[nsid]
136485307@N06
user[path_alias]
projectapolloarchive
user[username]
Apollo Image Gallery

Keywords for filenames and --filter:

category
flickr
extension
jpg
farm
2
filename
40563460950_df901aea9d_o.jpg
id
40563460950
isfamily
0
isfriend
0
ispublic
1
name
40563460950_df901aea9d_o
owner
136485307@N06
photo[height]
8238
photo[label]
Original
photo[media]
photo
photo[source]
https://farm2.staticflickr.com/1739/40563460950_df901aea9d_o.jpg
photo[width]
6590
secret
e165e0f3c9
server
1739
subcategory
user
title
Astronaut Alan L. Bean, Lunar Module Pilot of Apollo 12
user[nsid]
136485307@N06
user[path_alias]
projectapolloarchive
user[username]
Apollo Image Gallery

What syntax should I use for a) downloading general flickr photos by one or more keywords b) downloading flickr photos from a known flickr account?

I didn't find any example not here nor on the rest of the Internet.

TIA

[mikf]

Filters (--filter or image-filter) are, as the description says, simple Python expressions that should evaluate to True or False and the values listed by -K can be used as local variables. A simple filter for your example URL would be

$ gallery-dl --filter "isfriend" https://www.flickr.com/photos/projectapolloarchive/
# or
$ gallery-dl --filter "int(photo['width']) > 4000" https://www.flickr.com/photos/projectapolloarchive/

(Note that photo['width'] is a string and needs to be converted to an integer before comparing it to another number)

a) downloading general flickr photos by one or more keywords

I'm not entirely sure I understand what you mean here ...
If you want the results of a search on Flickr, just put the search URL into gallery-dl:

$ gallery-dl "https://www.flickr.com/search/?text=tree%20cloud%20house&color_codes=4&styles=minimalism"

b) downloading flickr photos from a known flickr account

Use the account's URL, i.e. gallery-dl https://www.flickr.com/photos/projectapolloarchive/
Again, I'm not really sure what your problem here is.

[nihelmasell]

Hi, your answer is fully comprehensive. Its me who hasn't expressed correctly. I'm gonna place two specific cases. 1) Suppose I want to download the photos with name in title "Apollo" and "11" from the previously mentioned Apollo Project (only inside their flickr account). Which syntax should I use? 2) I want to download all pictures from flickr with keywords "beach" "volley", not caring how many of them are (50.000+, idc) Is that possible? Thanks again.

[mikf]

1) Suppose I want to download the photos with name in title "Apollo" and "11" from the previously mentioned Apollo Project (only inside their flickr account). Which syntax should I use?

# split title into list of words and test if "Apollo" and "11" are in it
$ gallery-dl --filter "'Apollo' in title.split()  and  '11' in title.split()" https://www.flickr.com/photos/projectapolloarchive/

# or with a regular expression
$ gallery-dl --filter "re.search(r'\bApollo\b.*\b11\b', title)" https://www.flickr.com/photos/projectapolloarchive/

2) I want to download all pictures from flickr with keywords "beach" "volley"

Can't this be done with a search on Flickr?

$ gallery-dl "https://www.flickr.com/search/?text=beach%20volley"

[nihelmasell]

Thanks for your last response. I didn't actually know it accepted regex. One last question: Will $ gallery-dl "https://www.flickr.com/search/?text=beach%20volley" actually download all 49,253 current results? On its full-resolution size? Merci!

[mikf]

I didn't actually know it accepted regex

Filter expressions are just Python code, so anything you can do with Python you can do as a "filter".

Will gallery-dl "https://www.flickr.com/search/?text=beach%20volley" actually download all 49,253 current results? On its full-resolution size? Merci!

Maybe not always full resolution, but the highest-available one.
I'm pretty sure you get access to higher-resolution versions if you use an access-token and -secret (see OAuth Authentication)

[nihelmasell]

Hi, I'm sorry for bothering you again. I thank now with words the previous answers. I've finally been able to test the previously mentioned formula, regarding the two-worded flickr search. The app exited 0, with no error. Nevertheless I noticed that the actual images downloaded from the two keywords I used represent less than the third than the total result on the webpage. To what cause should this be attributed, to a ban for myself (I used credentials) -no error message on the app-, or to a miss-showing of results on flickr's homepage? It is my intention to test the same on Jdownloader (an app I hate due to memory consumption) but didn't have time. You were right about the credentials used, I got 20mb-6000mp images!! My best regards, my deepest THANKS.

[mikf]

To what cause should this be attributed, to a ban for myself (I used credentials) -no error message on the app-, or to a miss-showing of results on flickr's homepage?

The filter from above only inspect an image's title, nothing else. Flickr's own search algorithm is using a lot more metadata (like tags, image descriptions, etc.) when searching for images.

I'm not too familiar with Flickr itself, but by looking around I found a way to search for images in a specific Photostream. On the main Photostream page itself (https://www.flickr.com/photos/projectapolloarchive/) you just have to click the magnifying glass in the upper right. Searching for "Apollo 11" gets you to the following URL, which you can use with gallery-dl:
https://www.flickr.com/search/?user_id=136485307%40N06&view_all=1&text=Apollo%2011

[nihelmasell]

Hi, no, I actually meant I had an issue when trying to download a "general" flickr search like "
https://www.flickr.com/search/?text=beach%20volley " The results showed on flickr webpage are much more than the ones actually downloaded by the script. I have been only able to download a third of all reported images. After that the app exit, with no error (but the one you may notice in the difference between the numbers of results on the webpage and the actual files on your hard drive).

[mikf]

Maybe it has something to do with your account's SafeSearch settings?

Otherwise I really don't know why this is happening. The displayed number of search results is always a bit higher than what is actually returned, even on the site itself, but only 1/3 of 45k seems wrong.
gallery-dl is simply using the flickr.photos.search API endpoint with the same parameters as the original URL and returns all results as they come, so I would say this is Flickr's "fault".

[nihelmasell]

Hi, I invite you to try it for yourself.
https://www.flickr.com/search/?text=beach+handball (its not actually "beach volley", but "beach handball" the sport I'm interested in) Of the 13,817 displayed results I only get 3,916. Safesearch is turned off in my account.

[nihelmasell]

So, well 100% sure its flickr`s fault and not anything related to your app? I found a similar app to yours but haven't tried it yet..

[mikf]

I'm sure it is Flickr's fault.

I noticed that for higher page numbers (100+), all returned search results were the same, i.e. page 102 was the same as page 103, 104, etc. So I "investigated" a bit further and re-read the API docs until I found the following paragraph:

Please note that Flickr will return at most the first 4,000 results for any given search query. If this is an issue, we recommend trying a more specific query.

So everything after the first 4000 images / 40 pages are just repeats of the previous results (probably all the same as the first page)

I guess you can, as they said, work around this limitation by adding some extra search queries and negating them afterwards:

$ gallery-dl https://www.flickr.com/search/?text=beach+handball+water
$ gallery-dl https://www.flickr.com/search/?text=beach+handball+-water

[nihelmasell]

An awesomely interesting find. So, we could affirm, that any search done on flickr, independent to the application we use, retrieves 4000 results. I guess this also applies to the "Apollo" (specific user) search example. Am I right? Thanks in advance for your help (now that flickr is mass-eliminating photos your script acquires unlimited value)

[mikf]

So, we could affirm, that any search done on flickr, independent to the application we use, retrieves 4000 results.

Yes, I would think so. I wouldn't be surprised if the web search results on their own site have the same limitations.

I guess this also applies to the "Apollo" (specific user) search example. Am I right?

It is a search and uses the flickr.photos.search API endpoint, so yes, this limitation should apply here as well.