Aut: UDFs that filter on url should also filter on src

Created on 10 Feb 2020  路  5Comments  路  Source: archivesunleashed/aut

We are currently unable to run a number of DataFrame filters on .imageLinks() and webgraph() because they have src and/or dest columns instead of url. The DataFrame filters should be able to filter on those columns as well.

DataFrames Scala enhancement

Most helpful comment

Based on the chat @lintool and I where having in Slack this morning, it'd be amending the current functions. I _think_ we could just do this with try cases (oh, I don't know what the proper Scala term is for it :laughing: ) for url and src. I don't think we need to do dest or image_url, though @ianmilligan1 might have a use case for that. How's that sound?

All 5 comments

@SinghGursimran want this one since we're stuck in a holding pattern on the Python side of things until I sort out the Scala UDF -> Python UDF linkage?

@ruebot Shall I add a new function to incorporate src and dest OR accommodate this within the same function using an extra argument?
Amending the current function would require a change in docs as well...

Based on the chat @lintool and I where having in Slack this morning, it'd be amending the current functions. I _think_ we could just do this with try cases (oh, I don't know what the proper Scala term is for it :laughing: ) for url and src. I don't think we need to do dest or image_url, though @ianmilligan1 might have a use case for that. How's that sound?

Ok....

@SinghGursimran If it helps to see an actual use case/test case, this is how it popped up: https://gist.github.com/ruebot/60b5f848252284b7f380e3d5006d7135

I tried to run the .imagegraph() version of the script over the weekend only to realize I couldn't do it :man_facepalming:

Was this page helpful?
0 / 5 - 0 ratings

Related issues

lintool picture lintool  路  6Comments

ruebot picture ruebot  路  4Comments

ruebot picture ruebot  路  5Comments

lintool picture lintool  路  8Comments

obrienben picture obrienben  路  8Comments