Pgloader: New Feature: Conditionally filter logical rows in a CSV

[davidklebanoff]

I think the ability to conditionally skip/filter a logical row in a CSV, based on the value of a single field (or even better, multiple fields) could be super useful. The conditional logic could be a simple lisp function that returns a boolean value. If the boolean is false, the row is skipped, if true, it is inserted into the database.

A few scenarios to illustrate the value:

• You're importing a massive table of events, but you only want to import events from from last year. ( year == 2016)
• You're importing a massive table of language translations, but you only want english. (lang = "en")
• You're importing a massive inventory table, but you only want items that are in stock. (count > 0)

I'm happy to discuss the idea further if needed.

[dimitri]

That's a feature I have already been thinking about and I think we should work on it yes. One of the things I want to do here is to enable targeting more than a single table depending on what you find in the data, e.g. target directly the right yearly partition depending on the year field in the file.