Luigi: Feature Request: binary write mode for LocalTarget

[mwheeler-hdai]

I'd like to use pickle.dump to write to a LocalTarget after calling open in write mode, kind of like this:

def run(self):
    some_object = object()
    local_target = self.output()  # An instance of LocalTarget
    with local_target.open(mode="w") as f:  # <-- binary write not supported
        pickle.dump(some_object, f)

Unfortunately this causes an error: TypeError: write() argument must be str, not bytes because the LocalTarget open method doesn't allow binary write mode. This appears to be a deliberate decision, as the source code for the open method just strips the b out of the mode argument.

It's easy enough to work around this by using self.output().path as suggested by this kind stranger. Nonetheless I think it would be nice either to include this capability as part of the LocalTarget open method, or to throw a more informative error instead of just stripping out the b.

[lallea]

Hi,

I am not a maintainer, but I'm currently using these parts of Luigi heavily, and familiar with the code. If you create a PR, I am happy to review.

[charlesoblack]

Hey @mwheeler-hdai , iirc this is something of a design decision due to how the atomic file writes work. Instead of using format=UTF8 or whatever, use format=Nop - this fixes your problem. I'll try and dig up the issue that mentions this.

Edit: Here's an issue: #1647 and here's a PR: #2495 which unfortunately did not get merged. Maybe we could revisit this?

[mwheeler-hdai]

Or perhaps for now we just include a word or two about the option format=Nop in the docs for LocalTarget, or again in an error message at that point in the code. I feel like that's an acceptable middle ground, especially if this issue has been unsuccessfully raised before.

[Tarrasch]

Would love any kind of improvement (perhaps just docs and code comment), the "silently remove the b" way is not ideal.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If closed, you may revisit when your time allows and reopen! Thank you for your contributions.

[charlesoblack]

Ah, crap. Forgot to comment on here. Could we re-open this? I'll try and write a PR for docs within the next week or so.