Luigi: Output from Pandas DataFrame

[thiagueraGyn]

Hello guys. I'm new to using Luigi, so my question is probably pretty simple. I have a processing that is done through Pandas DataFrame, with that, I need the output to be the result of this DataFrame.
Here's an example of what I intend to do.

class HubTask(luigi.WrapperTask):
    def requires(self):
        for index, row in energy.get_files().iterrows():
            yield Process(id_arquivo = row['ID_CONTROLE_ARQ_CONVENIO_ICMS'])

class Process(luigi.Task):
    id_arquivo = luigi.Parameter()

    def requires(self):
        ...

    def run(self):
        ...

    def output(self):
        return self.data_frame.to_csv('../temp/process_{}'.format(self.id_arquivo))

How could this be done?

[raycheung]

You can just mark a local file target, i.e. wrapping the path in a LocalTarget like luigi.LocalTarget('../temp/process_{}'.format(self.id_arquivo))

[dlstadther]

Yes, @raycheung is correct. output()s should return a Target. So in the case of a df csv, you'd to_csv() near the end of your run() and it would reference your self.output().path.

[thiagueraGyn]

Like this @dlstadther ?

def run(self):
        time.sleep(5)
        df_invoices = pd.read_csv('../data/notas.csv', sep=';')
        df_invoices.loc[df_invoices['ID_CONTROLE_ARQ_CONVENIO_ICMS'] == self.file_id]
        self.data_frame = df_invoices
        self.data_frame.to_csv()
        ...
def output(self):
        return luigi.LocalTarget('../temp/invoices_{}.csv'.format(self.file_id))

But I still have not understood the question of self.output().path

[dlstadther]

Generally seems ok.

[dlstadther]

Here's a sample gist i created and updated awhile back. The base level of it is a dataframe thing

[thiagueraGyn]

That's right, thank you @dlstadther and @raycheung. My problem has been solved.

[dlstadther]

Glad we could help!