Esmvaltool: cmorize_obs logs to stderr

[mattiarighi]

Running cmorize_obs as a batch job at DKRZ, I noticed that the log is being written to the standard error instead of the standard output file.

Probably easy to fix.

[schlunma]

Ahhh, that explains why my log was empty yesterday even though the job finished successfully :grinning:

[valeriupredoi]

have you given the thought that your are doing everything wrong? :grin: Nah, seriously, I'll have a look now :beer:

[valeriupredoi]

OK guys how are you running the batch? Ie I am assuming that it's SLURM that you are using as scheduler and if it's not configured somewhat custom you will have this guy's problem https://stackoverflow.com/questions/46908216/slurm-sbatch-creates-many-small-output-files - did you turn off the --output and --error flags?

[valeriupredoi]

also here is a solution how to change the output frequency as well https://stackoverflow.com/questions/25170763/how-to-change-how-frequently-slurm-updates-the-output-file-stdout/25189364#25189364

[mattiarighi]

I'm using these settings (@schlunma as well, I guess)

#!/bin/bash
#SBATCH --partition=prepost
#SBATCH --exclusive
#SBATCH --nodes=1
#SBATCH --mem=0
#SBATCH --time=08:00:00
#SBATCH --mail-type=FAIL,END
#SBATCH --output=job_%j.out.log
#SBATCH --error=job_%j.err.log

Note that esmvaltool logs fine with the same settings.

[schlunma]

That's not an issue of SLURM. If I run

cmorize_obs -c ~/config/cmorizer.yml -o CRU 1>AAA.out

the file AAA.out is empty, if I run

cmorize_obs -c ~/config/cmorizer.yml -o CRU 2>AAA.out

the file includes the log. Since 2> re-directs only the stderr output, all the output of cmorizer_obs is written to stderr.

[valeriupredoi]

yeah correct, hold your horses, I am just about there, want to test it first (even if I don't have sbatch) :beer: