Picongpu: tbg fails in screen session

[PrometheusPi]

When using tbg within a screen session, the execution fails and the following error occures:

sed: -e expression #1, char 152: unterminated `s' command
ERROR: 13 variable(s) _not_ replaced from template (tpl):
!TBG_author !TBG_coresPerNode !TBG_dstPath !TBG_gpusPerNode !TBG_jobName !TBG_mailAddress !TBG_mailSettings !TBG_nodes !TBG_profile !TBG_programParams !TBG_queue !TBG_tasks !TBG_wallTime

I tested this for all *.tpl for hypnos with a default *.cfg from the Bunch example.
@ComputationalRadiationPhysics/picongpu-developers can you reproduce this bug?
Especially when working on a crappy mobile connection, screen is extremely useful and tbg thus should run in a screen session.

[theZiz]

At least I know of this bug and also told @psychocoderHPC but as it seems I never openend an issue. cough
So, yes, this would be very handy to work and tbh I habe no idea, why it does not.

[PrometheusPi]

offline discovery with @psychocoderHPC:
In the screen session, there exists the environment variable TERMCAP:

$ env # prints
TERMCAP=SC|screen|VT 100/ANSI X3.64 virtual terminal:\
        :DO=\E[%dB:LE=\E[%dD:RI=\E[%dC:UP=\E[%dA:bs:bt=\E[Z:\
        :cd=\E[J:ce=\E[K:cl=\E[H\E[J:cm=\E[%i%d;%dH:ct=\E[3g:\
        :do=^J:nd=\E[C:pt:rc=\E8:rs=\Ec:sc=\E7:st=\EH:up=\EM:\
        :le=^H:bl=^G:cr=^M:it#8:ho=\E[H:nw=\EE:ta=^I:is=\E)0:\
        :li#83:co#133:am:xn:xv:LP:sr=\EM:al=\E[L:AL=\E[%dL:\
        :cs=\E[%i%d;%dr:dl=\E[M:DL=\E[%dM:dc=\E[P:DC=\E[%dP:\
        :im=\E[4h:ei=\E[4l:mi:IC=\E[%d@:ks=\E[?1h\E=:\
        :ke=\E[?1l\E>:vi=\E[?25l:ve=\E[34h\E[?25h:vs=\E[34l:\
        :ti=\E[?1049h:te=\E[?1049l:us=\E[4m:ue=\E[24m:so=\E[3m:\
        :se=\E[23m:mb=\E[5m:md=\E[1m:mr=\E[7m:me=\E[m:ms:\
        :Co#8:pa#64:AF=\E[3%dm:AB=\E[4%dm:op=\E[39;49m:AX:\
        :vb=\Eg:G0:as=\E(0:ae=\E(B:\
        :ac=\140\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:\
        :po=\E[5i:pf=\E[4i:Km=\E[M:k0=\E[10~:k1=\EOP:k2=\EOQ:\
        :k3=\EOR:k4=\EOS:k5=\E[15~:k6=\E[17~:k7=\E[18~:\
        :k8=\E[19~:k9=\E[20~:k;=\E[21~:F1=\E[23~:F2=\E[24~:\
        :F3=\E[1;2P:F4=\E[1;2Q:F5=\E[1;2R:F6=\E[1;2S:\
        :F7=\E[15;2~:F8=\E[17;2~:F9=\E[18;2~:FA=\E[19;2~:kb=:\
        :K2=\EOE:kB=\E[Z:kF=\E[1;2B:kR=\E[1;2A:*4=\E[3;2~:\
        :*7=\E[1;2F:#2=\E[1;2H:#3=\E[2;2~:#4=\E[1;2D:%c=\E[6;2~:\
        :%e=\E[5;2~:%i=\E[1;2C:kh=\E[1~:@1=\E[1~:kH=\E[4~:\
        :@7=\E[4~:kN=\E[6~:kP=\E[5~:kI=\E[2~:kD=\E[3~:ku=\EOA:\
        :kd=\EOB:kr=\EOC:kl=\EOD:km:

Unsetting this variable unset TERMCAP lets tbg work properly. But this will alter the setting of screen. See this documentation.

[jkelling]

I can reproduce this issue working in a screen session. Unsetting TERMCAP also solves is for me.

The displayed sed error occurs in line 53 when it tries to process TERMCAP

tooltpl_replace_data=`echo "$tooltpl_replace_data" | sed "s/!$tooltpl_dst([^[:alnum:]_]{1,1})/$tooltpl_src_esc\1/g ; s/!$tooltpl_dst$/$tooltpl_src_esc/g"`

But the reason none of the TBG Variables is replaced may well be somewhere else.

One should escape all Variables passed to sed.

A more dirty approach could be whitelisting Variables starting with TBG_ . Would this break some existing code?

[ax3l]

I added a less intrusive black-listing in #2262

[ax3l]

the underlying problem in TERMCAP is the char \1 which sed uses for groups (\"N").

export AAA='\1'
tbg -c etc/picongpu/0004gpus.cfg -t etc/picongpu/bash/bash_mpirun.tpl ~/runs/khi_027
# fail
# sed: -e expression #1, char 54: invalid reference \1 on `s' command's RHS

let's fuzzy the env for tbg :D

[jkelling]

If you source the .cfg file to get the variables to be replaced in the template and then use env to obtain a list of them for processing, can you not just start a sub-shell with empty environment inside tbg before sourcing the .cfg and doing all the replacement? This way you could generically keep any unwanted Variables from slipping in. tbg has the -o option to pass additional variable definitions, so a user should not expect it to respect arbitrary variables defined in the environment in which tbg is started anyway.

[ax3l]

we found it. so far, we only escaped / but the \ and & are sed magic, too.
updated the PR.