Tinytex: Warning on shadowing format

[cderv]

This is related to #308 as the warning are the same.

This is the message we see:
tlmgr update --all --self fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for luatex/luatex!!! fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for tex/tex!!! fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for xetex/xetex!!! fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for pdftex/pdftex!!! fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for luahbtex/luahbtex!!! fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for dvilualatex/luatex!!! fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for lualatex/luahbtex!!! fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for dviluatex/luatex!!! fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for latex/pdftex!!! fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for mf/mf-nowin!!! fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for etex/pdftex!!! fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for xelatex/xetex!!! fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for pdflatex/pdftex!!! fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for pdfetex/pdftex!!!

This is from a very recent change in TeXLive - see https://www.tug.org/svn/texlive/trunk/Master/texmf-dist/scripts/texlive/fmtutil.pl?r1=58958&r2=58959

We now get that with TinyTeX each time, that is why I wonder if we have a correct setup. Step to reproduce

> tinytex::uninstall_tinytex()  
Restarting R session...  
> tinytex::is_tinytex() 
[1] FALSE 
> tinytex::install_tinytex() trying URL 'https://yihui.org/tinytex/TinyTeX-1.tar.gz' 
Content type 'application/octet-stream' length 67314486 bytes (64.2 MB) ================================================== 
downloaded 64.2 MB  

tlmgr option sys_bin ~/bin 
tlmgr: setting option sys_bin to /home/cderv/bin. 
tlmgr: updating /home/cderv/.TinyTeX/tlpkg/texlive.tlpdb 
> tinytex::tlmgr_install('xcolor') 
tlmgr update --all --self 
fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for xetex/xetex!!! 
fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for luatex/luatex!!! 
fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for tex/tex!!! 
fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for pdftex/pdftex!!! 
fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for luahbtex/luahbtex!!! 
fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for lualatex/luahbtex!!! 
fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for dvilualatex/luatex!!! 
fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for latex/pdftex!!! 
fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for pdflatex/pdftex!!! 
fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for mf/mf-nowin!!! 
fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for dviluatex/luatex!!! 
fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for xelatex/xetex!!! 
fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for etex/pdftex!!! 
fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for pdfetex/pdftex!!! 

tlmgr install xcolor 
tlmgr: package repository https://ctan.math.illinois.edu/systems/texlive/tlnet (verified) tlmgr install: package already present: xcolor
 

Just sharing it here if we need to do something on our side to prevent these cryptic warnings.
But maybe it is something in TeX Live itself and it will disappear.

[yihui]

I don't know what this message means. The message sounds like certain .fmt files shouldn't have existed. I tried to delete texmf-var/web2c/xetex.fmt, but tlmgr update --all --self would regenerate it, and then I got the same warning.

I think we will need @norbusan's help here.

[norbusan]

Hi @yihui
that is a new feature we added to fmtutil recently. The idea is that most formats are in TEXMFSYSVAR and everyone uses fmtutil -sys. But then at times it happens that formats appear in TEXMFVAR and thus later updates in TEXMFSYSVAR will be shadowed by the files in TEXMFVAR.

We now warn if a format file is updated in TEXMFSYSVAR, but it would be never used because the same format is also in TEXMFVAR.

We also change mktexfmt to try to write to TEXMFSYSVAR if writable instead of TEXMFVAR, so that on-the-fly generated formats are installed in system directories.

See the output of fmtutl --help or the commit message

commit d408beb505856c02bbae6121e29e4cb285077719
Author: Norbert Preining <[email protected]>
Date:   Sun May 9 02:23:44 2021 +0000

    fmtutil and tlmgr improvements

    fmtutil.pl
    - if run as mktexfmt and TEXMFSYSVAR is writable, use
      it instead of TEXMFVAR (actual code in TLUtils.pm).
    - add --dry-run | -n option
      it is not guaranteed to be 100% dry-run, though
    - add --status-file=FILE where the status of each
      format creating is written to
    - warn on running fmtutil in user mode the first time
      (similar to updmap)
    - warn on shadowing formats
    - documentation fixes

    tlmgr.pl
    - use fmtutil --status-file and report back failed
      format rebuilds
    - regeneration all formats now only exists existing
      formats (use --refresh with fmtutil)
    - report when commands are logged to the respective
      log file
    - documentation fixes

    TLUtils.pm
    - update setup_sys_user_mode for the fmtutil/mktexfmt
      changes mentioned above
    - documentation fixes

    git-svn-id: svn://tug.org/texlive/trunk@59143 c570f23f-e606-0410-a88d-b1316a301751

[yihui]

@norbusan Oh then I guess I know what the problem is now. TinyTeX is a self-contained portable directory, which means its TEXMFSYSVAR and TEXMFVAR are identical, so the warnings are actually false positives. Is it possible to skip the warning in check_and_warn_on_user_format() for this special case? Thanks!

[norbusan]

Indeed... Please send me an email with the last text as reminder, will fix it tonight

[norbusan]

Thanks for the reminder (I was on the beach and would have surely forgotten it).

But there is something strange going on - we already return ..

sub check_and_warn_on_user_format {
  my ($fmt, $eng) = @_;
  # do nothing if we are updating files in $TEXMFVAR
  return if ($opts{'fmtdir'} eq $TEXMFVAR);
  ...

So if TEXMFSYSVAR == TEXMFVAR then the above check should return without printing any warning.

Can you somehow show the actual values by doing something like:

sub check_and_warn_on_user_format {
  my ($fmt, $eng) = @_;
  print_deferred_warning("TEXMFVAR=$TEXMFVAR TEXMFSYSVAR=$TEXMFSYSVAR optsfmtdir=$opts{'fmtdir'}\n");
  # do nothing if we are updating files in $TEXMFVAR
  return if ($opts{'fmtdir'} eq $TEXMFVAR);
  ...

And recreate one shadowed format and report the output. That would help a lot!

[yihui]

@norbusan TEXMFVAR and TEXMFSYSVAR are identical (as expected), but optsfmtdir is the subdir web2c under TEXMFSYSVAR:

fmtutil [WARNING]: TEXMFVAR=/Users/yihui/Library/TinyTeX/texmf-var TEXMFSYSVAR=/Users/yihui/Library/TinyTeX/texmf-var optsfmtdir=/Users/yihui/Library/TinyTeX/texmf-var/web2c
fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for luatex/luatex!!!
fmtutil [WARNING]: TEXMFVAR=/Users/yihui/Library/TinyTeX/texmf-var TEXMFSYSVAR=/Users/yihui/Library/TinyTeX/texmf-var optsfmtdir=/Users/yihui/Library/TinyTeX/texmf-var/web2c
fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for eptex/eptex!!!
fmtutil [WARNING]: TEXMFVAR=/Users/yihui/Library/TinyTeX/texmf-var TEXMFSYSVAR=/Users/yihui/Library/TinyTeX/texmf-var optsfmtdir=/Users/yihui/Library/TinyTeX/texmf-var/web2c
fmtutil [WARNING]: you have a shadowing format dump in TEXMFVAR for xetex/xetex!!!
...

[norbusan]

Umpf indeed, how stupid of me. Sorry for that.
Fixed in svn, will be in tomorrow's update.

[yihui]

Great! Thank you!

[cderv]

Awesome! Thanks for the investigation and fix! 👌

[yihui]

I can confirm that the warnings are gone after I run tlmgr update --self.

[norbusan]

Thanks for the confirmation!