pandoc 2.7+ and cygwin pdflatex fails to convert input to pdf

[cc2x]

I'm creating this pandoc issue as requested by John MacFarlane in a pandoc-discuss post of the same name. I'm providing the relevant items from the post below:

I downloaded the src files for 2.6 and 2.7 as this transition is where my usage no longer worked as described. I noticed two things .. (1)starting with 2.7, PDF.hs is importing withSystemTempDirectory and (2) there's some #ifdef _WINDOWS conditional compilation statements that disappeared in 2.7.

1. I want to say that withSystemTempDirectory may be the what's giving me the windows/dos drive representation of the posix /tmp folder. according to System.IO.Temp, it behaves the same as withTempDirectory except the parent temp dir will be that returned by getCanonicalTemporaryDirectory. It looks like there's no way around the new behavior that's introduced as part of 2.7.x.

2. the release notes state that if the tmp dir is "~" for a truncated dos 8.3 name, then it will use the current directory (2.6 behavior). as i've mentioned before, this breaks the old 2.6 behavior since PDF.hs appends input.tex using the windows directory delimiter. this gives us a mix between forward and back slash (see my observation in an April 12 post). Digging through the code, i noticed that after appending input.tex, there's no "normalization" of the directory delimiters as we see in PDF.hs version 2.6 (line 279).

If i add #ifdef _WINDOWS and let file' = changePathSeparators file in runTexProgram a few lines after let file = tmpDir "input.tex", I no longer have the mixed directory delimiters. Additional changes were necessary in the code to use the new file' variable. I tested this out by downloading the haskell platform, making the change in the 2.7.2 pandoc source and running stack build. I can confirm that with this code change, if I set my TMP environment variable to "~", then i get the default 2.6 behavior. So I do have a viable workaround to get the 2.6 behavior and obtaining feature enhancements in 2.7.

Here's the link for the full thread: https://groups.google.com/forum/#!topic/pandoc-discuss/Hc5eTUv7FGg

[jgm]

When you tested this, did you compile pandoc in a cygwin environment? If you did, that may have led it to use the POSIX version of directory. If I'm right that the problem comes from using the normal pandoc Windows binary (not compiled in a POSIX environment) inside cygwin, then this might not be a sufficient test.

[cc2x]

I compiled from a Win10 environment (cmd shell) as I was unable to locate a haskell compiler for cygwin (and the searches didn't provide useful information) and I wasn't able to get much information on building a haskell compiler for cygwin.

The changes I added to the src for a windows build allowed me to get back to the 2.6 behavior by normalizing the input.tex temporary file so that I can use the pandoc binary in a cygwin environment.

[cc2x]

I just now compiled pandoc in a cygwin environment (cygwin bash) and using the ghc windows compiler. Unfortunately, the resultant binary exhibits the same behavior as the official windows pandoc 2.7.2 build with the directory name conventions:

output:

! I can't find file `C:'.

/
<*> C:/
cygwin64/tmp/tex2pdf.-bc4041a8eb600bcainput.tex
(Press Enter to retry, or Control-D to exit)
Please type another input file name
! Emergency stop.

[jgm]

If you compiled outside of cygwin, that's good. That suggests that (with your changes), the windows binaries we produce will also work in cygwin.

[jgm]

@cc2x it would be great if you could try compiling the current HEAD (preferably outside of cygwin) and running it in cygwin. I've optimistically assumed that this commit closes the issue, but it stands in need of testing.

If compiling is difficult, you could grab the build artifact from appveyor when the build completes.

[cc2x]

I took a snapshot of the head this morning and compiled from a windows cmd shell and ran the resultant binary in cygwin. With your changes, it does resolve #2 in the issue. So there is a work around for consumers like me by setting the TMP environment to "~" . This will create the te mpfolder in the current working directory like in the past.

It would be great to get #1 where running under cygwin pandoc would use the posix directory naming rather than the transformed windows directory name.

I also ran "stack test" from a windows cmd shell and it reported no errors for all 2087 tests.

[jgm]

OK, I'll reopen for # 1.

[jgm]

It would be great to get #1 where running under cygwin pandoc would use the posix directory naming rather than the transformed windows directory name.

So how can we determine at runtime that we're runinng under cygwin?

[jgm]

On reflection, I'm tempted to think that the best solution is just for cygwin users who use our precompiled binary to set TMP=~ or TMP=/tmp. Using the system temp directory is the right default.

[jiucenglou]

As shown in https://groups.google.com/forum/#!topic/pandoc-discuss/zBypkBXQOT8, I cannot run pandoc under Cygwin after upgrading (from 2.2) to 2.7.3.

I have been using TexLive 2017 and Pandoc 2.2 under Cygwin in Windows. The command line is

pandoc TEST.md --filter=pandoc-crossref --filter=pandoc-citeproc -s -o TEST.pdf --pdf-engine=xelatex

However, after upgrading to 2.7.3, the call to pandoc fails with the message below

Error producing PDF.
! I can't find file `C:'.
<to be read again>
                   \/
<*> C:\/
        cygwin64/tmp/tex2pdf.-75944d1b7df3faf7/input.tex
(Press Enter to retry, or Control-D to exit)
Please type another input file name
! Emergency stop.
<to be read again>
                   \/
<*> C:\/
        cygwin64/tmp/tex2pdf.-75944d1b7df3faf7/input.tex
No pages of output.
Transcript written on C:\/cygwin64/tmp/tex2pdf.-75944d1b7df3faf7/texput.log.

Could you help to comment whether this is a bug that should be issued ? Many thanks !

[jiucenglou]

It would be great to get #1 where running under cygwin pandoc would use the posix directory naming rather than the transformed windows directory name.

So how can we determine at runtime that we're runinng under cygwin?

Can you try uname as suggested here ? :D
https://stackoverflow.com/questions/4250280/shell-script-detecting-running-in-cygwin

[jgm]

@jiucenglou - seems the problem might be the backslash in the tempdir: you report

[makePDF] temp dir:
C:\/cygwin64/tmp/tex2pdf.-7affd465d9e97ca8

We apply changePathSeparators but it's defined as

changePathSeparators = intercalate "/" . splitDirectories

which, it turns out, won't remove the backslash after C:, since the whole C:\ is the first element in the list returned by splitDirectories.

Would Windows be okay with C:/cygwin64/etc (without the backslash)? IF so, it would be easy to achieve this.

You can try working around the issue by setting the environment variable TMP to C:/cygwin64/tmp. It would be a helpful diagnostic to know what happens when you do this.

[jiucenglou]

(1) Windows is fine with either / or as a directory separator.

(2) However, setting the environment variable TEMP or TMP does not change anything. In fact, under Windows "cmd.exe" console, it can be shown that the new value of %TEMP% or %TMP% is in effect. Nevertheless, under Cygwin bash, $TEMP or $TMP always points to the value of "/tmp". :D

[jgm]

Windows is fine with either / or as a directory separator.

I'm aware of that, but I'm wondering whether the use of both in C:\/cygwin64/tmp is causing the problem? Maybe you can determine that by experimenting?

[jiucenglou]

I am not sure that I get your idea. What I have tried is to change the TEMP & TMP to C:/cygwin64/tmp and I get exactly the same complaint as before. In other words, the change of env vars is not perceived by Pandoc.

[jgm]

Right, I understand that. The temp directory is being set by pandoc; I'm just wondering if it will help things if we set it to that it doesn't have the backslash. I thought you might be able to do some other experiments to determine whether Windows or latex on Windows has problems with C:\/ at the beginning of a path.

[jiucenglou]

Thanks for your explanation. TextLive 2017 installed under Cygwin does not accept the C: nor C:/, as shown below:

C: does not work

$ which xelatex
/usr/local/texlive/2017/bin/x86_64-cygwin/xelatex

$ xelatex "C:\cygwin64\tmp\hehe\input.tex"
This is XeTeX, Version 3.14159265-2.6-0.99998 (TeX Live 2017) (preloaded format=xelatex)
 restricted \write18 enabled.
entering extended mode
! Undefined control sequence.
<*> C:\cygwin
             64\tmp\hehe\input.tex
?
! Emergency stop.
<*> C:\cygwin
             64\tmp\hehe\input.tex
No pages of output.
Transcript written on ?.

C:/ does not work either

$ xelatex "C:/cygwin64/tmp/hehe/input.tex"
This is XeTeX, Version 3.14159265-2.6-0.99998 (TeX Live 2017) (preloaded format=xelatex)
 restricted \write18 enabled.
entering extended mode
! I can't find file `C:/cygwin64/tmp/hehe/input.tex'.
<*> C:/cygwin64/tmp/hehe/input.tex

(Press Enter to retry, or Control-D to exit)
Please type another input file name:

Cygwin path works

$ xelatex /cygdrive/c/cygwin64/tmp/hehe/input.tex
This is XeTeX, Version 3.14159265-2.6-0.99998 (TeX Live 2017) (preloaded format=xelatex)
 restricted \write18 enabled.
entering extended mode
(/cygdrive/c/cygwin64/tmp/hehe/input.tex
LaTeX2e <2017-04-15>
Babel <3.10> and hyphenation patterns for 84 language(s) loaded.
(/usr/local/texlive/2017/texmf-dist/tex/latex/base/article.cls
Document Class: article 2014/09/29 v1.4h Standard LaTeX document class
(/usr/local/texlive/2017/texmf-dist/tex/latex/base/size12.clo))

Posix path also works

$ xelatex /tmp/hehe/input.tex
This is XeTeX, Version 3.14159265-2.6-0.99998 (TeX Live 2017) (preloaded format=xelatex)
 restricted \write18 enabled.
entering extended mode
(/tmp/hehe/input.tex
LaTeX2e <2017-04-15>
Babel <3.10> and hyphenation patterns for 84 language(s) loaded.
(/usr/local/texlive/2017/texmf-dist/tex/latex/base/article.cls
Document Class: article 2014/09/29 v1.4h Standard LaTeX document class
(/usr/local/texlive/2017/texmf-dist/tex/latex/base/size12.clo))

[jiucenglou]

It should be added that the testing above is made with TexLive for Cygwin instead of TexLive for Windows.

https://www.tug.org/texlive/quickinstall.html
TeX Live - Quick install 
If you don't want to bother reading the full install documentation and just want to install everything in TeX Live, on a Unix-like system, a minimal recipe follows.

Special considerations for Windows are described separately.

Cygwin users can follow the instructions for Unix-like systems, but make sure you have the Cygwin prerequisites before beginning the installation. 

[jgm]

Question: is it really worth the effort to support cygwin? Hasn't this been obsoleted now by the new windows subsystem for linux?

[jiucenglou]

I can only comment for myself: (1) the workflow of (previous) Pandoc and TexLive on Cygwin works great; (2) Cygwin is at least as useful as WSL1 (no idea about WSL2). If the effort for compatibility is not worthwhile, I can stay with previous Pandoc for now and learn other workflow when necessary. Many thanks for your time.

[jiucenglou]

Question: is it really worth the effort to support cygwin? Hasn't this been obsoleted now by the new windows subsystem for linux?

I thought it might work if Posix path can be returned, but Haskell's getTemporaryDirectory or getEnv seems to always return with the prefix "C:cygwin64".... Therefore, it does look like that an ad hoc patch is needed, which may not be worthwhile.

$ export TMP=/Nonsense/tmpXX
$
$ ghcii.sh
GHCi, version 8.6.5: http://www.haskell.org/ghc/  :? for help
Prelude> import System.Directory
Prelude System.Directory> a <- getTemporaryDirectory
Prelude System.Directory> putStrLn a
C:\cygwin64\Nonsense\tmpXX\
Prelude System.Directory> import System.Environment
Prelude System.Directory System.Environment>  b <- getEnv "TMP"
Prelude System.Directory System.Environment> putStrLn b
C:\cygwin64\Nonsense\tmpXX
Prelude System.Directory System.Environment> :q
Leaving GHCi.
$

[jiucenglou]

Would it be feasible to provide a switch to allow previous behavior, specifically, temp tex2pdf dir in the working directory ?

[jgm]

Are cygwin tempdir paths always going to start with cygwin64? If so, maybe we could just check for that and use the working directory when that's the temp dir.

[jiucenglou]

I would think Cygwin tempdir should always start with the actual installation path but I am not sure. Specifically, I really do not understand how getEnv is implemented: I would think when a process starts, the TMP env var in the entire env var section that the process load should be /tmp. There should be some manipulation performed by certain internal module. My knowledge of Haskell is too limited at the moment. Sorry about this.

[jgm]

What does

uname -s

say on your system? Maybe we could just check that as you suggested.

[jiucenglou]

Cygwin On Windows shows the following :D

$ uname -o
Cygwin

$ uname -s
CYGWIN_NT-10.0

[jgm]

It should be fairly easy to run uname -o and check the output.
If we get an error, we'll assume it's not cygwin.

[jgm]

I've pushed a change that should fix this.
Please test when you have a chance.

[jgm]

I had to revert the fix.

[jgm]

Is there an environment variable or something we can check to see if we're in a cygwin environment?

[austinjp]

Hi @jgm

Just checking if this is fixed in upcoming releases. I've installed Pandoc 2.8 from this Appveyor artifact but unfortunately when running /cygdrive/c/Program\ Files/pandoc/pandoc.exe input.md -o output.pdf --pdf-engine xelatex I get the following error:

Error producing PDF.
! I can't find file `C:'.
<to be read again>
                   \/
<*> C:\/
        cygwin/tmp/tex2pdf.-7856726bf9e44e68/input.tex
(Press Enter to retry, or Control-D to exit)
Please type another input file name
! Emergency stop.
<to be read again>
                   \/
<*> C:\/
        cygwin/tmp/tex2pdf.-7856726bf9e44e68/input.tex
No pages of output.
Transcript written on C:\/cygwin/tmp/tex2pdf.-7856726bf9e44e68/texput.log.

Setting TMP, TEMP and TEXMFOUTPUT to /tmp don't change the situation.

I installed Tex Live from tug.org manually inside Cygwin, and copied /usr/local/texlive/2019/bin/x86_64-cygwin/xelatex to /usr/bin/xelatex.exe which Pandoc seems to find and use.

Copying /usr/local/texlive/2019/bin/x86_64-cygwin/pdftex.exe to /usr/bin/pdflatex.exe and running pandoc input.md -o output.pdf results in the same error.

This sort of scenario might be uncommon, but it's an unfortunate pain for some. I have a Windows PC in my workplace, and Cygwin is my only option to run a *nix style workflow. Windows Subsystem for Linux is disabled and I can't change that. I can get by with markdown -> docx -> Word -> save as PDF, of course.

Many thanks for all your work on Pandoc.

[jgm]

Yes, the fix should be in the dev version 2.8 that you tested with from AppVeyor.

@jiucenglou can you also test with the artifact from AppVeyor to see if you run into the same problem, since you're the one who originally reported this? If the fix doesn't work, we should reopen this issue. (I'm sorry, I don't really have a windows/cygwin setup to test on.)

@austinjp what does uname -o report on your system?

[jiucenglou]

Yes, the fix should be in the dev version 2.8 that you tested with from AppVeyor.

@jiucenglou can you also test with the artifact from AppVeyor to see if you run into the same problem, since you're the one who originally reported this? If the fix doesn't work, we should reopen this issue. (I'm sorry, I don't really have a windows/cygwin setup to test on.)

@austinjp what does uname -o report on your system?

Sorry for not responding promptly to your previous fix. I will try to test this weekend and report. Many thanks for your kind efforts to fix !

[austinjp]

@austinjp what does uname -o report on your system?

@jgm Just this:

$ uname -o
Cygwin

[jgm]

I think I see the problem. I need to strip newlines from the output of uname before comparing with "Cygwin". I have pushed a fix. If someone could compile from source, or test with the nightly tomorrow, that would be great.

[austinjp]

I think I see the problem. I need to strip newlines from the output of uname before comparing with "Cygwin". I have pushed a fix. If someone could compile from source, or test with the nightly tomorrow, that would be great.

I'll look at this tomorrow @jgm when I'm back at my work PC.

[austinjp]

Hi @jgm

Artifact downloaded from here works perfectly. Many thanks!

FYI, I'm calling pandoc on Cygwin like so:

pandoc \
    -o out.pdf \
    --data-dir . \
    --number-sections \
    --pdf-engine xelatex \
    --top-level-division=chapter \
    input.docx

No error messages, and the resulting PDF is as expected.

[jiucenglou]

I think I see the problem. I need to strip newlines from the output of uname before comparing with "Cygwin". I have pushed a fix. If someone could compile from source, or test with the nightly tomorrow, that would be great.

The fix works great. Tested after compiling using stack. :) Many thanks !

[cc2x]

I feel I let the community down by not following through and testing the 2.7.3 release that first incorporated the patch for this issue. Since @jgm's patch that I locally built worked, I have been using that binary in addition to my workaround (#2 referenced above). The 2.7.3 version appears to have additional code on the PDF generation where even my workaround no longer works.

I'm pleased that there are others using pandoc in a cywin environment and that this issue is fully resolved starting with version 2.8.