Stack: commitBuffer: invalid argument (invalid character)

[ThibaudDauce]

Hi,

When I'm running stack build on my Debian 8 (for a Hakyll project), I receive this error after a few packages installed:

regex-tdfa-1.2.0: configure
zlib-0.5.4.2: download
regex-tdfa-1.2.0: build
transformers-base-0.4.4: configure
transformers-base-0.4.4: build
texmath-0.8.2: configure
texmath-0.8.2: build
zlib-0.5.4.2: configure
zlib-0.5.4.2: build
zlib-0.5.4.2: install
Progress: 4/27
--  While building package regex-tdfa-1.2.0 using:
      /home/thibaud/.stack/programs/x86_64-linux/ghc-7.8.4/bin/runhaskell -package=Cabal-1.18.1.5 -clear-package-db -global-package-db -package-db=/home/thibaud/.stack/snapshots/x86_64-linux/lts-2.15/7.8.4/pkgdb/ /tmp/stack15788/Setup.hs --builddir=.stack-work/dist/x86_64-linux/Cabal-1.18.1.5/ build --ghc-options -hpcdir .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/hpc/.hpc/ -ddump-hi -ddump-to-file
    Process exited with code: ExitFailure (-9)
    Logs have been written to: /var/www/.stack-work/logs/regex-tdfa-1.2.0.log

    Configuring regex-tdfa-1.2.0...
    Building regex-tdfa-1.2.0...
    Preprocessing library regex-tdfa-1.2.0...
    [ 1 of 23] Compiling Text.Regex.TDFA.NewDFA.Uncons ( Text/Regex/TDFA/NewDFA/Uncons.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/NewDFA/Uncons.o )
    [ 2 of 23] Compiling Text.Regex.TDFA.IntArrTrieSet ( Text/Regex/TDFA/IntArrTrieSet.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/IntArrTrieSet.o )
    [ 3 of 23] Compiling Paths_regex_tdfa ( .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/autogen/Paths_regex_tdfa.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Paths_regex_tdfa.o )
    [ 4 of 23] Compiling Data.IntSet.EnumSet2 ( Data/IntSet/EnumSet2.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Data/IntSet/EnumSet2.o )
    [ 5 of 23] Compiling Data.IntMap.EnumMap2 ( Data/IntMap/EnumMap2.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Data/IntMap/EnumMap2.o )
    [ 6 of 23] Compiling Data.IntMap.CharMap2 ( Data/IntMap/CharMap2.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Data/IntMap/CharMap2.o )
    [ 7 of 23] Compiling Text.Regex.TDFA.Common ( Text/Regex/TDFA/Common.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/Common.o )
    [ 8 of 23] Compiling Text.Regex.TDFA.Pattern ( Text/Regex/TDFA/Pattern.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/Pattern.o )
    [ 9 of 23] Compiling Text.Regex.TDFA.ReadRegex ( Text/Regex/TDFA/ReadRegex.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/ReadRegex.o )
    [10 of 23] Compiling Text.Regex.TDFA.CorePattern ( Text/Regex/TDFA/CorePattern.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/CorePattern.o )
    [11 of 23] Compiling Text.Regex.TDFA.NewDFA.MakeTest ( Text/Regex/TDFA/NewDFA/MakeTest.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/NewDFA/MakeTest.o )
    [12 of 23] Compiling Text.Regex.TDFA.NewDFA.Tester ( Text/Regex/TDFA/NewDFA/Tester.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/NewDFA/Tester.o )
    [13 of 23] Compiling Text.Regex.TDFA.NewDFA.Engine_FA ( Text/Regex/TDFA/NewDFA/Engine_FA.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/NewDFA/Engine_FA.o )
    [14 of 23] Compiling Text.Regex.TDFA.NewDFA.Engine_NC ( Text/Regex/TDFA/NewDFA/Engine_NC.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/NewDFA/Engine_NC.o )
    [15 of 23] Compiling Text.Regex.TDFA.NewDFA.Engine_NC_FA ( Text/Regex/TDFA/NewDFA/Engine_NC_FA.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/NewDFA/Engine_NC_FA.o )
    [16 of 23] Compiling Text.Regex.TDFA.NewDFA.Engine ( Text/Regex/TDFA/NewDFA/Engine.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/Regex/TDFA/NewDFA/Engine.o )


--  While building package texmath-0.8.2 using:
      /home/thibaud/.stack/programs/x86_64-linux/ghc-7.8.4/bin/runhaskell -package=Cabal-1.18.1.5 -clear-package-db -global-package-db -package-db=/home/thibaud/.stack/snapshots/x86_64-linux/lts-2.15/7.8.4/pkgdb/ /tmp/stack15788/Setup.hs --builddir=.stack-work/dist/x86_64-linux/Cabal-1.18.1.5/ build --ghc-options -hpcdir .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/hpc/.hpc/ -ddump-hi -ddump-to-file
    Process exited with code: ExitFailure (-9)
    Logs have been written to: /var/www/.stack-work/logs/texmath-0.8.2.log

    Configuring texmath-0.8.2...
    Building texmath-0.8.2...
    Preprocessing library texmath-0.8.2...
    [ 1 of 18] Compiling Text.TeXMath.Unicode.ToASCII ( src/Text/TeXMath/Unicode/ToASCII.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/TeXMath/Unicode/ToASCII.o )
    [ 2 of 18] Compiling Text.TeXMath.TeX ( src/Text/TeXMath/TeX.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/TeXMath/TeX.o )
    [ 3 of 18] Compiling Text.TeXMath.Compat ( src/Text/TeXMath/Compat.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/TeXMath/Compat.o )
    [ 4 of 18] Compiling Text.TeXMath.Readers.MathML.EntityMap ( src/Text/TeXMath/Readers/MathML/EntityMap.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/TeXMath/Readers/MathML/EntityMap.o )
    [ 5 of 18] Compiling Text.TeXMath.Readers.TeX.Macros ( src/Text/TeXMath/Readers/TeX/Macros.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/TeXMath/Readers/TeX/Macros.o )
    [ 6 of 18] Compiling Text.TeXMath.Types ( src/Text/TeXMath/Types.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/TeXMath/Types.o )
    [ 7 of 18] Compiling Text.TeXMath.Shared ( src/Text/TeXMath/Shared.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/TeXMath/Shared.o )
    [ 8 of 18] Compiling Text.TeXMath.Readers.MathML.MMLDict ( src/Text/TeXMath/Readers/MathML/MMLDict.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/TeXMath/Readers/MathML/MMLDict.o )
    [ 9 of 18] Compiling Text.TeXMath.Unicode.ToUnicode ( src/Text/TeXMath/Unicode/ToUnicode.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/TeXMath/Unicode/ToUnicode.o )
    [10 of 18] Compiling Text.TeXMath.Unicode.ToTeX ( src/Text/TeXMath/Unicode/ToTeX.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Text/TeXMath/Unicode/ToTeX.o )


--  While building package transformers-base-0.4.4 using:
      /home/thibaud/.stack/programs/x86_64-linux/ghc-7.8.4/bin/runhaskell -package=Cabal-1.18.1.5 -clear-package-db -global-package-db -package-db=/home/thibaud/.stack/snapshots/x86_64-linux/lts-2.15/7.8.4/pkgdb/ /tmp/stack15788/Setup.hs --builddir=.stack-work/dist/x86_64-linux/Cabal-1.18.1.5/ build --ghc-options -hpcdir .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/hpc/.hpc/ -ddump-hi -ddump-to-file
    Process exited with code: ExitFailure 1
    Logs have been written to: /var/www/.stack-work/logs/transformers-base-0.4.4.log

    Configuring transformers-base-0.4.4...
    Building transformers-base-0.4.4...
    Preprocessing library transformers-base-0.4.4...
    [1 of 1] Compiling Control.Monad.Base ( src/Control/Monad/Base.hs, .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/Control/Monad/Base.o )
    .stack-work/dist/x86_64-linux/Cabal-1.18.1.5/build/src/Control/Monad/Base.dump-hi: commitBuffer: invalid argument (invalid character)

It could happens at 3 or 4 / n process. When I restart stack build everythink is working then fail again after a few packages installed.

Do you have an idea?
Thanks

[borsboom]

What does the locale command on your system output? And can you try running export LANG=en_US.UTF-8 before running stack?

[thelmstedt]

I had this same issue and this advice fixed it for me. Here's what locale was set to:

LANG=C
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=

[snoyberg]

I've just sent a GHC patch to address this:

https://phabricator.haskell.org/D1151

@borsboom In the short term, should we set the LANG env var in stack before calling Setup.hs build?

[borsboom]

Given we're doing the equivalent in Windows, probably makes sense. It should probably append the encoding if there isn't already one, or replace it if there already is.

[borsboom]

Hmm, actually maybe always using C.UTF-8 for GHC output would be better (and easier).

[snoyberg]

Perhaps what we should do is:

• Have GHC provide output in UTF-8
• When logging to a file, we get UTF-8 encoded files
• When logging to the console, first decode that data to Text, and then use standard textual output to get the correct character encoding. We can use transliteration for unhandled characters, as I just did in https://phabricator.haskell.org/D1153

[borsboom]

Fix that sets LC_ALL for GHC pushed to master.

[MatthiasKauer]

I just want to add to this and the discussion over on the mailing list: https://mail.haskell.org/pipermail/haskell-cafe/2015-October/121847.html

Setting

export LANG=en_US.UTF-8

was not sufficient on my Ubuntu 12.04 server for compiling gitit. It used lts
I also tried upgrading

resolver: lts-3.1

to lts-3.13 since you mentioned that you fixed things upstream but that didn't seem to help either. Using

export LC_ALL=C.UTF-8

worked however.

I also used stack clean a number of times in between although I have no idea if that improved things in any way.

The question is now: How would I upgrade to that new GHC version you mention cleanly? Or should I wait for the package managers of the individual repos? Or do I misunderstand and GHC version isn't tied to that resolver parameter?

Another question: Does stack help with static linking in any way? I really would've liked to build gitit on my notebook and then upload the binary, but I just couldn't make it work with cabal.

[borsboom]

@MatthiasKauer: what is the output of the locale command before you
change the environment variables? Stack modifies locale environment
variables itself before invoking ghc, but it tries to be smart about only
doing so when needed and staying as close to your current settings as
possible. It's possible that you've hit a corner case that it's not
handling correctly. Also, what is the output of stack --version?

Regarding upgrading GHC, there is not currently a released version of GHC
that includes the upstream fixes, and anyway Stack will need to be modified
slightly to support enabling those when they do get released. But Stack
does its best to work around any issues with the current version.

Regarding static linking, that's being tracked by #1032.

[MatthiasKauer]

matthias@ugvps:~$ stack --version
Version 0.1.6.0, Git revision e22271f5ce9afa2cb5be3bad9cafa392c623f85c (2313 commits) x86_64

I'm running on a vserver, so locale may look odd.

matthias@ugvps:~/dvl/gitit$ locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=en_SG.utf8
LANGUAGE=
LC_CTYPE="en_SG.utf8"
LC_NUMERIC="en_SG.utf8"
LC_TIME="en_SG.utf8"
LC_COLLATE="en_SG.utf8"
LC_MONETARY="en_SG.utf8"
LC_MESSAGES="en_SG.utf8"
LC_PAPER="en_SG.utf8"
LC_NAME="en_SG.utf8"
LC_ADDRESS="en_SG.utf8"
LC_TELEPHONE="en_SG.utf8"
LC_MEASUREMENT="en_SG.utf8"
LC_IDENTIFICATION="en_SG.utf8"
LC_ALL=

Actually, this appears to be the same locale that I also have on my notebook (where the build succeeded, although not through stack but with cabal directly).
Is there a chance that locale is executed locally and not through ssh?

Will it help if I run further checks?

[borsboom]

@MatthiasKauer: Which Linux distribution are you using? And what are the values of the the LANG, LANGUAGE, and LC_ALL environment variables.

[borsboom]

Just to add: I've tried this on Ubuntu, setting export LANG=en_SG.utf8 (which produced identical locale output to yours) and successfully been able to see error messages that would normally cause this type of error. If I don't actually have the en_SG.utf8 locale installed, the special characters are transliterated to ASCII, and after I did install the locale, the special characters were shown correctly.

Does you project's source code contain any Unicode characters? Would it be possible for you to share it to aid with reproduction?

Also, have you checked that the en_SG.utf8 locale is installed? Usually those first three messages in the locale output don't appear if the locale is there. You can see a list of installed locales using locale -a. On Ubuntu, you can install the locale using sudo apt-get install language-pack-en (not sure about other distros).

[MatthiasKauer]

Thanks for looking into this!

This is a virtual server running Ubuntu 12.04. It's under OpenVZ and I'm not sure whether the hosting company would let me upgrade by now. Last time I asked they warned me about kernel issues....
Anyhow.

matthias@ugvps:~/dvl/gitit$ echo $LANG
en_SG.utf8
matthias@ugvps:~/dvl/gitit$ echo $LANGUAGE

matthias@ugvps:~/dvl/gitit$ echo $LC_ALL

I'm not quite sure why the locale is set that way there as I said. I'm in Singapore but I most probably wouldn't have bothered to tinker with that on a vserver.

Your other comment:

matthias@ugvps:~/dvl/gitit$ locale -a
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_COLLATE to default locale: No such file or directory
C
C.UTF-8
POSIX

So I'm running a locale that I don't have installed, am I?
I still wonder whether the terminal sets that locale b/c my notebook has it.

The project in question is not mine. I was trying to compile https://github.com/jgm/gitit for deployment. I believe however, that the error came from a dependency. Here is the output snippet that I have stored.

    Configuring transformers-base-0.4.4...
    Building transformers-base-0.4.4...
    Preprocessing library transformers-base-0.4.4...
    [1 of 1] Compiling Control.Monad.Base ( src/Control/Monad/Base.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.4.0/build/Control/Monad/Base.o )

    src/Control/Monad/Base.hs:22:1: Warning:
        The import of `Data.Monoid' is redundant
          except perhaps to import instances from `Data.Monoid'
        To import instances alone, use: import Data.Monoid()

    src/Control/Monad/Base.hs:24:1: Warning:
        The import of `Control.Applicative' is redundant
          except perhaps to import instances from `Control.Applicative'
        To import instances alone, use: import Control.Applicative()
    .stack-work/dist/x86_64-linux/Cabal-1.22.4.0/build/src/Control/Monad/Base.dump-hi: commitBuffer: invalid argument (invalid character)

[borsboom]

Stack should work just fine on Ubuntu 12.04. Do you have root access to the machine? I wonder if running sudo locale-gen en_SG.utf8 makes a difference.

[MatthiasKauer]

That seems to make a difference too.
I ran stack clean, got the same error building (although somewhere else, probably b/c I didn't clean dependencies too), entered sudo locale-gen en_SG.utf8 and built successfully.
Hope this helps.

[domenkozar]

Since this is the first hit on google, I'll note how to fix this issue for Nix. Using custom shell.nix:

with (import <nixpkgs> {});

haskell.lib.buildStackProject {
  name = "myenv";

  buildInputs = [
    zlib
    ...
    # avoid on circleci: commitBuffer: invalid argument (invalid character)
    glibcLocales
  ];

  # avoid on circleci: commitBuffer: invalid argument (invalid character)
  LANG = "en_US.utf8";
}

As nix-shell: nix-shell -p glibcLocales --run "LANG=en_US.utf8 CMD"

[qrilka]

@domenkozar probably you have some good ideas how to add that LANG into https://github.com/commercialhaskell/stack/blob/master/src/Stack/Nix.hs#L83 - the problem applies also to #4095

[domenkozar]

Hmm, since LANG=C is a default, we might as well just change that? Workaround is STACK_IN_NIX_EXTRA_ARGS: LANG=en_US.UTF-8.

[qrilka]

@domenkozar STACK_IN_NIX_EXTRA_ARGS is about a different thing (passing extra flags) but adding ,"LANG=en_US.UTF-8;" beside it looks to be a good enough workaround, I thought maybe there is some neater way to pass this locale but probably it should be ok so no extra nix file will be required for Unicode output. Thanks

[mihaimaruseac]

I think #4294 closes this

[lf-]

This is making stack unusable for me today. Version: 2.1.3.1

[Delanii]

Same issue here, version 2.5.1 x86_64 hpack-0.34.2. Cant build pandoc-citeproc library required for pandoc-crossref.