Data.table: valgrind uninitialized variables

[mattdowle]

With R-devel compiled with --with-valgrind-instrumentation=1, I see the following. The runtime is 62 mins, so not too bad.

The 9 test fails and I've noted before the reasons in CRAN_Release.cmd.

Some are in R or the libraries it uses and are known to be ignored. I've emailed this to CRAN to ask.

valgrind output

$ Rdevel -d "valgrind --tool=memcheck --leak-check=full --track-origins=yes --show-leak-kinds=definite"
==15054== Memcheck, a memory error detector
==15054== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==15054== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==15054== Command: /home/mdowle/build/R-devel/bin/exec/R --vanilla
==15054== 
==15054== Conditional jump or move depends on uninitialised value(s)
==15054==    at 0x6E901A7: __wcsnlen_avx2 (strlen-avx2.S:261)
==15054==    by 0x6DC4CAA: wcsrtombs (wcsrtombs.c:104)
==15054==    by 0x6D4C740: wcstombs (wcstombs.c:34)
==15054==    by 0x39AD71: tre_parse_bracket_items (tre-parse.c:336)
==15054==    by 0x39B340: tre_parse_bracket (tre-parse.c:453)
==15054==    by 0x39D1E6: tre_parse (tre-parse.c:1380)
==15054==    by 0x392247: tre_compile (tre-compile.c:1920)
==15054==    by 0x38C11E: tre_regcompb (regcomp.c:150)
==15054==    by 0x26BFE4: do_gsub (grep.c:1828)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==  Uninitialised value was created by a stack allocation
==15054==    at 0x39AA27: tre_parse_bracket_items (tre-parse.c:266)
==15054== 
==15054== Conditional jump or move depends on uninitialised value(s)
==15054==    at 0x6D367F3: internal_utf8_loop (loop.c:298)
==15054==    by 0x6D367F3: __gconv_transform_internal_utf8 (skeleton.c:609)
==15054==    by 0x6DC4CDD: wcsrtombs (wcsrtombs.c:110)
==15054==    by 0x6D4C740: wcstombs (wcstombs.c:34)
==15054==    by 0x39AD71: tre_parse_bracket_items (tre-parse.c:336)
==15054==    by 0x39B340: tre_parse_bracket (tre-parse.c:453)
==15054==    by 0x39D1E6: tre_parse (tre-parse.c:1380)
==15054==    by 0x392247: tre_compile (tre-compile.c:1920)
==15054==    by 0x38C11E: tre_regcompb (regcomp.c:150)
==15054==    by 0x26BFE4: do_gsub (grep.c:1828)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==  Uninitialised value was created by a stack allocation
==15054==    at 0x39AA27: tre_parse_bracket_items (tre-parse.c:266)
==15054== 
==15054== Conditional jump or move depends on uninitialised value(s)
==15054==    at 0x6D367FC: internal_utf8_loop (loop.c:303)
==15054==    by 0x6D367FC: __gconv_transform_internal_utf8 (skeleton.c:609)
==15054==    by 0x6DC4CDD: wcsrtombs (wcsrtombs.c:110)
==15054==    by 0x6D4C740: wcstombs (wcstombs.c:34)
==15054==    by 0x39AD71: tre_parse_bracket_items (tre-parse.c:336)
==15054==    by 0x39B340: tre_parse_bracket (tre-parse.c:453)
==15054==    by 0x39D1E6: tre_parse (tre-parse.c:1380)
==15054==    by 0x392247: tre_compile (tre-compile.c:1920)
==15054==    by 0x38C11E: tre_regcompb (regcomp.c:150)
==15054==    by 0x26BFE4: do_gsub (grep.c:1828)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==  Uninitialised value was created by a stack allocation
==15054==    at 0x39AA27: tre_parse_bracket_items (tre-parse.c:266)
==15054== 
==15054== Conditional jump or move depends on uninitialised value(s)
==15054==    at 0x6D36843: internal_utf8_loop (loop.c:298)
==15054==    by 0x6D36843: __gconv_transform_internal_utf8 (skeleton.c:609)
==15054==    by 0x6DC4CDD: wcsrtombs (wcsrtombs.c:110)
==15054==    by 0x6D4C740: wcstombs (wcstombs.c:34)
==15054==    by 0x39AD71: tre_parse_bracket_items (tre-parse.c:336)
==15054==    by 0x39B340: tre_parse_bracket (tre-parse.c:453)
==15054==    by 0x39D1E6: tre_parse (tre-parse.c:1380)
==15054==    by 0x392247: tre_compile (tre-compile.c:1920)
==15054==    by 0x38C11E: tre_regcompb (regcomp.c:150)
==15054==    by 0x26BFE4: do_gsub (grep.c:1828)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==  Uninitialised value was created by a stack allocation
==15054==    at 0x39AA27: tre_parse_bracket_items (tre-parse.c:266)
==15054== 

R Under development (unstable) (2018-05-02 r74682) -- "Unsuffered Consequences"
Copyright (C) 2018 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> print(Sys.time()); require(data.table); print(Sys.time()); started.at<-proc.time(); try(test.data.table()); print(Sys.time()); print(timetaken(started.at))
[1] "2018-05-07 13:15:46 PDT"
Loading required package: data.table
==15054== Conditional jump or move depends on uninitialised value(s)
==15054==    at 0x6E901A7: __wcsnlen_avx2 (strlen-avx2.S:261)
==15054==    by 0x6DC4CAA: wcsrtombs (wcsrtombs.c:104)
==15054==    by 0x6D4C740: wcstombs (wcstombs.c:34)
==15054==    by 0x1AC6D7: do_makenames (character.c:924)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23968D: do_set (eval.c:2778)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054==    by 0x235BE3: Rf_applyClosure (eval.c:1692)
==15054== 
==15054== Conditional jump or move depends on uninitialised value(s)
==15054==    at 0x6D367F3: internal_utf8_loop (loop.c:298)
==15054==    by 0x6D367F3: __gconv_transform_internal_utf8 (skeleton.c:609)
==15054==    by 0x6DC4CDD: wcsrtombs (wcsrtombs.c:110)
==15054==    by 0x6D4C740: wcstombs (wcstombs.c:34)
==15054==    by 0x1AC6D7: do_makenames (character.c:924)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23968D: do_set (eval.c:2778)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054==    by 0x235BE3: Rf_applyClosure (eval.c:1692)
==15054== 
==15054== Conditional jump or move depends on uninitialised value(s)
==15054==    at 0x6D367FC: internal_utf8_loop (loop.c:303)
==15054==    by 0x6D367FC: __gconv_transform_internal_utf8 (skeleton.c:609)
==15054==    by 0x6DC4CDD: wcsrtombs (wcsrtombs.c:110)
==15054==    by 0x6D4C740: wcstombs (wcstombs.c:34)
==15054==    by 0x1AC6D7: do_makenames (character.c:924)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23968D: do_set (eval.c:2778)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054==    by 0x235BE3: Rf_applyClosure (eval.c:1692)
==15054== 
==15054== Conditional jump or move depends on uninitialised value(s)
==15054==    at 0x6D36843: internal_utf8_loop (loop.c:298)
==15054==    by 0x6D36843: __gconv_transform_internal_utf8 (skeleton.c:609)
==15054==    by 0x6DC4CDD: wcsrtombs (wcsrtombs.c:110)
==15054==    by 0x6D4C740: wcstombs (wcstombs.c:34)
==15054==    by 0x1AC6D7: do_makenames (character.c:924)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23968D: do_set (eval.c:2778)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054==    by 0x235BE3: Rf_applyClosure (eval.c:1692)
==15054== 
==15054== Conditional jump or move depends on uninitialised value(s)
==15054==    at 0x6E9001C: __wcsnlen_avx2 (strlen-avx2.S:103)
==15054==    by 0x6DC4CAA: wcsrtombs (wcsrtombs.c:104)
==15054==    by 0x6D4C740: wcstombs (wcstombs.c:34)
==15054==    by 0x1AC6D7: do_makenames (character.c:924)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23968D: do_set (eval.c:2778)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054==    by 0x235BE3: Rf_applyClosure (eval.c:1692)
==15054== 
==15054== Invalid read of size 32
==15054==    at 0x6E8FFC1: __wcsnlen_avx2 (strlen-avx2.S:62)
==15054==    by 0x6DC4CAA: wcsrtombs (wcsrtombs.c:104)
==15054==    by 0x6D4C740: wcstombs (wcstombs.c:34)
==15054==    by 0x1AC6D7: do_makenames (character.c:924)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23968D: do_set (eval.c:2778)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054==    by 0x235BE3: Rf_applyClosure (eval.c:1692)
==15054==  Address 0xa8f4c50 is 0 bytes inside a block of size 8 alloc'd
==15054==    at 0x4C31B25: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==15054==    by 0x28F148: R_chk_calloc (memory.c:3279)
==15054==    by 0x1AC61F: do_makenames (character.c:909)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23968D: do_set (eval.c:2778)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054==    by 0x235BE3: Rf_applyClosure (eval.c:1692)
==15054==    by 0x245A27: bcEval (eval.c:6740)
==15054== 
data.table 1.11.1 IN DEVELOPMENT built 2018-05-07 20:12:59.400 UTC; mdowle
**********
This installation of data.table has not detected OpenMP support. It should still work but in single-threaded mode. If this is a Mac, please ensure you are using R>=3.4.0 and have installed the MacOS binary package from CRAN: see ?install.packages, the 'type=' argument and the 'Binary packages' section. If you compiled from source, please reinstall and precisely follow the installation instructions on the data.table homepage. This warning message should not occur on Windows or Linux. If it does and you've followed the installation instructions on the data.table homepage, please file a GitHub issue.
**********
  The fastest way to learn (by data.table authors): https://www.datacamp.com/courses/data-analysis-the-data-table-way
  Documentation: ?data.table, example(data.table) and browseVignettes("data.table")
  Release notes, videos and slides: http://r-datatable.com
[1] "2018-05-07 13:18:25 PDT"
Running /home/mdowle/build/R-devel/library/data.table/tests/tests.Rraw 

**** Suggested package bit64 is not installed. Tests using it will be skipped.


**** Suggested package xts is not installed. Tests using it will be skipped.


**** Suggested package nanotime is not installed. Tests using it will be skipped.

Running test id 83.1      ==15054== Conditional jump or move depends on uninitialised value(s)
==15054==    at 0x6E901C7: __wcsnlen_avx2 (strlen-avx2.S:275)
==15054==    by 0x6DC4CAA: wcsrtombs (wcsrtombs.c:104)
==15054==    by 0x6D4C740: wcstombs (wcstombs.c:34)
==15054==    by 0x1AC6D7: do_makenames (character.c:924)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23968D: do_set (eval.c:2778)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054==    by 0x235BE3: Rf_applyClosure (eval.c:1692)
==15054== 
Running test id 598      ==15054== Conditional jump or move depends on uninitialised value(s)
==15054==    at 0x6E901E7: __wcsnlen_avx2 (strlen-avx2.S:290)
==15054==    by 0x6DC4CAA: wcsrtombs (wcsrtombs.c:104)
==15054==    by 0x6D4C740: wcstombs (wcstombs.c:34)
==15054==    by 0x1AC6D7: do_makenames (character.c:924)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23968D: do_set (eval.c:2778)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054==    by 0x235BE3: Rf_applyClosure (eval.c:1692)
==15054== 
Running test id 648      Test 648 ran without errors but failed check that x equals y:
> x = any(is.na(ans1$V1)) && !any(is.nan(ans1$V1)) 
First 6 of 1 (type 'logical'): [1] FALSE
> y = TRUE 
First 6 of 1 (type 'logical'): [1] TRUE
1 element mismatch
Running test id 819      Test 819 ran without errors but failed check that x equals y:
> x = system.time(X[Y, allow.cartesian = TRUE])["user.self"] < 10 
First 6 of 1 (type 'logical'): user.self 
    FALSE 
> y = TRUE 
First 6 of 1 (type 'logical'): [1] TRUE
1 element mismatch
Running test id 983      ==15054== Conditional jump or move depends on uninitialised value(s)
==15054==    at 0x6E90207: __wcsnlen_avx2 (strlen-avx2.S:305)
==15054==    by 0x6DC4CAA: wcsrtombs (wcsrtombs.c:104)
==15054==    by 0x6D4C740: wcstombs (wcstombs.c:34)
==15054==    by 0x1AC6D7: do_makenames (character.c:924)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23968D: do_set (eval.c:2778)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054==    by 0x235BE3: Rf_applyClosure (eval.c:1692)
==15054== 
Running test id 1018      Test 1018 ran without errors but failed check that x equals y:
> x = identical(fread("1.46761e-313\n"), data.table(V1 = 1e-313)) 
First 6 of 1 (type 'logical'): [1] FALSE
> y = TRUE 
First 6 of 1 (type 'logical'): [1] TRUE
1 element mismatch
Running test id 1262      Test 1262 ran without errors but failed check that x equals y:
> x = length(unique(c(1.23456789012346, 1.23456789012346, 1.23456789012346,      1.23456789012346))) 
First 6 of 1 (type 'integer'): [1] 3
> y = 2L 
First 6 of 1 (type 'integer'): [1] 2
Mean relative difference: 0.3333333
Running test id 1375.1      ==15054== Conditional jump or move depends on uninitialised value(s)
==15054==    at 0x6E901A7: __wcsnlen_avx2 (strlen-avx2.S:261)
==15054==    by 0x6DC4CAA: wcsrtombs (wcsrtombs.c:104)
==15054==    by 0x6D4C740: wcstombs (wcstombs.c:34)
==15054==    by 0x1ACCD0: do_tolower (character.c:1018)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054==    by 0x235BE3: Rf_applyClosure (eval.c:1692)
==15054==    by 0x233FF1: Rf_eval (eval.c:747)
==15054==    by 0x23A909: do_eval (eval.c:3157)
==15054==  Uninitialised value was created by a heap allocation
==15054==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==15054==    by 0x291DA0: R_AllocStringBuffer (memory.c:4096)
==15054==    by 0x1A9C6A: do_substr (character.c:344)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054==    by 0x235BE3: Rf_applyClosure (eval.c:1692)
==15054==    by 0x245A27: bcEval (eval.c:6740)
==15054==    by 0x233821: Rf_eval (eval.c:624)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054== 
==15054== Conditional jump or move depends on uninitialised value(s)
==15054==    at 0x6D367F3: internal_utf8_loop (loop.c:298)
==15054==    by 0x6D367F3: __gconv_transform_internal_utf8 (skeleton.c:609)
==15054==    by 0x6DC4CDD: wcsrtombs (wcsrtombs.c:110)
==15054==    by 0x6D4C740: wcstombs (wcstombs.c:34)
==15054==    by 0x1ACCD0: do_tolower (character.c:1018)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054==    by 0x235BE3: Rf_applyClosure (eval.c:1692)
==15054==    by 0x233FF1: Rf_eval (eval.c:747)
==15054==    by 0x23A909: do_eval (eval.c:3157)
==15054==  Uninitialised value was created by a heap allocation
==15054==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==15054==    by 0x291DA0: R_AllocStringBuffer (memory.c:4096)
==15054==    by 0x1A9C6A: do_substr (character.c:344)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054==    by 0x235BE3: Rf_applyClosure (eval.c:1692)
==15054==    by 0x245A27: bcEval (eval.c:6740)
==15054==    by 0x233821: Rf_eval (eval.c:624)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054== 
==15054== Conditional jump or move depends on uninitialised value(s)
==15054==    at 0x6D367FC: internal_utf8_loop (loop.c:303)
==15054==    by 0x6D367FC: __gconv_transform_internal_utf8 (skeleton.c:609)
==15054==    by 0x6DC4CDD: wcsrtombs (wcsrtombs.c:110)
==15054==    by 0x6D4C740: wcstombs (wcstombs.c:34)
==15054==    by 0x1ACCD0: do_tolower (character.c:1018)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054==    by 0x235BE3: Rf_applyClosure (eval.c:1692)
==15054==    by 0x233FF1: Rf_eval (eval.c:747)
==15054==    by 0x23A909: do_eval (eval.c:3157)
==15054==  Uninitialised value was created by a heap allocation
==15054==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==15054==    by 0x291DA0: R_AllocStringBuffer (memory.c:4096)
==15054==    by 0x1A9C6A: do_substr (character.c:344)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054==    by 0x235BE3: Rf_applyClosure (eval.c:1692)
==15054==    by 0x245A27: bcEval (eval.c:6740)
==15054==    by 0x233821: Rf_eval (eval.c:624)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054== 
==15054== Conditional jump or move depends on uninitialised value(s)
==15054==    at 0x6D36843: internal_utf8_loop (loop.c:298)
==15054==    by 0x6D36843: __gconv_transform_internal_utf8 (skeleton.c:609)
==15054==    by 0x6DC4CDD: wcsrtombs (wcsrtombs.c:110)
==15054==    by 0x6D4C740: wcstombs (wcstombs.c:34)
==15054==    by 0x1ACCD0: do_tolower (character.c:1018)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054==    by 0x235BE3: Rf_applyClosure (eval.c:1692)
==15054==    by 0x233FF1: Rf_eval (eval.c:747)
==15054==    by 0x23A909: do_eval (eval.c:3157)
==15054==  Uninitialised value was created by a heap allocation
==15054==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==15054==    by 0x291DA0: R_AllocStringBuffer (memory.c:4096)
==15054==    by 0x1A9C6A: do_substr (character.c:344)
==15054==    by 0x293399: do_internal (names.c:1371)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x23858E: do_begin (eval.c:2395)
==15054==    by 0x233C8B: Rf_eval (eval.c:699)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054==    by 0x235BE3: Rf_applyClosure (eval.c:1692)
==15054==    by 0x245A27: bcEval (eval.c:6740)
==15054==    by 0x233821: Rf_eval (eval.c:624)
==15054==    by 0x235EC5: R_execClosure (eval.c:1764)
==15054== 
Running test id 1729.4      Test 1729.4 didn't produce correct output:
Expected: <<V1\n9999999999.99\n9.9e-16\n9e-22\n0.9\n9\n9.1\n99.9\n1e-21\n1e+29>>
Observed: <<V1\n9999999999.99\n9.90000000000000e-16\n9.00000000000000e-22\n0.9\n9\n9.1\n99.9\n1.00000000000000e-21\n1e+29>>
Running test id 1729.8      Test 1729.8 ran without errors but failed check that x equals y:
> x = x 
First 6 of 505 (type 'character'): [1] "-6e-05"            "-123456789.123457" "-1000"            
[4] "-983"              "-966"              "-949"             
> y = y 
First 6 of 505 (type 'character'): [1] "-6e-05"            "-123456789.123457" "-1000"            
[4] "-983"              "-966"              "-949"             
18 string mismatches
    row            fwrite              write.csv
 1: 413            -7e-20  -7.00000000000000e-20
 2: 414            -7e-19  -7.00000000000000e-19
 3: 415            -7e-18  -7.00000000000000e-18
 4: 416            -7e-17  -7.00000000000000e-17
 5: 417            -7e-16  -7.00000000000000e-16
 6: 418            -7e-15  -7.00000000000000e-15
 7: 454             7e-20   7.00000000000000e-20
 8: 455             7e-19   7.00000000000000e-19
 9: 456             7e-18   7.00000000000000e-18
10: 457             7e-17   7.00000000000000e-17
11: 458             7e-16   7.00000000000000e-16
12: 459             7e-15   7.00000000000000e-15
13: 500  5.123456789e-290  5.12345678900000e-290
14: 501 -5.123456789e-290 -5.12345678900000e-290
15: 502  5.123456789e-307  5.12345678900000e-307
16: 503 -5.123456789e-307 -5.12345678900000e-307
17: 504  5.123456789e+307  5.12345678900000e+307
18: 505 -5.123456789e+307 -5.12345678900000e+307
Running test id 1729.9      Test 1729.9 ran without errors but failed check that x equals y:
> x = binary(DT[[1]]) 
First 6 of 8 (type 'character'): [1] "0 11111100101 111010011010000100010111101110000100 11110100 00001000"
[2] "1 11111100101 111010011010000100010111101110000100 11110100 00001000"
[3] "0 00000001001 110000010110110001011100010100100101 00110101 01110001"
[4] "0 11111110100 001000111010010100010110111010000010 11011001 10111101"
[5] "0 00000001010 000011011010011101101010100101111100 10111001 10101010"
[6] "0 11111110100 010111011111100101001110101100000011 01101011 10110000"
> y = c("0 11111100101 111010011010000100010111101110000100 11110100 00000100",      "1 11111100101 111010011010000100010111101110000100 11110100 00000100",      "0 00000001001 110000010110110001011100010100100101 00110101 01110101",      "0 11111110100 001000111010010100010110111010000010 11011001 10111010",      "0 00000001010 000011011010011101101010100101111100 10111001 10101101",      "0 11111110100 010111011111100101001110101100000011 01101011 10101100",      "0 00000001010 000101000110010100110011101010000110 00111110 01010001",      "0 11111110100 011001101011100100100011110110110000 01001110 01011101") 
First 6 of 8 (type 'character'): [1] "0 11111100101 111010011010000100010111101110000100 11110100 00000100"
[2] "1 11111100101 111010011010000100010111101110000100 11110100 00000100"
[3] "0 00000001001 110000010110110001011100010100100101 00110101 01110101"
[4] "0 11111110100 001000111010010100010110111010000010 11011001 10111010"
[5] "0 00000001010 000011011010011101101010100101111100 10111001 10101101"
[6] "0 11111110100 010111011111100101001110101100000011 01101011 10101100"
8 string mismatches
Running test id 1729.12      Test 1729.12 didn't produce correct output:
Expected: <<V1\n5.123456789e+300\n-5.123456789e+300\n1e-305\n1e+305\n1.2e-305\n1.2e+305\n1.23e-305\n1.23e+305>>
Observed: <<V1\n5.12345678900000e+300\n-5.12345678900000e+300\n9.99999999999999e-306\n1.00000000000000e+305\n1.20000000000000e-305\n1.20000000000000e+305\n1.23000000000000e-305\n1.23000000000000e+305>>
Running test id 1729.14      Test 1729.14 ran without errors but failed check that x equals y:
> x = capture.output(fwrite(DT)) 
First 6 of 5 (type 'character'): [1] "V1"                    "2.22044604925031e-16"  "1.11022302462516e-16" 
[4] "2.2250738585072e-308"  "1.79769313486232e+308"
> y = capture.output(write.csv(DT, row.names = FALSE, quote = FALSE)) 
First 6 of 5 (type 'character'): [1] "V1"                    "2.22044604925031e-16"  "1.11022302462516e-16" 
[4] "2.22507385850720e-308" "1.79769313486232e+308"
1 string mismatch
Running test id 1759      Test 1760 not run because this session either has no OpenMP or has been limited to one thread (e.g. under UBSAN and ASAN)
Running test id 1910          
10 longest running tests took 1377s (36% of 3768s)
      ID    time nTest
 1: 1848 187.561     1
 2: 1223 169.776   728
 3: 1438 145.614   354
 4: 1835 141.094     1
 5: 1253 139.518   485
 6: 1648 133.140    45
 7: 1652 126.180    45
 8: 1650 126.119    45
 9: 1874 108.173     5
10: 1642  99.912    45
Error in eval(exprs[i], envir) : 
  9 errors out of 6932 in 01:02:48 on Mon May  7 14:21:19 2018. [endian==little, sizeof(long double)==16, sizeof(pointer)==8, TZ=America/Los_Angeles, locale='LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=en_US.UTF-8;LC_COLLATE=en_US.UTF-8;LC_MONETARY=en_US.UTF-8;LC_MESSAGES=en_US.UTF-8;LC_PAPER=en_US.UTF-8;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=en_US.UTF-8;LC_IDENTIFICATION=C']. Search inst/tests/tests.Rraw for test numbers: 648, 819, 1018, 1262, 1729.4, 1729.8, 1729.9, 1729.12, 1729.14.
[1] "2018-05-07 14:21:19 PDT"
[1] "01:02:51"

[mattdowle]

I wrote to CRAN :

I see both cbuf (local) and cbuff (global) in the lines following here which doesn't look quite right to my eyes on first glance :
https://github.com/wch/r-source/blob/trunk/src/main/character.c#L984
and the valgrind messages I'm seeing mention do_tolower.

[mattdowle]

Currently, this is front (and only) runner for #2767.
An uninitialized variable being passed to an allocating function would explain the behaviour we're seeing: 'Memory exhausted'. It's as though an allocating function somewhere is suddenly being passed a large value.

[mattdowle]

I've submitted 1.11.2 to CRAN with the following message :

Dear CRAN,
All CRAN check errors resolved.
Several memory faults fixed.
There should now be 2 rchk messages (both freadR.c) down from 189. I'll need to speak to Tomas Kalibera if there's a way to get those final 2 to pass rchk. It's to do with freadR.c calling fread.c. The test suite passes gctorture(step=50) and I believe the usage is ok in this unusual case.
I have not rerun rev deps as this release has been kept minimal.
You may still see memory faults in R-devel and perhaps R 3.5.0 (but not before). I've emailed CRAN with the valgrind output pointing to an uninitialized variable in R's character.c:do_tolower.
I'm submitting now to give you the option to take it or not. It's certainly not worse than what's on CRAN now and will be useful to clear existing errors. If you'd prefer to wait for an answer on the valgrind messages, that's fine too.
I'm on vacation for one day tomorrow and you asked me for the update asap so that's in my mind as well, hence submitting now.
Thanks and best,
Matt
```

[MichaelChirico]

thanks for this! gc is not the only thing that's been tortured the past week, i see

[mattdowle]

Reply from Tomas Kalibera :

I don't see a problem with "cbuf" in the current code in do_tolower. I don't see a problem with "cbuff", either. When you are debugging valgrind reports, I think the best way is to experiment yourself with the code - if initializing some variable more explicitly makes valgrind happy, we could just add it to the code on those grounds. But from your reports I would not be suspecting do_tolower.

We have a new report from Kurt Hornik on a valgrind message from data.table, but that's the small leak I knew about and filed here #2812. I can't imagine that is causing the problem we are seeing, but we should clear it up to rule it out :

Btw, whereas my Debian pretests worked fine where Uwe's failed, checking with current R-devel and --use-valgrind finds

Log

> ==8258== 304 bytes in 1 blocks are possibly lost in loss record 321 of 2,561
> ==8258==    at 0x4C2EBA5: calloc (coregrind/m_replacemalloc/vg_replace_malloc.c:711)
> ==8258==    by 0x40111E1: allocate_dtv (/build/glibc-kAz5Pl/glibc-2.27/elf/../elf/dl-tls.c:286)
> ==8258==    by 0x4011B6D: _dl_allocate_tls (/build/glibc-kAz5Pl/glibc-2.27/elf/../elf/dl-tls.c:530)
> ==8258==    by 0x5916079: allocate_stack (/build/glibc-kAz5Pl/glibc-2.27/nptl/allocatestack.c:627)
> ==8258==    by 0x5916079: pthread_create@@GLIBC_2.2.5 (/build/glibc-kAz5Pl/glibc-2.27/nptl/pthread_create.c:644)
> ==8258==    by 0x56F5AE1: ??? (in /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0)
> ==8258==    by 0x56ECBA9: GOMP_parallel (in /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0)
> ==8258==    by 0x175DE6E5: reorder (/home/Hornik/tmp/CRAN/data.table.Rcheck/00_pkg_src/data.table/src/reorder.c:64)
> ==8258==    by 0x4F67B5A: bcEval (/home/Hornik/src/R/src/main/eval.c:7280)
> ==8258==    by 0x4F7205F: Rf_eval (/home/Hornik/src/R/src/main/eval.c:624)
> ==8258==    by 0x4F73B6E: R_execClosure (/home/Hornik/src/R/src/main/eval.c:1764)
> ==8258==    by 0x4F69950: bcEval (/home/Hornik/src/R/src/main/eval.c:6740)
> ==8258==    by 0x4F7205F: Rf_eval (/home/Hornik/src/R/src/main/eval.c:624)

> in the example .Rout and 

> ==8375== HEAP SUMMARY:
> ==8375==     in use at exit: 149,628,072 bytes in 25,346 blocks
> ==8375==   total heap usage: 2,528,220 allocs, 2,502,874 frees, 25,812,555,296 bytes allocated
> ==8375== 
> ==8375== 32 bytes in 1 blocks are definitely lost in loss record 24 of 2,695
> ==8375==    at 0x4C2CB8F: malloc (coregrind/m_replacemalloc/vg_replace_malloc.c:299)
> ==8375==    by 0x173BC927: forder (/home/Hornik/tmp/CRAN/data.table.Rcheck/00_pkg_src/data.table/src/forder.c:1210)
> ==8375==    by 0x4F67B5A: bcEval (/home/Hornik/src/R/src/main/eval.c:7280)
> ==8375==    by 0x4F7205F: Rf_eval (/home/Hornik/src/R/src/main/eval.c:624)
> ==8375==    by 0x4F73B6E: R_execClosure (/home/Hornik/src/R/src/main/eval.c:1764)
> ==8375==    by 0x4F69950: bcEval (/home/Hornik/src/R/src/main/eval.c:6740)
> ==8375==    by 0x4F7205F: Rf_eval (/home/Hornik/src/R/src/main/eval.c:624)
> ==8375==    by 0x4F73B6E: R_execClosure (/home/Hornik/src/R/src/main/eval.c:1764)
> ==8375==    by 0x4F72229: Rf_eval (/home/Hornik/src/R/src/main/eval.c:747)
> ==8375==    by 0x4F72963: forcePromise (/home/Hornik/src/R/src/main/eval.c:520)
> ==8375==    by 0x4F72DE7: FORCE_PROMISE (/home/Hornik/src/R/src/main/eval.c:4971)
> ==8375==    by 0x4F72DE7: getvar (/home/Hornik/src/R/src/main/eval.c:5013)
> ==8375==    by 0x4F6788E: bcEval (/home/Hornik/src/R/src/main/eval.c:6504)
> ==8375== 
> ==8375== 304 bytes in 1 blocks are possibly lost in loss record 105 of 2,695
> ==8375==    at 0x4C2EBA5: calloc (coregrind/m_replacemalloc/vg_replace_malloc.c:711)
> ==8375==    by 0x40111E1: allocate_dtv (/build/glibc-kAz5Pl/glibc-2.27/elf/../elf/dl-tls.c:286)
> ==8375==    by 0x4011B6D: _dl_allocate_tls (/build/glibc-kAz5Pl/glibc-2.27/elf/../elf/dl-tls.c:530)
> ==8375==    by 0x5916079: allocate_stack (/build/glibc-kAz5Pl/glibc-2.27/nptl/allocatestack.c:627)
> ==8375==    by 0x5916079: pthread_create@@GLIBC_2.2.5 (/build/glibc-kAz5Pl/glibc-2.27/nptl/pthread_create.c:644)
> ==8375==    by 0x56F5AE1: ??? (in /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0)
> ==8375==    by 0x56ECBA9: GOMP_parallel (in /usr/lib/x86_64-linux-gnu/libgomp.so.1.0.0)
> ==8375==    by 0x173DB6E5: reorder (/home/Hornik/tmp/CRAN/data.table.Rcheck/00_pkg_src/data.table/src/reorder.c:64)
> ==8375==    by 0x4F67B5A: bcEval (/home/Hornik/src/R/src/main/eval.c:7280)
> ==8375==    by 0x4F7205F: Rf_eval (/home/Hornik/src/R/src/main/eval.c:624)
> ==8375==    by 0x4F73B6E: R_execClosure (/home/Hornik/src/R/src/main/eval.c:1764)
> ==8375==    by 0x4F69950: bcEval (/home/Hornik/src/R/src/main/eval.c:6740)
> ==8375==    by 0x4F7205F: Rf_eval (/home/Hornik/src/R/src/main/eval.c:624)

> in tests/main.Rout.fail, which fails with

>   8 errors out of 7707 in 00:15:39 on Tue May 8 18:17:47
>   2018. [endian==little, sizeof(long double)==16, sizeof(pointer)==8,
>   TZ=Europe/Vienna,
>   locale='LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=en_US.UTF-8;LC_COLLATE=C;LC_MONETARY=en_US.UTF-8;LC_MESSAGES=en_US.UTF-8;LC_PAPER=en_US.UTF-8;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=en_US.UTF-8;LC_IDENTIFICATION=C']. Search
>   inst/tests/tests.Rraw for test numbers: 648, 1018, 1262, 1729.4,
>   1729.8, 1729.9, 1729.12, 1729.14.

[mattdowle]

Tomas checked and said this was a false positive from valgrind. Indeed this seems correct since we've now found and resolved the memory problem we were looking for (#2866).
We could experiment another time to silence valgrind on this one, but there is no plan to. Hence closing.