Xgboost: src/tree/updater_histmaker.cc:301: Check failed: (istart) != (hist.size)

[jq]

For bugs or installation issues, please provide the following information.
The more information you provide, the more easily we will be able to offer
help and advice.

Environment info

Operating System:
ubuntu
Compiler:
gcc
Package used (python/R/jvm/C++):
jvm
xgboost version used:
If installing from source, please provide

1. The commit hash (git rev-parse HEAD)
   7ff742ebf7ec70c747f9aec004689fc727ba3304
2. Logs will be helpful (If logs are large, please upload as attachment).
   18:04:31] dmlc-core/include/dmlc/logging.h:235: [18:04:31] src/tree/updater_histmaker.cc:301: Check failed: (istart) != (hist.size)
   [18:04:31] dmlc-core/include/dmlc/logging.h:235: [18:04:31] src/tree/updater_histmaker.cc:301: Check failed: (istart) != (hist.size)
   [18:04:31] dmlc-core/include/dmlc/logging.h:235[18:04:31] dmlc-core/include/dmlc/logging.h:235: [18:04:31] src/tree/updater_histmaker.cc:301: Check failed: (istart) != (hist.size)
   : [18:04:31] src/tree/updater_histmaker.cc:301: Check failed: (istart) != (hist.size)

Steps to reproduce

1. large data set with 4796118 samples, about 2.55G

What have you tried?

1. I hit it before, it was NaN, but this time seems not, I cat the sample | grep -i NaN, found nothing,
2. also grep Null, ve, as suggested in https://github.com/dmlc/xgboost/issues/1305
3. I try to cut the data set in half, each half can train without problem.
4. running on python right now, will see if it works on python or not.

[nikunjgit]

I saw the same thing on large data. Reducing the data size works for me too! https://github.com/dmlc/xgboost/issues/1868

[willettk]

I have the same set of errors as @jq running XGBoost (v0.6) in Python on a very large dataset of ~14 million rows, ~300 features. I don't have any errors running XGBoost in the same environment when I decrease either the number of features or number of rows, so I'm wondering if this is related to memory. I'm running XGBoost on a very bulked-up EC2 instance (244 GB of memory and 32 vCPUs), so I hadn't expected either memory or processing speed to be an issue.

Is there a limit on the size of the data that XGBoost can deal with? Decreasing the size of my total dataset unfortunately isn't an option.

[tqchen]

try https://github.com/dmlc/xgboost/issues/2053

[willettk]

My apologies for having to bump this again, @tqchen, but the fix in #2053 didn't change my results. I downloaded and installed the latest commit from source and tried re-running it with the same data and parameters. The full (and quite long) stack trace is below - I'm afraid most of the details there are fairly opaque to me, but it seems to throw its fatal error when fv=inf and hist.last=inf.

Please let me know if I should respin this into a separate issue, or whether we can continue in this thread.

host:/$ python xgb_data.py
[21:59:45] Tree method is automatically selected to be 'approx' for faster speed. to use old behavior(exact greedy algorithm on single machine), set tree_method to 'exact'
[22:00:06] [22:00:06] src/tree/updater_histmaker.cc:304: fv=[22:00:06src/tree/updater_histmaker.cc[[[[[22:00:06] src/tree/updater_histmaker.cc[[22:00:06] src/tree/updater_histmaker.cc:[22:00:06] src/tree/updater_histmaker.cc:[22:00:06] src/tree/updater_histmaker.cc:304: fv=[22:00:06] src/tree/updater_histmaker.cc:304: fv=inf, hist.size=54
[22:00:06] src/tree/updater_histmaker.cc:306: hist[0]=22:00:06] src/tree/updater_histmaker.cc:304: 22:00:06] src/tree/updater_histmaker.cc:304: fv=inf, hist.size=64
[22:00:06] src/tree/updater_histmaker.cc:[22:00:06] src/tree/updater_histmaker.cc:304: fv=[22:00:06] src/tree/updater_histmaker.cc:304: fv=inf, hist.size=59
[22:00:06] src/tree/updater_histmaker.cc:[0.00787302
[22:00:06] src/tree/updater_histmaker.cc:306: hist[1]=0.0120472
[22:00:06] src/tree/updater_histmaker.cc:304: fv=inf, hist.size=65
[22:00:06] src/tree/updater_histmaker.cc:306: hist[0]=0.00349077
[22:00:06] src/tree/updater_histmaker.cc:306: hist[1]=0.00948591
[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=[22:00:06] src/tree/updater_histmaker.cc:inf, hist.size=65
[22:00:06[22:00:06] src/tree/updater_histmaker.cc:304: fv=inf, hist.size=63
[22:00:06] src/tree/updater_histmaker.cc:306: hist[22:00:06] src/tree/updater_histmaker.cc:304: fv=inf, hist.size=65
[22:00:06] src/tree/updater_histmaker.cc:306: [22:00:06] src/tree/updater_histmaker.cc:304: fv=inf, hist.size=65
[22:00:06] src/tree/updater_histmaker.cc:306: [22:00:06] src/tree/updater_histmaker.cc:304: fv=0]=0.00451068
[22:00:06] src/tree/updater_histmaker.cc:306: hist[1]=0.01041570.0161098
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.0232926
[22:00:06] src/tree/updater_histmaker.cc:306: hist[4]=0.0327339
[22:00:06] src/tree/updater_histmaker.cc:306: hist[5]=0.0451545
[22:00:06] src/tree/updater_histmaker.cc:306: hist[6]=fv=inf, hist.size=65
304: fv=inf, hist.size=59
[22:00:06] src/tree/updater_histmaker.cc:304: fv=inf, hist.size=64
[22:00:06] src/tree/updater_histmaker.cc:306: hist[0]=0.00689908
[22:00:06] src/tree/updater_histmaker.cc:306: hist[1]=0.0145975
[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.0227498
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=inf, hist.size=61
[22:00:06] src/tree/updater_histmaker.cc[22:00:06] src/tree/updater_histmaker.cc:304: fv=inf, hist.size=65
[22:00:06] src/tree/updater_histmaker.cc:306: hist[0]=0.00135401
[22:00:06] src/tree/updater_histmaker.cc:306: hist[1]=0.00478042
[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.00943423
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.0130444
[22:00:06] src/tree/updater_histmaker.cc:306: hist[4]=0.0194969
[22:00:06] src/tree/updater_histmaker.cc:306: hist[5]=0.0275581
[22:00:06] src/tree/updater_histmaker.cc:306: hist[6]=0.0359702
[22:00:06] src/tree/updater_histmaker.cc:306: hist[7]=0.0451532
[22:00:06] src/tree/updater_histmaker.cc:306: hist[8]=0.0555546
[22:00:06] src/tree/updater_histmaker.cc:[22:00:06[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.0212756
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.0312252
[22:00:06] src/tree/updater_histmaker.cc:306: hist[:[306: hist[0]=0.00420068
[22:00:06] src/tree/updater_histmaker.cc:306: hist[1]=0.0111514
[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.018017
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.024999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[4]=0.0333323
[22:00:06] src/tree/updater_histmaker.cc:306: hist[5]=0.0434773
[22:00:06304: fv=inf, hist.size=inf, hist.size=65
[22:00:06] src/tree/updater_histmaker.cc:306: hist[0:304: fv=inf, hist.size=63
[22:00:06] src/tree/updater_histmaker.cc:306: hist[0]=] src/tree/updater_histmaker.cc:306: hist[0]=0.00251235[22:00:06] src/tree/updater_histmaker.cc:304: fv=inf, hist.size=56
[22:00:06] src/tree/updater_histmaker.cc:306: hist[[]=0.00296864
[22:00:06] src/tree/updater_histmaker.cc:306: hist[1]=0.0085359
[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.0144852
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.0213483
[22:00:06] src/tree/updater_histmaker.cc:306: hist[4]=0.0306533
[22:00:06] src/tree/updater_histmaker.cc:306: hist[5]=0.039397
[22:00:06] src/tree/updater_histmaker.cc:306: hist[6]=304: 0.00235749
[] src/tree/updater_histmaker.cc:hist[0]=0.145262
[22:00:06] src/tree/updater_histmaker.cc:306: hist[1]=0.221851
[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.279783[22:00:06] src/tree/updater_histmaker.cc:304: fv=inf, hist.size=65
[22:00:06] src/tree/updater_histmaker.cc:306: hist[0]=0.00158003
[22:00:06] src/tree/updater_histmaker.cc:306: hist[1]=0.0046073
[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.00994576
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.0144918
[22:00:06] src/tree/updater_histmaker.cc:306: hist[4]=0.0213323
[22:00:06] src/tree/updater_histmaker.cc:306: hist[5]=0.0278186
[22:00:06] src/tree/updater_histmaker.cc:306: hist[6]=0.0359702
[22:00:06] src/tree/updater_histmaker.cc:306: hist[7]=0.0451118
[22:00:06] src/tree/updater_histmaker.cc:306: hist[8]=0.0555546
[22:00:06] src/tree/updater_histmaker.cc:306: hist[9]=0.0681808
[22:00:06] src/tree/updater_histmaker.cc:306: hist[10]=0.0810801
[22:00:0622:00:0622:00:06] src/tree/updater_histmaker.cc:304: fv=inf, hist.size=63
[22:00:06] [] src/tree/updater_histmaker.cc:306306: hist[0:306: hist[0]=0.00248922
[22:00:06] src/tree/updater_histmaker.cc:306: hist[1]=0.00678430.0558521
[22:00:06] src/tree/updater_histmaker.cc:306: hist[7]=0.0688308

[22:00:06] src/tree/updater_histmaker.cc0.0353506
[22:00:06] src/tree/updater_histmaker.cc:306: hist[4]=0.0452563
[22:00:06] src/tree/updater_histmaker.cc:306: hist[5]=] src/tree/updater_histmaker.cc:304: fv=inf, hist.size=hist[0]=0.0546065
[22:00:06] src/tree/updater_histmaker.cc:306: hist[1]=0.0943079
[22:00:06] src/tree/updater_histmaker.cc:304: fv=inf, hist.size=65
[22:00:06] src/tree/updater_histmaker.cc:306[22:00:06] src/tree/updater_histmaker.cc:304: fv=inf, hist.size=64
[22:00:06] src/tree/updater_histmaker.cc:306: hist[0]=[22:00:06] src/tree/updater_histmaker.cc:306: hist[8]=0.0860691
[22:00:06] src/tree/updater_histmaker.cc:306: hist[9]=0.103231
[22:00:06] src/tree/updater_histmaker.cc:306: hist[10]=0.119283
[22:00:06] src/tree/updater_histmaker.cc:306: hist[11]=0.140261
[22:00:06] src/tree/updater_histmaker.cc:306: hist[12]=0.163371
[22:00:06] src/tree/updater_histmaker.cc:306: hist[13]=0.186391
[22:00:06] src/tree/updater_histmaker.cc:306: hist[14]=0.210481
[22:00:06] src/tree/updater_histmaker.cc:306: hist[15]=0.235404
[22:00:06] src/tree/updater_histmaker.cc:306: hist[16]=0.261368
[22:00:06] src/tree/updater_histmaker.cc:306: hist[17]=0.288155
[22:00:06] src/tree/updater_histmaker.cc:306: hist[18]=0.32096
[22:00:06] src/tree/updater_histmaker.cc:306: hist[19]=0.353355
[22:00:06] src/tree/updater_histmaker.cc:306: hist[20]=0.389514
[22:00:06] src/tree/updater_histmaker.cc:306: hist[21]=0.422725
[22:00:06] src/tree/updater_histmaker.cc:306: hist[22]=0.464018
[22:00:06] src/tree/updater_histmaker.cc:306: hist[23]=0.505587
[22:00:06] src/tree/updater_histmaker.cc:306: hist[24]=0.550871
[22:00:06] src/tree/updater_histmaker.cc:306: hist[25]=0.59479
[22:00:06] src/tree/updater_histmaker.cc:306: hist[26]=0.64211
[22:00:06] src/tree/updater_histmaker.cc:306: hist[27]=0.696741
[22:00:06] src/tree/updater_histmaker.cc:306: hist[28]=0.748797
[22:00:06] src/tree/updater_histmaker.cc:306: hist[29]=0.810648
[22:00:06] src/tree/updater_histmaker.cc:306: hist[30]=0.872051]=0.271002
[22:00:06] src/tree/updater_histmaker.cc:306: hist[1]=0.561038
[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.624999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.68337
[22:00:06] src/tree/updater_histmaker.cc] src/tree/updater_histmaker.cc:306: hist[11]=0.0930223
[22:00:06
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.323682
[22:00:06] src/tree/updater_histmaker.cc:306: hist[4]=0.370329
[22:00:06] src/tree/updater_histmaker.cc:306: hist[5]=0.41008
[22:00:06] src/tree/updater_histmaker.cc:306: hist[6]=0.442095
[22:00:06] src/tree/updater_histmaker.cc:306: hist[7]=0.477201
[22:00:06] src/tree/updater_histmaker.cc:306: hist[8]=0.507646
[22:00:06] src/tree/updater_histmaker.cc:306: hist[9]=0.539937
[22:00:06] src/tree/updater_histmaker.cc:306: hist[10]=0.566418
[22:00:06] src/tree/updater_histmaker.cc:306: hist[11]=0.59492
fv=inf, hist.size=65
[22:00:06] src/tree/updater_histmaker.cc:30622:00:06] src/tree/updater_histmaker.cc:304: fv=inf, hist.size=57
[22:00:06] src/tree/updater_histmaker.cc:306: hist[0]=304: fv=inf, hist.size=46
[22:00:06] src/tree/updater_histmaker.cc:
[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.012499
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.0203942
30663[22:00:06[22:00:06inf0]=0.00297519
[22:00:06] src/tree/updater_histmaker.cc:306: hist[1]=0.00913142
[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.0153836
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.0227263[22:00:06] src/tree/updater_histmaker.cc:304: fv=inf, hist.size=64
[22:00:06] src/tree/updater_histmaker.cc:306: hist[0]=-21.7368304
306: hist[9]=0.0696193
[22:00:06] src/tree/updater_histmaker.cc:306: hist[10]=0.079999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[11]=
[22:00:06] src/tree/updater_histmaker.cc:306: hist[31]=0.941135
[22:00:06] src/tree/updater_histmaker.cc:306: hist[32]=1.00701
[22:00:06] src/tree/updater_histmaker.cc:306: hist[33]=1.07705
[22:00:06] src/tree/updater_histmaker.cc:306: hist[34]=1.17062

[[22:00:06] src/tree/updater_histmaker.cc:304: fv=inf, hist.size=65
[22:00:06] src/tree/updater_histmaker.cc:306: hist[0]=0.00653311[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.0168665
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.0237379
[22:00:06] src/tree/updater_histmaker.cc:306: hist[4]=0.0334957
[22:00:06] src/tree/updater_histmaker.cc:306: hist[5]=0.043749
[22:00:06] src/tree/updater_histmaker.cc:306: hist[6]=0.0548771
[22:00:06] src/tree/updater_histmaker.cc:306: hist[7]=0.0678723
[22:00:06] src/tree/updater_histmaker.cc:306: hist[8]=0.080401
[22:00:06:306: hist[4]=0.729613
[22:00:06] src/tree/updater_histmaker.cc:306: hist[5]=0.754544
[22:00:06] src/tree/updater_histmaker.cc:306: hist[6]=0.792207
[22:00:06] src/tree/updater_histmaker.cc:306: hist[7]=0.799999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[8]=0.813332
[22:00:06] src/tree/updater_histmaker.cc:306: hist[9]=4]=0.0384605
[22:00:06] src/tree/updater_histmaker.cc:306: hist[5]=0.049999
0.0518222[22:00:06] src/tree/updater_histmaker.cc:306: hist[12]=0.619047
[22:00:06] src/tree/updater_histmaker.cc:306: hist[13]=0.647952
[22:00:06] src/tree/updater_histmaker.cc:306: hist[14]=0.670018
[22:00:06] src/tree/updater_histmaker.cc:306: hist[15]=0.696034
[22:00:06] src/tree/updater_histmaker.cc:306: hist[16]=0.720151
[22:00:06] src/tree/updater_histmaker.cc:306: hist[17]=0.739499
[22:00:06] ] ] 0.008619690.83333222:00:06] src/tree/updater_histmaker.cc:304] src/tree/updater_histmaker.cc:306: hist[12]=0.11111
[22:00:06] src/tree/updater_histmaker.cc:306: hist[13]=0.125409
[22:00:06] src/tree/updater_histmaker.cc:306: hist[14]=0.142856
[22:00:06] src/tree/updater_histmaker.cc:306: hist[15]=0.164347
[22:00:06] src/tree/updater_histmaker.cc:306: hist[16]=0.187878
[22:00:06] src/tree/updater_histmaker.cc:306: hist[17]=0.20707
[22:00:06] src/tree/updater_histmaker.cc:306: hist[18]=0.233749
[22:00:06] src/tree/updater_histmaker.cc:306: hist[19]=0.257575
[22:00:06] src/tree/updater_histmaker.cc:306: hist[20]=0.285565
: : fv=: fv=[22:00:06] src/tree/updater_histmaker.cc:306: hist[6]=0.0603438
[22:00:06] src/tree/updater_histmaker.cc:306: hist[7]=0.0714276
[22:00:06] src/tree/updater_histmaker.cc:306: hist[8]=0.0833323
[22:00:06] src/tree/updater_histmaker.cc:306: hist[9]=0.099999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[10]=:306: hist[1]=0.00951546
[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.0157329
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.022166
[22:00:06]
[22:00:06] src/tree/updater_histmaker.cc:306: hist[7]=0.0645413
[22:00:06] src/tree/updater_histmaker.cc:306: hist[8]=0.0796975
[22:00:06] src/tree/updater_histmaker.cc:306: hist[9]=0.0965344
[22:00:06] src/tree/updater_histmaker.cc:306: hist[10]=0.113189
[22:00:06] src/tree/updater_histmaker.cc:306: hist[11]=0.132093
[22:00:06] src/tree/updater_histmaker.cc:306: hist[12]=0.150094
[22:00:06] src/tree/updater_histmaker.cc:306: hist[13]=0.171817
[22:00:06] src/tree/updater_histmaker.cc:306: hist[14]=0.192292
[22:00:06] src/tree/updater_histmaker.cc:306: hist[15]=0.218077
[22:00:06] src/tree/updater_histmaker.cc:306: hist[16]=0.244705
[22:00:06] src/tree/updater_histmaker.cc:: hist[] src/tree/updater_histmaker.cc:304: fv=inf, hist.size=65
[22:00:06] src/tree/updater_histmaker.cc:306: hist[0]=0.00314951
hist[0]=0.00409877
63
[22:00:0622:00:06] src/tree/updater_histmaker.cc:304: fv=inf, hist.size=65

[22:00:06] src/tree/updater_histmaker.cc:306: hist[4]=, hist.size=65
[22:00:06[22:00:06] src/tree/updater_histmaker.cc:
infsrc/tree/updater_histmaker.cc:306: hist[422:00:06] src/tree/updater_histmaker.cc:3060.11111
[22:00:06] src/tree/updater_histmaker.cc:306: hist[11]=0.124999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[12]=0.142856
[22:00:06] src/tree/updater_histmaker.cc:306: hist[13]=0.157894
[22:00:06] src/tree/updater_histmaker.cc:306: hist[14]=0.173912
[22:00:06] src/tree/updater_histmaker.cc:306: hist[15]=0.199999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[16]=0.208332
[22:00:06] src/tree/updater_histmaker.cc:306: hist[17]=0.249999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[18]=0.272726
[22:00:06] src/tree/updater_histmaker.cc:306: hist[19]=0.333332
[22:00:06] src/tree/updater_histmaker.cc:306: hist[20]=0.366666
[22:00:06] src/tree/updater_histmaker.cc:306: hist[21]=0.399999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[22]=0.486485
[22:00:06] src/tree/updater_histmaker.cc:306: hist[23]=0.499999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[24]=0.571428
[22:00:06] src/tree/updater_histmaker.cc:306: hist[25]=0.666666
[22:00:06] src/tree/updater_histmaker.cc:306: hist[26]=0.735293
[22:00:06] src/tree/updater_histmaker.cc:306: hist[27]=0.799999
[22:00:06src/tree/updater_histmaker.cc:]=0.0298942: hist[: hist[0]=0.0033163
[22:00:06] src/tree/updater_histmaker.cc:306: hist[35]=1.25907
[22:00:06] src/tree/updater_histmaker.cc:306: hist[36]=1.36134
[22:00:06] src/tree/updater_histmaker.cc:306: hist[37]=1.46551
[22:00:06] src/tree/updater_histmaker.cc:306: hist[38]=1.58345
[22:00:06] src/tree/updater_histmaker.cc:306: hist[39]=1.72714
[22:00:06] src/tree/updater_histmaker.cc:306: hist[40]=1.87025
[22:00:06] src/tree/updater_histmaker.cc:
[22:00:06] src/tree/updater_histmaker.cc:306: hist[1]=-10.42130.00367735
0]=0.00313677
[22:00:06[22:00:06] src/tree/updater_histmaker.cc:306: hist[4]=0.0286755
[22:00:06] src/tree/updater_histmaker.cc:306: hist[5]=0.0381669
[22:00:06] src/tree/updater_histmaker.cc:306: hist[[22:00:06, hist.size=65
[22:00:06] src/tree/updater_histmaker.cc:306: hist[0]=0.00258434[22:00:06] src/tree/updater_histmaker.cc:306: hist[0]=0.00377582
[22:00:06] src/tree/updater_histmaker.cc:306: hist[1]=0.0102747
[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.0173815
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.0244164
[22:00:06] src/tree/updater_histmaker.cc:306: hist[4]=0.0340238
[22:00:06] src/tree/updater_histmaker.cc:306: hist[5]=0.0450012
[22:00:06] src/tree/updater_histmaker.cc:306: hist[6]=0.0562869
[22:00:06] src/tree/updater_histmaker.cc:306: hist[7]=0.0679702
[22:00:06] src/tree/updater_histmaker.cc:306: hist[8]=0.0824821
[22:00:06] src/tree/updater_histmaker.cc:306: hist[9]=0.100092
[22:00:06] src/tree/updater_histmaker.cc:306: hist[10]=0.114162
[0.0574615
[22:00:06] src/tree/updater_histmaker.cc:306: hist[306: hist[21]=0.314285] src/tree/updater_histmaker.cc:306: hist[28]=0.999999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[29]=1.04762
[22:00:06] src/tree/updater_histmaker.cc:306: hist[30]=1.2
[22:00:06] src/tree/updater_histmaker.cc:306: hist[31]=1.33333
[22:00:06] src/tree/updater_histmaker.cc:306: hist[32306: hist[0]=0.00246408
: hist[1] src/tree/updater_histmaker.cc22:00:06] 0.0303787src/tree/updater_histmaker.cc[22:00:06] src/tree/updater_histmaker.cc:306: hist[41]=2.02474
[22:00:06] src/tree/updater_histmaker.cc:306: hist[42]=2.22222
[22:00:06] src/tree/updater_histmaker.cc:306: hist[43]=2.43312
[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=22:00:06] src/tree/updater_histmaker.cc:306: hist[11]=0.133919
[22:00:06] src/tree/updater_histmaker.cc:306: hist[12]=0.154674
[22:00:06] src/tree/updater_histmaker.cc:306: hist[13]=0.176932
[22:00:06] src/tree/updater_histmaker.cc:306: hist[14]=0.198322
[22:00:06] src/tree/updater_histmaker.cc:306: hist[15]=0.221445
[22:00:06] src/tree/updater_histmaker.cc:306: hist[16]=0.250103
[22:00:06] src/tree/updater_histmaker.cc:306: hist[17]=0.279395
[22:00:06] src/tree/updater_histmaker.cc[22:00:06] src/tree/updater_histmaker.cc:306: hist[1]=0.00900801
[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.0142293
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3306: hist[0]=0.099999
[22:00:06] src/tree/updater_histmaker.cc22:00:06]=1.5
[22:00:06] src/tree/updater_histmaker.cc:306: hist[33]=1.75
[22:00:06] src/tree/updater_histmaker.cc:306: hist[34]=
[
[22:00:06] src/tree/updater_histmaker.cc:306: hist[10]=0.847869
[22:00:06] src/tree/updater_histmaker.cc:306: hist[11]=0.858368
[22:00:06] src/tree/updater_histmaker.cc:306: hist[12]=] src/tree/updater_histmaker.cc:306: hist[6]=0.0722423
[0]=0.00299004
[22:00:06] src/tree/updater_histmaker.cc:306: hist[1]=0.00792732
[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.0133859
[0.869564
[22:00:06] src/tree/updater_histmaker.cc:306: hist[13]=0.882352
[22:00:06] src/tree/updater_histmaker.cc:306: hist[14]=0.892018
[22:00:06] src/tree/updater_histmaker.cc:306: hist[15]=0.899999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[16]=0.905262
[22:00:06] src/tree/updater_histmaker.cc:306: hist[17]=0.913999
[22:00:06] src/tree/updater_histmaker.cc[22:00:06] [22:00:06
[22:00:06] src/tree/updater_histmaker.cc:306: hist[22]=0.344261
[22:00:06] src/tree/updater_histmaker.cc:306: hist[23]=2
22:00:06] src/tree/updater_histmaker.cc:306: hist[1]=0.0196068
[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.0286728
:306: hist[0]=0.00390589
[[22:00:06] src/tree/updater_histmaker.cc:306: hist[1]=0.00546348
[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.00798987
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.0108686
[22:00:06] src/tree/updater_histmaker.cc:306: hist[4]=0.0147049
[22:00:06] src/tree/updater_histmaker.cc:306: hist[5]=0.0197358
[22:00:06] src/tree/updater_histmaker.cc:306: hist[6]=0.0285704
[22:00:06] src/tree/updater_histmaker.cc:306: hist[7]=0.037036
[22:00:06] src/tree/updater_histmaker.cc:306: hist[8]=0.049999
] ] src/tree/updater_histmaker.cc
[22:00:06] src/tree/updater_histmaker.cc:306: hist[50]=src/tree/updater_histmaker.cc:0.374999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[24]=0.40909
[22:00:06] src/tree/updater_histmaker.cc:306: hist[25]=0.444443
[22:00:06] src/tree/updater_histmaker.cc:306: hist[26]=0.492423
[22:00:06] src/tree/updater_histmaker.cc:306: hist[27]=0.521738
[22:00:06] src/tree/updater_histmaker.cc:306: hist[28]=0.565394
[22:00:0622:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.019849
[22:00:06[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.039999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[4]=0.0526306
[22:00:06] src/tree/updater_histmaker.cc:306: hist[5]=0.0642192
[22:00:06] src/tree/updater_histmaker.cc:306: hist[66]=0.0476181
] src/tree/updater_histmaker.cc:306: hist[1]=0.0102713
[:306: hist[6]=] src/tree/updater_histmaker.cc:306: hist[1]=0.0950362
[22:00:06] src/tree/updater_histmaker.cc:306: hist[12]=0.111110.122918
[22:00:06] src/tree/updater_histmaker.cc::306: hist[18]=0.921999
] src/tree/updater_histmaker.cc:306: hist[9]=0.062499
[22:00:06
0.0526306306: hist[170.0103502
:306: hist[1]=0.0102031
[22:00:06] src/tree/updater_histmaker.cc:inf, hist.size=65
[22:00:06[22:00:06] src/tree/updater_histmaker.cc:306: hist[7]=0.0618946
[22:00:06] src/tree/updater_histmaker.cc:306: hist[]=0.00650659
[22:00:06] src/tree/updater_histmaker.cc:306src/tree/updater_histmaker.cc:306: hist[44]=2.65957
[22:00:06] src/tree/updater_histmaker.cc:306: hist[45]=2.94154
[22:00:06] src/tree/updater_histmaker.cc:306: hist[46]=3.22254
[22:00:06] src/tree/updater_histmaker.cc:306: hist[47]=3.63618
[22:00:06] src/tree/updater_histmaker.cc:306: hist[48]=4.03017
[22:00:06] src/tree/updater_histmaker.cc:306: hist[49]=4.51524]=src/tree/updater_histmaker.cc:306: hist[1]=0.0109873
[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.0165509[8]=]=0.274604
[22:00:06] src/tree/updater_histmaker.cc:306: hist[18]=0.301327
[22:00:06] src/tree/updater_histmaker.cc:306: hist[19]=0.332641
[22:00:06] src/tree/updater_histmaker.cc:306: hist[20]=0.365658
[22:00:0622:00:06] src/tree/updater_histmaker.cc:306: hist[1]=0.0740731
[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.0166282[22:00:06] src/tree/updater_histmaker.cc:306: hist[35]=2.2
[22:00:06] src/tree/updater_histmaker.cc:306: hist[36]=2.5
[22:00:06] src/tree/updater_histmaker.cc:306: hist[37]=3
[22:00:06] src/tree/updater_histmaker.cc:306: hist[38]=3.5
[22:00:06] src/tree/updater_histmaker.cc:306: hist[39]=4
[22:00:06] src/tree/updater_histmaker.cc:306: hist[40]=4.5
[22:00:06] src/tree/updater_histmaker.cc:306: hist[4122:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.0185698
306: hist[18]=0.757224
]=306: hist[3]=0.150501
[22:00:06] src/tree/updater_histmaker.cc:306: hist[4]=0.183638
[22:00:06] src/tree/updater_histmaker.cc:306: hist[22:00:060.0116438[22:00:06] src/tree/updater_histmaker.cc:306: hist[9]=0.0909081
[22:00:06] src/tree/updater_histmaker.cc:306: hist[10]=22:00:06] src/tree/updater_histmaker.cc:306: hist[] src/tree/updater_histmaker.cc:306: :306
[22:00:06] src/tree/updater_histmaker.cc:306: hist[7]=0.0645151
[22:00:06] src/tree/updater_histmaker.cc:306: hist[8]=0.0769221
[22:00:06] src/tree/updater_histmaker.cc:306: hist[9]=0.0877183
[22:00:06] src/tree/updater_histmaker.cc:306: hist[10]=0.103447
[22:00:06] src/tree/updater_histmaker.cc:306: hist[0]=0.00319787
] src/tree/updater_histmaker.cc:306: hist[29]=0.62295
[22:00:06] src/tree/updater_histmaker.cc:306: hist[30]=src/tree/updater_histmaker.cc:306: hist[1]=0.0126034
] src/tree/updater_histmaker.cc:306: hist[10]=0.079999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[11]=0.0952371
[22:00:06] src/tree/updater_histmaker.cc:306: hist[12]=0.11111
[22:00:06] src/tree/updater_histmaker.cc:306[22:00:06] src/tree/updater_histmaker.cc:306: hist[19]=0.927999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[20]=0.933332
[22:00:06] src/tree/updater_histmaker.cc:306: hist[0.666666
]=5
[22:00:06] src/tree/updater_histmaker.cc:306: hist[42]=6
[22:00:06] src/tree/updater_histmaker.cc:306: hist[43]=[22:00:06] src/tree/updater_histmaker.cc306: hist[9]=0.0935242
[22:00:06] src/tree/updater_histmaker.cc:306: hist[10]=0.11111
[22:00:06: ]=] src/tree/updater_histmaker.cc:306: hist[1]=[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.0264712
[22:00:06] src/tree/updater_histmaker.cc:306: hist[4[22:00:06] src/tree/updater_histmaker.cc:306: hist[4]=0.0274779
[22:00:06] src/tree/updater_histmaker.cc:306: hist[5]=0.038004
[22:00:06] src/tree/updater_histmaker.cc:306: hist[6]=0.0354537
21]=0.941999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[22]=0.947499
[22:00:06] src/tree/updater_histmaker.cc:306: hist[23]=0.953389
[22:00:06] src/tree/updater_histmaker.cc:[: hist[13]=0.124999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[14]=0.142856
[22:00:06] src/tree/updater_histmaker.cc:306: hist[15]=0.166666
[22:00:06] src/tree/updater_histmaker.cc:306: hist[16]=0.18644
[22:00:06] src/tree/updater_histmaker.cc:306: hist[17]=0.205881
[22:00:06] src/tree/updater_histmaker.cc:306: hist[18]=0.238094
[22:00:06] src/tree/updater_histmaker.cc:306: hist[19]=0.249999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[20]=0.294117
[22:00:06] src/tree/updater_histmaker.cc:306: hist[21]=0.333332
[22:00:06] src/tree/updater_histmaker.cc:306: hist[22]=0.380951
[22:00:06] src/tree/updater_histmaker.cc:306: hist[23]=0.421874
[22:00:06] src/tree/updater_histmaker.cc:306: hist[24]=0.499999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[25]=0.546874
[22:00:06] src/tree/updater_histmaker.cc:306: hist[26]=0.596773
[22:00:06] src/tree/updater_histmaker.cc:306: hist[27]=0.686566
[22:00:06] src/tree/updater_histmaker.cc:306: hist[28]=0.749999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[29]=0.833332
[22:00:06] src/tree/updater_histmaker.cc:306: hist[30]=0.973509
[22:00:06] src/tree/updater_histmaker.cc:306: hist[31]=0.999999
[22:00:06] src/tree/updater_histmaker.cc:306:
[22:00:06] src/tree/updater_histmaker.cc:306
[22:00:06]
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.0789464
[22:00:06] src/tree/updater_histmaker.cc:306: hist[7]=0.0945664
[22:00:06] src/tree/updater_histmaker.cc:306: hist[8]=0.11111
[22:00:06] src/tree/updater_histmaker.cc:306: hist[9]=0.124999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[10]=0.142856
[22:00:06] src/tree/updater_histmaker.cc:306: hist[11]=0.166666
[22:00:06] src/tree/updater_histmaker.cc:306: hist[12]=0.181817
[22:00:06] src/tree/updater_histmaker.cc:306: hist[13]=0.199999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[14]=0.222221
[22:00:06] src/tree/updater_histmaker.cc:306: hist[15]=0.249999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[16]=] src/tree/updater_histmaker.cc:306: hist[21]=0.397101
[22:00:06] src/tree/updater_histmaker.cc:306: hist[22]=0.433683
[22:00:06] src/tree/updater_histmaker.cc:306: hist[23]=0.471358
[22:00:06] src/tree/updater_histmaker.cc:306: hist[24]=0.517438
[22:00:06] src/tree/updater_histmaker.cc:306: hist[25]=0.554208
[22:00:06] src/tree/updater_histmaker.cc:306: hist[26]=0.605106
[22:00:06] src/tree/updater_histmaker.cc:306: hist[27]=0.653078
[22:00:06] src/tree/updater_histmaker.cc:306: hist[28]=0.700048
[22:00:06] src/tree/updater_histmaker.cc:306: hist[29]=0.760846
[22:00:06] src/tree/updater_histmaker.cc:306: hist[30]=0.823148
[22:00:06] src/tree/updater_histmaker.cc:306: hist[31]=0.880765
[22:00:060.00753272
[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.107142
[
[22:00:06] src/tree/updater_histmaker.cc:306: hist[50]=5.04915
hist[0]=0.0024768
[22:00:06] src/tree/updater_histmaker.cc:306: hist[1]=0.00753515: hist[2
[22:00:06] src/tree/updater_histmaker.cc::30622:00:06] 7]=0.0851054[22:00:06] src/tree/updater_histmaker.cc:306: hist[31]=5]=0.217686
[22:00:06] src/tree/updater_histmaker.cc:306: hist[6]=0.239532
[22:00:06] src/tree/updater_histmaker.cc:306: hist[7]=0.269503
[22:00:06] src/tree/updater_histmaker.cc:306: hist[8]=0.294895
[22:00:06] src/tree/updater_histmaker.cc:306: hist[9]=0.328719
[22:00:06] src/tree/updater_histmaker.cc:306: hist[10]=0.350916
[22:00:06] src/tree/updater_histmaker.cc:306: hist[11]=0.37864
[22:00:06] src/tree/updater_histmaker.cc:306: hist[12]=0.410525
[22:00:06] src/tree/updater_histmaker.cc:306: hist[13]=0.438943
[22:00:06] src/tree/updater_histmaker.cc:306: hist[14]=0.464911
[22:00:06] src/tree/updater_histmaker.cc:306: hist[15]=0.496402
[22:00:06] src/tree/updater_histmaker.cc:306: hist[16]=0.521738
[22:00:06] src/tree/updater_histmaker.cc:306: hist[17]=0.548468
[22:00:06] src/tree/updater_histmaker.cc:306: hist[18]=0.577817
[22:00:06] src/tree/updater_histmaker.cc:306: hist[19]=: hist[3]=0.0251511
[22:00:06] src/tree/updater_histmaker.ccsrc/tree/updater_histmaker.cc:6.5
[22:00:06] src/tree/updater_histmaker.cc:306: hist[44]=7
[22:00:06] src/tree/updater_histmaker.cc:306: hist[45]=8.41667
[22:00:06] src/tree/updater_histmaker.cc:306: hist[src/tree/updater_histmaker.cc:306: hist[11]=0.117646
[22:00:06] src/tree/updater_histmaker.cc:306: hist[12]=0.133332
[22:00:06] src/tree/updater_histmaker.cc:306: hist[13]=0.148147
[22:00:06] src/tree/updater_histmaker.cc:306: hist[14]=0.166666
[22:00:06] src/tree/updater_histmaker.cc:306: hist[15]=0.182795
[22:00:06] src/tree/updater_histmaker.cc:306: hist[16]=0.199999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[17]=0.230768
[22:00:06] src/tree/updater_histmaker.cc:306: hist[18]=0.249999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[19]=0.266666
[22:00:06] src/tree/updater_histmaker.cc:306: hist[20]=0.304347
[22:00:06] src/tree/updater_histmaker.cc:306: hist[21]=0.333332
[22:00:06] src/tree/updater_histmaker.cc:306: hist[22]=0.362318
[22:00:06] src/tree/updater_histmaker.cc:306: hist[23]=0.40909
[22:00:06] src/tree/updater_histmaker.cc:] 0.718749
0.0405925
[22:00:06] src/tree/updater_histmaker.cc:306: hist[6]=0.0497561
[22:00:06] src/tree/updater_histmaker.cc:306: hist[7]=: hist[0.0228948
[22:00:06306: hist[24]=0.958591
[22:00:06] src/tree/updater_histmaker.cc:306: hist[25]=0.96436
[22:00:06] src/tree/updater_histmaker.cc:306: hist[26]=0.969564
[22:00:06] src/tree/updater_histmaker.cc:306: hist[27]=0.974025
[22:00:06] src/tree/updater_histmaker.cc:306: hist[28]=0.979999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[29]=0.985221
[22:00:06] src/tree/updater_histmaker.cc:306: hist[30]=hist[32]=1.12703
[22:00:06] src/tree/updater_histmaker.cc:306: hist[33]=1.24265
[22:00:06] src/tree/updater_histmaker.cc:306: hist[34]=1.3125
[22:00:06] src/tree/updater_histmaker.cc:306: hist[35]=1.47619
[22:00:06] src/tree/updater_histmaker.cc:306: hist[36]=1.66667
[22:00:06] src/tree/updater_histmaker.cc:306: hist[37]=1.86364
[22:00:06] src/tree/updater_histmaker.cc:306: hist[38]=2]=0.0119038
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.0186325
[22:00:06] src/tree/updater_histmaker.cc:306: hist[4]=0.0259218
[22:00:06] src/tree/updater_histmaker.cc:306: hist[5]=0.0344818
[[22:00:06] src/tree/updater_histmaker.cc:306: hist[32]=0.775861
[22:00:06] src/tree/updater_histmaker.cc:306: hist[33]=0.847644
[22:00:06] src/tree/updater_histmaker.cc:306: hist[34]=0.914215
[22:00:06] src/tree/updater_histmaker.cc:306: hist[35]=0.991999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[36]=1.04383
[22:00:06] src/tree/updater_histmaker.cc:306: hist[37]=1.14729
[22:00:06] src/tree/updater_histmaker.cc:306: hist[38]=1.23478
[22:00:06] src/tree/updater_histmaker.cc:306: hist[39]=1.32877
[22:00:06] src/tree/updater_histmaker.cc:306: hist[40]=1.46175
[22:00:06] src/tree/updater_histmaker.cc:306: hist[41]=1.55555
[22:00:06] src/tree/updater_histmaker.cc:306: hist[42]=1.69079
[22:00:06] src/tree/updater_histmaker.cc:306: hist[43]=1.86956
[22:00:06] src/tree/updater_histmaker.cc:306: hist[44]=2
[22:00:06] src/tree/updater_histmaker.cc:306: hist[45]=2.2268
[22:00:06] src/tree/updater_histmaker.cc:306: 306: [22:00:06] src/tree/updater_histmaker.cc:306: hist[19]=src/tree/updater_histmaker.cc:306: hist[1]=0.124999

[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=-6.09953
0.607476
[22:00:06] src/tree/updater_histmaker.cc:306: hist[20]=0.636871
[22:00:06] src/tree/updater_histmaker.cc:306: hist[21]=0.666666
[22:00:06] src/tree/updater_histmaker.cc:306: hist[22]=0.700786
[22:00:06] src/tree/updater_histmaker.cc:306: hist[23]=0.728642
[22:00:06] src/tree/updater_histmaker.cc:306: hist[24]=0.763888
[22:00:06] src/tree/updater_histmaker.cc0.00262367
] src/tree/updater_histmaker.cc:306: hist[32]=0.952997
0.0234239
0.0120627
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.0183455
[22:00:06] src/tree/updater_histmaker.cc:[22:00:06] src/tree/updater_histmaker.cc:306: hist[2]=0.166666
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.181817

[22:00:06] src/tree/updater_histmaker.cc:306: hist[5]=0.039999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[6]=0.0495617
[22:00:06] src/tree/updater_histmaker.cc3060.263157
[22:00:06] src/tree/updater_histmaker.cc:306: hist[17]=0.294117
[
[22:00:06:306: hist[4]=0.0338164
[22:00:06] src/tree/updater_histmaker.cc:306: hist[5]=] src/tree/updater_histmaker.cc:306: hist[4]=0.0313151
[[22:00:06] src/tree/updater_histmaker.cc:306: hist[51]=5.80075
[22:00:06] src/tree/updater_histmaker.cc:306: hist[52]=6.6615
[22:00:06] src/tree/updater_histmaker.cc:306: hist[53]=7.54628
[22:00:06] src/tree/updater_histmaker.cc:306: hist[54]=8.9614
[22:00:06] src/tree/updater_histmaker.cc:306: hist[55]=10.8565
[22:00:06] src/tree/updater_histmaker.cc:306: hist[56]=13.2011
[22:00:06] src/tree/updater_histmaker.cc:306: hist[57]=16.0035
[22:00:06] src/tree/updater_histmaker.cc:0.779567
[22:00:06] src/tree/updater_histmaker.cc:306: hist[20]=0.798183
[22:00:06] src/tree/updater_histmaker.cc:306: hist[21]=0.817941
[22:00:06] src/tree/updater_histmaker.cc:306: hist[22]=:306[22:00:06] src/tree/updater_histmaker.cc:306: hist[33]=1.00624
[22:00:06] src/tree/updater_histmaker.cc:306: hist[34]=1.09754
[22:00:06] src/tree/updater_histmaker.cc:306: hist[35]=1.1797
[22:00:06] src/tree/updater_histmaker.cc:306: hist[36]=1.26501
[22:00:06] src/tree/updater_histmaker.cc:306: hist[37]=1.37703
1]=0.00855487
22:00:06] src/tree/updater_histmaker.cc:306: hist[7]=0.062499
[22:00:0646]=10
[22:00:06] src/tree/updater_histmaker.cc:306: hist[47]=12
[22:00:06] src/tree/updater_histmaker.cc:306: hist[48]=15
[22:00:06] src/tree/updater_histmaker.cc:306: hist[49]=20.1429
[22:00:06] src/tree/updater_histmaker.cc:306: hist[50]=31
[22:00:06] src/tree/updater_histmaker.cc:306: hist[51]=61
[22:00:06] src/tree/updater_histmaker.cc:306: hist[52]=inf
[22:00:06] src/tree/updater_histmaker.cc:306: hist[53]=inf
src/tree/updater_histmaker.cc:306: hist[2]=0.0206363
[22:00:06] src/tree/updater_histmaker.cc[[] src/tree/updater_histmaker.cchist[hist[2]=0.0144918
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=0.0223837
[22:00:06] src/tree/updater_histmaker.cc:306: hist[22:00:06] src/tree/updater_histmaker.cc:hist[18]=0.311102
[22:00:06] src/tree/updater_histmaker.cc:306: hist[19]=0.339946
[22:00:06] src/tree/updater_histmaker.cc:306: hist[20]=0.371953
[22:00:06] src/tree/updater_histmaker.cc:306: hist[[22:00:06:306: hist[5]=0.0460232
[22:00:06] src/tree/updater_histmaker.cc:306: hist[] src/tree/updater_histmaker.cc:306: hist[8]=0.0727263
[22:00:06] src/tree/updater_histmaker.cc:306: hist[9]=0.0893236
[22:00:06] src/tree/updater_histmaker.cc:306: hist[10]=0.102661
[22:00:06] src/tree/updater_histmaker.cc:306: hist[11]=0.121211
[22:00:06] src/tree/updater_histmaker.cc:306: hist[12]=0.137379
[22:00:06] src/tree/updater_histmaker.cc:306: hist[13]=0.153845
[22:00:06] src/tree/updater_histmaker.cc:306: hist[14]=0.170212
[22:00:06] src/tree/updater_histmaker.cc:306: hist[15]=0.199999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[11]=0.124999
:306: hist[25]=0.794642
[22:00:06] src/tree/updater_histmaker.cc:306: hist[26]=0.822221
[22:00:06] src/tree/updater_histmaker.cc:306: hist[27]=0.857723
[22:00:06] src/tree/updater_histmaker.cc:306: hist[28]=0.892856
[22:00:06] src/tree/updater_histmaker.cc:306: hist[29]=0.925925
[22:00:06] src/tree/updater_histmaker.cc:306: hist[30]=0.964717
[22:00:06] src/tree/updater_histmaker.cc:306: hist[31]=0.999999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[32]=1.03529
[22:00:06] src/tree/updater_histmaker.cc:306: hist[33]=1.07917
[22:00:06] src/tree/updater_histmaker.cc:306: hist[34]=1.11382
[22:00:06]
[22:00:06] src/tree/updater_histmaker.cc:306: hist[39]=2.22456
[22:00:06] src/tree/updater_histmaker.cc:306: hist[40]=2.5
[22:00:06] src/tree/updater_histmaker.cc:306: hist[41]=2.80097
[22:00:06] src/tree/updater_histmaker.cc:306: hist[42]=3.25
[22:00:06] src/tree/updater_histmaker.cc:306: hist[43]=3.7619
[22:00:06] src/tree/updater_histmaker.cc:306: hist[44]=4.2
]=0.0474567
[22:00:06] src/tree/updater_histmaker.cc:306: hist[7]=0.0604327
[22:00:06] src/tree/updater_histmaker.cc:306: hist[8]=0.0744957
[22:00:06] src/tree/updater_histmaker.cc:306: hist[9]=0.0902771
[22:00:06] src/tree/updater_histmaker.cc:306: hist[306: hist[[22:00:06] src/tree/updater_histmaker.cc:306: hist[18]=0.333332
[22:00:06] src/tree/updater_histmaker.cc:306: hist[19]=0.363635
[22:00:06] src/tree/updater_histmaker.cc:306: hist[20]=0.399999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[21]=0.42857
[22:00:06] src/tree/updater_histmaker.cc:306: hist[22]=0.499999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[23]=0.548386
[22:00:06] src/tree/updater_histmaker.cc:306: hist[24]=0.599999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[25]=0.666666
[22:00:06] src/tree/updater_histmaker.cc:306: hist[26]=0.749999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[27]=0.785713
[22:00:06] src/tree/updater_histmaker.cc:306: hist[28]=0.833332
[22:00:06] src/tree/updater_histmaker.cc:306: hist[29]=0.956521
[22:00:06] src/tree/updater_histmaker.cc:306: hist[30]=0.999999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[31]=1.03125
[22:00:06] src/tree/updater_histmaker.cc:306: hist[32]=1.16667
[22:00:06] src/tree/updater_histmaker.cc:306: hist[33]=1.26667
[22:00:06] src/tree/updater_histmaker.cc:306: hist[34]=1.33333
[22:00:06] src/tree/updater_histmaker.cc:306: hist[35]=1.5
[22:00:06] src/tree/updater_histmaker.cc:306: hist[36]=1.66667
[22:00:06] src/tree/updater_histmaker.cc:306: hist[37]=1.83333
[22:00:06] src/tree/updater_histmaker.cc:306: hist[38]=2
[22:00:06] src/tree/updater_histmaker.cc:306: hist[39]=2.33333
[22:00:06] src/tree/updater_histmaker.cc:306: hist[40]=2.54545
[22:00:06] src/tree/updater_histmaker.cc:306: hist[41]=3
[22:00:06] src/tree/updater_histmaker.cc:306: hist[42]=3.5
[22:00:06] src/tree/updater_histmaker.cc:306: hist[43]=4
[22:00:06:306: src/tree/updater_histmaker.cc:306: hist[13]=0.126463
[22:00:06] src/tree/updater_histmaker.cc:306: hist[14]=0.147818
[22:00:06] src/tree/updater_histmaker.cc:306: hist[15]=0.166666
[22:00:06] src/tree/updater_histmaker.cc:306: hist[16]=0.188376
[22:00:06] src/tree/updater_histmaker.cc:306: hist[17[22:00:06] src/tree/updater_histmaker.cc306: hist[58]=21.494
[22:00:06] src/tree/updater_histmaker.cc:306: hist[59]=28.62
[22:00:06] src/tree/updater_histmaker.cc:306: hist[60]=40.9485
[22:00:06] src/tree/updater_histmaker.cc:306: hist[61]=72.4772
[22:00:06] src/tree/updater_histmaker.cc:306: hist[62]=201.129
[22:00:06] src/tree/updater_histmaker.cc:306: hist[63]=inf
[22:00:06] src/tree/updater_histmaker.cc:306: hist[64]=inf
[22:00:06] src/tree/updater_histmaker.cc:306: hist[3]=-4.08219
[22:00:06] src/tree/updater_histmaker.cc:306: hist[4]=-2.95749
[22:00:06] src/tree/updater_histmaker.cc:306: hist[5]=-2.28378
[22:00:06] src/tree/updater_histmaker.cc:306: hist[6]=-1.92563
[22:00:06] src/tree/updater_histmaker.cc:306: hist[7]=-1.54386
[22:00:06] src/tree/updater_histmaker.cc:306: hist[8]=-1.31677
[22:00:06] src/tree/updater_histmaker.cc:306: hist[9]=-1.09502
[22:00:06] src/tree/updater_histmaker.cc:306: hist[10]=-0.873811
[22:00:06] src/tree/updater_histmaker.cc:306: hist[11]=-0.714287
[22:00:06] src/tree/updater_histmaker.cc:306: hist[12]=-0.591347
[22:00:06] src/tree/updater_histmaker.cc:306: hist[13]=-0.488126
[22:00:06] src/tree/updater_histmaker.cc:306: hist[14]=-0.382043
[22:00:06] src/tree/updater_histmaker.cc[22:00:06[22:00:06] dmlc-core/include/dmlc/logging.h:235: [22:00:06] src/tree/updater_histmaker.cc:308: fv=inf, hist.last=inf
0.835927
[22:00:06] src/tree/updater_histmaker.cc:306: hist[23]=0.85673
[22:00:06] src/tree/updater_histmaker.cc:306: hist[24]=0.874094
[22:00:064[22:00:06] src/tree/updater_histmaker.cc:306: hist[16]=0.214285
[22:00:06] src/tree/updater_histmaker.cc:306: hist[17]=0.249999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[18306: hist[35]=0.0427451
[22:00:06
[22:00:06] src/tree/updater_histmaker.cc:306: hist[8]=0.10294
[22:00:06] src/tree/updater_histmaker.cc:306: hist[9]=0.120207
[22:00:06] src/tree/updater_histmaker.cc:306: hist[10]=0.138297
[22:00:06] src/tree/updater_histmaker.cc:306: hist[11]=0.159532
[22:00:06] src/tree/updater_histmaker.cc:306: hist[12]=0.179411
[22:00:06] src/tree/updater_histmaker.cc:306: hist[13]=0.199999
[22:00:06] src/tree/updater_histmaker.cc:306: 22:00:06hist[346]=2.45454
[22:00:06] src/tree/updater_histmaker.cc:306: hist[47]=2.72289
[22:00:06] src/tree/updater_histmaker.cc:306: hist[48]=306: hist[24]=0.482758
[22:00:06] src/tree/updater_histmaker.cc:306: hist[25]=0.499999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[26]=0.599999
[22:00:061.15761
[22:00:06] src/tree/updater_histmaker.cc:306: hist[36]=0.99294
[22:00:06] src/tree/updater_histmaker.cc:306: hist[31]=0.999999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[32]=1.00452
[22:00:06] src/tree/updater_histmaker.cc:306: hist[33]=1.01304
[22:00:06] src/tree/updater_histmaker.cc:306: hist[34]=1.01896
[22:00:06] src/tree/updater_histmaker.cc:306: hist[35]=1.02459
[22:00:06] src/tree/updater_histmaker.cc:306: hist[36]=1.03093
[22:00:06] src/tree/updater_histmaker.cc:306: hist[37]=1.03555
[22:00:06] src/tree/updater_histmaker.cc:306: hist[38]=1.04167
[22:00:06] src/tree/updater_histmaker.cc:306: hist[39]=1.04822
[22:00:06] src/tree/updater_histmaker.cc:306: hist[40]=1.05263
[22:00:06] src/tree/updater_histmaker.cc:306: hist[41]=1.05932
[22:00:06] src/tree/updater_histmaker.cc:306: hist[42]=1.06838
[22:00:06] src/tree/updater_histmaker.cc:306: hist[43]=1.07296
[22:00:06] src/tree/updater_histmaker.cc:306: hist[44]=1.08193
[22:00:06] src/tree/updater_histmaker.cc:306: hist[45]=1.08932
[22:00:06] src/tree/updater_histmaker.cc:306: hist[46]=1.1
[22:00:06] src/tree/updater_histmaker.cc:306: hist[47]=1.11111
[22:00:06] src/tree/updater_histmaker.cc:306: hist[48]=1.11573
[22:00:06] src/tree/updater_histmaker.cc:306: hist[49]=1.12676
[22:00:06] src/tree/updater_histmaker.cc:306: hist[50]=1.14155
[10]=0.106413
[22:00:06] src/tree/updater_histmaker.cc:] ]=: hist[0.0603814
[22:00:06] src/tree/updater_histmaker.cc:306: hist[] src/tree/updater_histmaker.cc:306: hist[45]=5
[22:00:06] src/tree/updater_histmaker.cc:306: hist[46]=5.56044
[22:00:0622:00:06] src/tree/updater_histmaker.cc:306: hist[12]=:306: hist[1]=0.00729827
[22:00:06] src/tree/updater_histmaker.cc8]=: hist[4]=0.0273544
3
:306: hist[15]=-0.292888
[22:00:06] src/tree/updater_histmaker.cc:306: hist[16]=-0.23077
[22:00:06] src/tree/updater_histmaker.cc:306: hist[17]=-0.161881
[22:00:06] src/tree/updater_histmaker.cc:[22:00:06] src/tree/updater_histmaker.cc:306: hist[6]=0.0539135
[22:00:06] src/tree/updater_histmaker.cc:306: hist[7]=0.0638435
6]=0.0567096
[22:00:06] src/tree/updater_histmaker.cc:306: hist[7]=0.0667048
[22:00:06] src/tree/updater_histmaker.cc:306: hist[8]=0.0829348
[22:00:06] src/tree/updater_histmaker.cc:306: hist[9]=0.0966214
[22:00:06] src/tree/updater_histmaker.cc:306: hist[10]=0.109863
[22:00:06] src/tree/updater_histmaker.cc:306: hist[11]=0.126529
[22:00:06] src/tree/updater_histmaker.cc:306: hist[12]=]=0.212669
[22:00:06] src/tree/updater_histmaker.cc:306: hist[18]=0.235293
[22:00:06] src/tree/updater_histmaker.cc:306: hist[19]=0.260273
[22:00:06] src/tree/updater_histmaker.cc:306: hist[20]=0.289679
[22:00:06] src/tree/updater_histmaker.cc:306: hist[21]=0.319402
[22:00:06] src/tree/updater_histmaker.cc:306: hist[22]=0.349881
[22:00:06] src/tree/updater_histmaker.cc:306: hist[23]=0.384277
[22:00:06] src/tree/updater_histmaker.cc:306: hist[24]=0.422559
22:00:06] src/tree/updater_histmaker.cc:306: hist[] src/tree/updater_histmaker.cc:306: hist[440.142856
[22:00:06] src/tree/updater_histmaker.cc:306: hist[13]=0.166666
[22:00:06] src/tree/updater_histmaker.cc:306: hist[14]=0.181817
[22:00:06] src/tree/updater_histmaker.cc:306: hist[15]=0.199999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[16]=51]=1.155
[22:00:06] src/tree/updater_histmaker.cc:3062]=[22:00:06] src/tree/updater_histmaker.cc:306: hist[38]=] src/tree/updater_histmaker.cc:306: hist[2]=0.0142335
1.2125
[22:00:06] src/tree/updater_histmaker.cc:306: hist[37]=1.25588
[22:00:06] src/tree/updater_histmaker.cc:306: hist[38]=1.31944
[22:00:06] src/tree/updater_histmaker.cc:306: hist[39]=1.36796
[22:00:06] src/tree/updater_histmaker.cc:306: hist[40]=1.42857
[22:00:06] src/tree/updater_histmaker.cc:306: hist[41]=[22:00:06] src/tree/updater_histmaker.cc:306: hist[49]=3.37815
[22:00:06] src/tree/updater_histmaker.cc:306: ]=src/tree/updater_histmaker.cc:306: hist[4]=0.199999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[5]=0.235293
[22:00:060.073724
[22:00:06] src/tree/updater_histmaker.cc:306: hist[9]=0.0885428
[22:00:06] src/tree/updater_histmaker.cc:306: 306: hist[0.015893
:306: hist[4]=0.0327859
terminate called after throwing an instance of 'dmlc::Error'
] src/tree/updater_histmaker.cc:306: hist[25]=0.889275
[22:00:06] src/tree/updater_histmaker.cc:306: hist[26]=0.902837
[22:00:06] src/tree/updater_histmaker.cc:306: hist[27]=0.919999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[28]=0.934447
[22:00:06] src/tree/updater_histmaker.cc:306: hist[29]=0.949999
[22:00:06] src/tree/updater_histmaker.cc:306: hist[30]=0.961455
[22:00:06] src/tree/updater_histmaker.cc:306: hist[31]=0.975186
[22:00:06] src/tree/updater_histmaker.cc:306: hist[32]=0.986714
[22:00:06] src/tree/updater_histmaker.cc:306: hist[33]=0.999533
[22:00:06] src/tree/updater_histmaker.cc:306: hist[34]=1.00573
[22:00:06] src/tree/updater_histmaker.cc:306: hist[35]=  what():  [22:00:06] src/tree/updater_histmaker.cc:308: fv=inf, hist.last=inf
hist[10]=0.104608
[22:00:06] zsh: abort      python xgb_data.py

[tqchen]

OK that fix deletes the NAN but did not delete inf. One way you can resolve that is to remove inf manually from your dataset.

[willettk]

I pre-processed the data to replace np.inf with np.nan, and XGBoost is churning away now (slowly, since it's a massive dataset). But the histmaker errors above are gone. Many thanks, @tqchen.

In the interest of posterity/curiosity - do you know why the errors with NaN/inf seem to only be thrown with very large datasets? I'm unclear on why this works when I restrict either the number of rows or columns in the training set.

[tqchen]

It should also be thrown on small dataset, but probably there are only few of them in the data

[tqchen]

Another reason is that xgboost automatically switchs to histogram algorithm which get hit by the error, the default exact greedy kinds of hide it away, but still could get affected. BTW, you might be interested in trying https://github.com/dmlc/xgboost/issues/1950

[willettk]

Interesting - I'll see what the tree_method="hist" gives as far as both runtime and performance. Thanks, @tqchen.