Gluon-nlp: BERT produces slightly different values for different batch sizes.

[davisliang]

Description

For two inputs, the first a batch size of 300, and the second a 27 batch slice of the first, we get _slightly different_ results from BERT.

Error Message

Here are the outputs from the first item of the batch for batch size 300 and 27, respectively.

BATCH SIZE 300:
[[-0.2168 0.2137 -0.05142 ... 0.07556 0.774 -0.1343 ]
[-0.23 -0.001999 0.1725 ... -0.1262 0.6 -0.2517 ]
[-0.4565 0.1788 0.6187 ... -0.2104 0.2776 -0.5166 ]
...
[-0.10944 0.2292 0.858 ... -0.3677 0.3657 -0.4097 ]
[-0.1173 0.29 0.888 ... -0.2922 0.4011 -0.4702 ]
[-0.2341 0.3228 0.9746 ... -0.3906 0.3662 -0.5415 ]]

BATCH SIZE 27:
[[-0.2178 0.2146 -0.0519 ... 0.075 0.7734 -0.1339 ]
[-0.2301 -0.002167 0.1725 ... -0.1265 0.5986 -0.2512 ]
[-0.4558 0.178 0.619 ... -0.2117 0.2761 -0.517 ]
...
[-0.10913 0.2301 0.857 ... -0.3677 0.3647 -0.41 ]
[-0.1176 0.289 0.887 ... -0.2927 0.401 -0.471 ]
[-0.2358 0.3223 0.973 ... -0.392 0.3662 -0.5415 ]]

To Reproduce

!pip install mxnet-cu102==1.7.0
!pip install gluonnlp==0.10.0

%env MXNET_SAFE_ACCUMULATION=1
import mxnet
from mxnet import nd
from gluonnlp.model import bert
import gluonnlp as nlp; import mxnet as mx;

# load model
model, vocab = nlp.model.get_model('bert_12_768_12', ctx=mx.gpu(0), dataset_name='book_corpus_wiki_en_uncased', use_pooler=True, use_classifier=False, use_decoder=False);
tokenizer = nlp.data.BERTTokenizer(vocab, lower=True);
transform = nlp.data.BERTSentenceTransform(tokenizer, max_seq_length=512, pair=False, pad=False);
model.cast('float16')

# construct examples with batch size of 300 and first 27 out of 300.
mini_length=27
input_batch = nd.random.uniform(-10,10,(300,100), dtype='float16').as_in_context(mx.gpu(0))
input_batch_mini = input_batch[:mini_length] 

valid_len = nd.array([100]*300, dtype='float16').as_in_context(mx.gpu(0))
segments = nd.zeros((300,100), dtype='float16').as_in_context(mx.gpu(0))
out = model(input_batch, segments, valid_len)
print("Batch Size 300: ", out[0][1])

valid_len_mini = nd.array([100]*mini_length, dtype='float16').as_in_context(mx.gpu(0))
segments_mini = nd.zeros((mini_length,100), dtype='float16').as_in_context(mx.gpu(0))
out_mini = model(input_batch_mini, segments_mini, valid_len_mini)
print("Batch Size 27: ", out_mini[0][1])

Steps to reproduce

(Paste the commands you ran that produced the error.)

1. See above. This can be run directly in a Jupyter Notebook.

What have you tried to solve it?

1. Tested it on MxNet1.7 + GluonNLP0.10.0, also tried MxNet1.6 and GluonNLP 0.9
2. The issue also persists for fp32, but it is not as significant as for fp16.
   

   Environment

Clean EC2 instance with latest Deep Learning AMI.

We recommend using our script for collecting the diagnositc information. Run the following command and paste the outputs below:

curl --retry 10 -s https://raw.githubusercontent.com/dmlc/gluon-nlp/master/tools/diagnose.py | python

# paste outputs here

[sxjscience]

I can reproduce the problem and I can make sure that if we use "fp32", the precision is higher. One possibility is that by changing the batchsize, CUBLAS is picking different internal GEMM implementations.

[sxjscience]

@davisliang I think it might be related to the internal implementation of CUBLAS. For example, batch_size=200 and batch_size=300 have the same results:

import mxnet
from mxnet import nd
from gluonnlp.model import bert
import gluonnlp as nlp; import mxnet as mx;

# load model
model, vocab = nlp.model.get_model('bert_12_768_12', ctx=mx.gpu(0), dataset_name='book_corpus_wiki_en_uncased', use_pooler=True, use_classifier=False, use_decoder=False);
tokenizer = nlp.data.BERTTokenizer(vocab, lower=True);
transform = nlp.data.BERTSentenceTransform(tokenizer, max_seq_length=512, pair=False, pad=False);
model.cast('float16')

# construct examples with batch size of 300 and first 27 out of 300.
mini_length=200
input_batch = nd.random.uniform(-10,10,(300,100), dtype='float16').as_in_context(mx.gpu(0))
input_batch_mini = input_batch[:mini_length] 

valid_len = nd.array([100]*300, dtype='float16').as_in_context(mx.gpu(0))
segments = nd.zeros((300,100), dtype='float16').as_in_context(mx.gpu(0))
out = model(input_batch, segments, valid_len)
print("Batch Size 300: ", out[0][1])

valid_len_mini = nd.array([100]*mini_length, dtype='float16').as_in_context(mx.gpu(0))
segments_mini = nd.zeros((mini_length,100), dtype='float16').as_in_context(mx.gpu(0))
out_mini = model(input_batch_mini, segments_mini, valid_len_mini)
print("Batch Size 200: ", out_mini[0][1])

Output:

Batch Size 300:  
[[ 0.04575   0.1301    0.2268   ... -0.08203  -0.02853  -0.01185 ]
 [ 0.1333    0.253     0.4124   ...  0.0467    0.2424    0.006683]
 [ 0.249     0.3232    0.3547   ... -0.2207    0.1804    0.004776]
 ...
 [ 0.3464    0.55      0.4507   ... -0.1503    0.0838    0.1586  ]
 [ 0.3286    0.619     0.521    ... -0.1721    0.02725   0.1776  ]
 [ 0.2197    0.6885    0.658    ... -0.246    -0.00241   0.2101  ]]
<NDArray 100x768 @gpu(0)>
Batch Size 27:  
[[ 0.04575   0.1301    0.2268   ... -0.08203  -0.02853  -0.01185 ]
 [ 0.1333    0.253     0.4124   ...  0.0467    0.2424    0.006683]
 [ 0.249     0.3232    0.3547   ... -0.2207    0.1804    0.004776]
 ...
 [ 0.3464    0.55      0.4507   ... -0.1503    0.0838    0.1586  ]
 [ 0.3286    0.619     0.521    ... -0.1721    0.02725   0.1776  ]
 [ 0.2197    0.6885    0.658    ... -0.246    -0.00241   0.2101  ]]
<NDArray 100x768 @gpu(0)>

I think that CUBLAS is doing something special when the shape of the matrix is small. @MoisesHer would you think that it's attributed to CUBLAS?

[davisliang]

That's what we initially thought too. However, the difference (for fp16 in the 3rd significant digit) is quite large (which I didn't expect from just choosing a different GEMM implementation) and could significantly impact the final outputs.

[sxjscience]

I agree. Looks like something else is wrong:

import mxnet
import os
from mxnet import nd
from gluonnlp.model import bert
import numpy.testing as npt
import gluonnlp as nlp; import mxnet as mx;

os.environ['MXNET_USE_FUSION'] = '0'
os.environ['MXNET_SAFE_ACCUMULATION'] = '1'

data_type = 'float16'

# load model
model, vocab = nlp.model.get_model('bert_12_768_12', ctx=mx.gpu(0), dataset_name='book_corpus_wiki_en_uncased', use_pooler=True, use_classifier=False, use_decoder=False);
model.cast(data_type)

# construct examples with batch size of 300 and first 27 out of 300.
mini_length=2
input_batch = nd.random.uniform(-10,10,(300,100), dtype=data_type).as_in_context(mx.gpu(0))
input_batch_mini = input_batch[:mini_length] 

valid_len = nd.array([100]*300, dtype=data_type).as_in_context(mx.gpu(0))
segments = nd.zeros((300,100), dtype=data_type).as_in_context(mx.gpu(0))
out = model(input_batch, segments, valid_len)
print("Batch Size 300: ", out[0][1])

valid_len_mini = nd.array([100]*mini_length, dtype=data_type).as_in_context(mx.gpu(0))
segments_mini = nd.zeros((mini_length,100), dtype=data_type).as_in_context(mx.gpu(0))
out_mini = model(input_batch_mini, segments_mini, valid_len_mini)
print("Batch Size 2: ", out_mini[0][1])

npt.assert_allclose(out[0][1].asnumpy(), out_mini[0][1].asnumpy(), 1E-4, 1E-4)

Output:

AssertionError: 
Not equal to tolerance rtol=0.0001, atol=0.0001

Mismatched elements: 62389 / 76800 (81.2%)
Max absolute difference: 0.01758
Max relative difference: 56.
 x: array([[-0.2172  ,  0.2144  , -0.05106 , ...,  0.07526 ,  0.772   ,
        -0.1335  ],
       [-0.2283  , -0.001389,  0.172   , ..., -0.1278  ,  0.6006  ,...
 y: array([[-0.2181  ,  0.2146  , -0.05167 , ...,  0.0749  ,  0.7734  ,
        -0.1337  ],
       [-0.2291  , -0.001083,  0.1732  , ..., -0.128   ,  0.5996  ,...

[MoisesHer]

Hi, I was able to reproduce the issue on a G4 instance.
I also checked and the issue does not happen on P3 instances (V100).
My opinion is that different accumulation strategies across K dim is causing this behavior.

I looked at the cuBLAS kernels being called on G4:
BS300 (BatchSize 300)

• 24 x turing_fp16_s1688gemm_fp16_128x256_ldg8_f2f_tn (outProjection & FFN2)
• 12 x turing_fp16_s1688gemm_fp16_256x128_ldg8_f2f_tn (FFN1)
• 12 x turing_fp16_s1688gemm_fp16_128x128_ldg8_f2f_tn (q/k/v projection)
• 12 x volta_fp16_sgemm_fp16_128x64_nn (Soft x Value)
• 12 x volta_fp16_sgemm_fp16_128x64_tn (QxK)

BS27 (BatchSize 27)

• 24 x turing_fp16_s1688gemm_fp16_128x128_ldg8_f2f_tn (q/k/v projection & FFN2)
• 12 turing_fp16_s1688gemm_fp16_256x128_ldg8_f2f_tn (FFN1)
• 12 turing_fp16_s1688gemm_fp16_128x256_ldg8_f2f_tn (outProjection)
• 12 volta_fp16_sgemm_fp16_128x64_nn (Soft x Value)
• 12 volta_fp16_sgemm_fp16_128x64_tn (QxK)

The only difference is computing the second GEMM in FeedForward (FFN2).
BS300 uses different tile size (MxN subdivisions) compared to BS27 case.
Normally, accumulation across K dimension is computed by a single thread, for any tile size in principle.
So that should not cause the different results here.

However, when K dim is large compared to M and N, the algorithm splits it into several chunks to improve performance
(documented in https://docs.nvidia.com/cuda/cublas/index.html#gemm-algorithms).

In this case, FFN2 dimensions are:
M: 128*BatchSize,
N: 768
K: 3072
And with BS27, K is considered large compare to M and N. Therefor K is split.
If for the GEMMs you use FP32 accumulation (default mode) and FP16 inputs, each intermediate output is cast to FP16, and finally all the intermediates are summed up in a determinism way.
Here, several cast ops may lead to a different output compared to a single cast (when K is not split).

Those cast can be avoided if you use the same accumulation data-type as input data-type, i.e. FP16, by doing:
export MXNET_FC_TRUE_FP16=1.

[davisliang]

Thank you, @MoisesHer, for the quick response, deep analysis, and effective solution!

One minor point: I ran into the same problem running the above code on a p3.24dn.xlarge on Driver Version: 440.33.01, CUDA Version: 10.2. However, using export MXNET_FC_TRUE_FP16 fixed the issue there as well.

Thanks again. Resolving this ticket as solved.

[davisliang]

@MoisesHer, I found that this issue still persists if even smaller batch sizes (e.g. 5) is used. Are you able to reproduce this? I'm guessing it's another GEMM that's causing the issue. Would there be an easy way to prevent this chunking behavior or would this be very detrimental to performance?

[MoisesHer]

Hi Davis,
I just tested on G4, but comparing BS300 vs BS5 I get same results:

Batch Size 300:  
[[-0.2183   0.2122  -0.0538  ...  0.07495  0.77    -0.1342 ]
 [-0.2307  -0.00264  0.1754  ... -0.1316   0.5967  -0.2522 ]
 [-0.4597   0.1744   0.619   ... -0.2089   0.2754  -0.5156 ]
 ...
 [-0.1115   0.2285   0.8613  ... -0.3718   0.3672  -0.4097 ]
 [-0.1188   0.2896   0.89    ... -0.294    0.4006  -0.4697 ]
 [-0.2388   0.3208   0.9727  ... -0.3904   0.3638  -0.543  ]]
<NDArray 100x768 @gpu(0)>
Batch Size 5:  
[[-0.2183   0.2122  -0.0538  ...  0.07495  0.77    -0.1342 ]
 [-0.2307  -0.00264  0.1754  ... -0.1316   0.5967  -0.2522 ]
 [-0.4597   0.1744   0.619   ... -0.2089   0.2754  -0.5156 ]
 ...
 [-0.1115   0.2285   0.8613  ... -0.3718   0.3672  -0.4097 ]
 [-0.1188   0.2896   0.89    ... -0.294    0.4006  -0.4697 ]
 [-0.2388   0.3208   0.9727  ... -0.3904   0.3638  -0.543  ]]
<NDArray 100x768 @gpu(0)>

But in any case, I think that even if there are not fp32- fp16 casts, since accumulation strategy is different, what you are observing may happens.
A single thread accumulating(1...N), may end with different results than
accumulation(1...N/2) + accumulation(N/2+1 .... N), specially with lower precision types.

[sxjscience]

@davisliang How many points have you lost due to this error?

[gpu]

Some usernames may cause you to receive potentially interesting notifications :-)

The discussion here is on a very high technical level, and I cannot contribute anything, except for, maybe, leaving a pointer to the "batched" functions at https://docs.nvidia.com/cuda/cublas/index.html#cublas-lt-t-gt-gemmbatched that (to my limited understanding) have been introduced exactly for the use case of doing a particularly efficient GEMM for many ("small") matrices - not sure whether this is relevant for you, though...

[davisliang]

@davisliang How many points have you lost due to this error?

Hi @sxjscience, we find that the difference is quite small. Marking as resolved, thanks all!