Spark-nlp: bert sentence embeddings always yield same result

[CyborgDroid]

The sentence_embeddings for bert are broken, the output is the same for all sentences.

from pyspark.sql.functions import *
from pyspark.sql.types import *

from sparknlp.base import *
from sparknlp.annotator import *
from sparknlp.common import *

data = [
  ('Game of Thrones',1, 1, "In the Seven Kingdoms of Westeros, a soldier of the ancient Night's Watch order survives an attack by supernatural creatures known as the White Walkers, thought until now to be mythical."),
  ('Game of Thrones',1,2,"In King's Landing, the capital, Jon Arryn, the King's Hand, dies under mysterious circumstances."),
  ('Game of Thrones',3,1,"Tyrion makes saddle modifications for Bran that will allow the paraplegic boy to ride.")
]

df = spark.createDataFrame(data, ['series','episode','sentence_number','text'])

display(df)

document_assembler = DocumentAssembler() \
    .setInputCol('text')

word_embeddings = BertEmbeddings.pretrained('bert_uncased', 'en')\
    .setInputCols(["document"])\
    .setOutputCol('bert')

pipeline = Pipeline().setStages([document_assembler, word_embeddings])

bert_df = pipeline.fit(df).transform(df)

bert_df = bert_df.withColumn('bert', col('bert')[0]['sentence_embeddings']).drop('document')

display(bert_df)

output is the data + a column with an array of bert vectors

bert_vectors = bert_df.select(*bert_df.columns, posexplode('bert').alias('bert_id', 'bert_item')).drop('bert')\
        .groupby('series', 'episode', 'sentence_number').pivot('bert_id').agg(first('bert_item'))\
        .drop('series', 'episode', 'sentence_number')

output is 3 rows with all the sentence vectors in different columns

display(bert_vectors.distinct())

output is one row since all the vectors are the same

[maziyarpanahi]

Hi @CyborgDroid,

Is this the case if you use bert_multi_cased? Just want to be sure if it's the model or it's the way we trained it.

word_embeddings = BertEmbeddings.pretrained('bert_multi_cased', 'xx')\
    .setInputCols(["document"])\
    .setOutputCol('bert')

[CyborgDroid]

Hi Maziyar,

BertEmbeddings.pretrained('bert_multi_cased', 'xx') yields the same result.

print(bert_vectors.collect())

output:

[Row(0=-0.3828985393047333, 1=0.12982381880283356, 2=-0.15237723290920258, 3=-0.30286455154418945, 4=0.07984059303998947, 5=0.24567967653274536, 6=-0.4774293005466461, 7=0.11368699371814728, 8=-0.43600544333457947, 9=0.0015827585011720657, 10=-0.18087996542453766, 11=0.012072999961674213, 12=0.44286948442459106, 13=0.3444838225841522, 14=-0.10974393784999847, 15=0.2576451003551483, 16=0.34980666637420654, 17=0.008537352085113525, 18=0.3259563148021698, 19=-0.05231372267007828, 20=0.2902263402938843, 21=0.46089696884155273, 22=-0.1315918117761612, 23=-0.6931538581848145, 24=-0.12955236434936523, 25=0.12681958079338074, 26=-0.3282795250415802, 27=0.3309135138988495, 28=0.18025650084018707, 29=-0.17974373698234558, 30=-0.12690448760986328, 31=0.2799723744392395, 32=-0.30506038665771484, 33=0.2364874929189682, 34=-0.06421627104282379, 35=-0.4623970091342926, 36=-0.4120138883590698, 37=0.29458779096603394, 38=0.22279199957847595, 39=0.13119202852249146, 40=0.23632945120334625, 41=0.3276778757572174, 42=-0.09378167986869812, 43=-0.4410403370857239, 44=-0.27837029099464417, 45=0.687725305557251, 46=-0.13885533809661865, 47=0.2768166661262512, 48=0.9898761510848999, 49=0.29929932951927185, 50=-0.2958138585090637, 51=-0.22763559222221375, 52=-0.5000461339950562, 53=-0.456381231546402, 54=0.5633571743965149, 55=-0.0806773453950882, 56=-0.22764116525650024, 57=-0.352817177772522, 58=-0.22321543097496033, 59=-0.10916472971439362, 60=-0.29375842213630676, 61=0.049102891236543655, 62=-0.35317322611808777, 63=-0.2738627791404724, 64=0.40873318910598755, 65=-0.05459516867995262, 66=0.46635472774505615, 67=-0.32608309388160706, 68=-0.4124443531036377, 69=-0.36642986536026, 70=0.11626649647951126, 71=0.09299281239509583, 72=0.06576022505760193, 73=0.37549731135368347, 74=-0.18591205775737762, 75=0.2776002585887909, 76=-0.07947368919849396, 77=0.005565340630710125, 78=-0.12807142734527588, 79=0.6194952130317688, 80=-0.11788779497146606, 81=0.3837911784648895, 82=0.07781966775655746, 83=0.38846638798713684, 84=-0.3825984597206116, 85=0.15459533035755157, 86=-0.11893496662378311, 87=-0.0008560251444578171, 88=-0.5020004510879517, 89=0.3710005581378937, 90=-0.21979665756225586, 91=-0.13489516079425812, 92=-0.36333250999450684, 93=0.12563833594322205, 94=-0.39627963304519653, 95=0.1846710443496704, 96=0.23816588521003723, 97=0.504469633102417, 98=0.39792609214782715, 99=0.10264253616333008, 100=-0.16383711993694305, 101=0.3081285357475281, 102=-0.2959822714328766, 103=0.38059693574905396, 104=-0.0192202627658844, 105=0.08322116732597351, 106=0.14358305931091309, 107=-0.37024572491645813, 108=0.00025602802634239197, 109=-0.11155793070793152, 110=-0.15934599936008453, 111=-0.2876470386981964, 112=-0.5796794891357422, 113=-0.045676134526729584, 114=0.13739138841629028, 115=-0.3791455924510956, 116=-0.2531186044216156, 117=-0.11568623036146164, 118=-0.18219803273677826, 119=0.3195190131664276, 120=-0.20271077752113342, 121=0.18304714560508728, 122=-0.23445013165473938, 123=-0.09943544864654541, 124=0.030131343752145767, 125=0.38270896673202515, 126=-0.0835808590054512, 127=0.19541577994823456, 128=0.05114699900150299, 129=0.059720203280448914, 130=0.08824637532234192, 131=0.15253619849681854, 132=0.16178305447101593, 133=0.41068974137306213, 134=0.33532848954200745, 135=0.4894946217536926, 136=-0.15127144753932953, 137=0.04980859160423279, 138=-0.032458383589982986, 139=0.3900485336780548, 140=0.28783345222473145, 141=0.003131020814180374, 142=-0.28225046396255493, 143=0.17181256413459778, 144=-0.4567558467388153, 145=-0.0842592641711235, 146=0.1949719488620758, 147=-0.19299033284187317, 148=-0.204234778881073, 149=-0.07661204040050507, 150=-0.13178813457489014, 151=-0.18955537676811218, 152=0.12464132905006409, 153=-0.5529391169548035, 154=0.2887957990169525, 155=0.003092937171459198, 156=-0.0037227813154459, 157=0.5580743551254272, 158=-0.1969381868839264, 159=0.11123789846897125, 160=0.25598040223121643, 161=0.47395527362823486, 162=-0.1801462471485138, 163=-0.4388729929924011, 164=-0.38223153352737427, 165=0.012956555932760239, 166=-0.24710333347320557, 167=0.43666723370552063, 168=0.24677318334579468, 169=0.23773133754730225, 170=0.444384902715683, 171=-0.22997356951236725, 172=0.27833858132362366, 173=-0.39479929208755493, 174=0.21081610023975372, 175=-0.4874005615711212, 176=-0.3916911482810974, 177=-0.4020669162273407, 178=0.25062501430511475, 179=0.48264938592910767, 180=0.09846019744873047, 181=-0.1328398436307907, 182=0.12488850951194763, 183=-0.24129866063594818, 184=0.019469425082206726, 185=-0.21942366659641266, 186=-0.46079862117767334, 187=0.0012602112255990505, 188=-0.4591357707977295, 189=0.45616742968559265, 190=0.4057249426841736, 191=-0.7133828401565552, 192=0.1098237931728363, 193=0.055745214223861694, 194=0.19739742577075958, 195=-5.053869247436523, 196=-0.30236685276031494, 197=0.2796938717365265, 198=-0.25181832909584045, 199=0.300956666469574, 200=-0.602235734462738, 201=0.22278019785881042, 202=0.08122294396162033, 203=0.10978507995605469, 204=-0.6756348609924316, 205=-0.23615959286689758, 206=0.009627088904380798, 207=0.6062808036804199, 208=0.10739795863628387, 209=0.1361292004585266, 210=-0.0867241844534874, 211=0.2764720320701599, 212=-0.25111475586891174, 213=-0.35227569937705994, 214=-0.0027881551068276167, 215=0.42142659425735474, 216=0.38106662034988403, 217=0.26074835658073425, 218=0.11948724836111069, 219=0.24488465487957, 220=0.16162556409835815, 221=-0.05234997719526291, 222=-0.045644309371709824, 223=0.12627844512462616, 224=-0.0780147835612297, 225=-0.2982993721961975, 226=0.3270186185836792, 227=-1.2103849649429321, 228=-0.30864593386650085, 229=-0.08962051570415497, 230=0.3299592137336731, 231=-0.15174178779125214, 232=0.27171388268470764, 233=-0.1554953157901764, 234=0.13880865275859833, 235=-0.2654488980770111, 236=0.019639015197753906, 237=0.23643316328525543, 238=0.25318774580955505, 239=0.11243179440498352, 240=0.09229936450719833, 241=-0.02133374661207199, 242=0.4456542730331421, 243=0.5731046795845032, 244=-0.10235664993524551, 245=0.0009454526007175446, 246=0.2619055211544037, 247=-0.1284642070531845, 248=0.31462305784225464, 249=0.4402744174003601, 250=0.1305093616247177, 251=0.11256440728902817, 252=0.507465124130249, 253=-0.5029115080833435, 254=0.0027040820568799973, 255=0.006506666541099548, 256=-0.28479498624801636, 257=-0.1944963037967682, 258=-0.10936438292264938, 259=0.327178418636322, 260=-0.20705845952033997, 261=-0.2635635435581207, 262=-0.19220000505447388, 263=-0.045894015580415726, 264=0.27590417861938477, 265=-0.24078702926635742, 266=0.3788028359413147, 267=0.08569777011871338, 268=-0.0017673298716545105, 269=-0.3724876046180725, 270=-0.5869436860084534, 271=-0.4695771336555481, 272=0.12816813588142395, 273=-0.3862929046154022, 274=-0.10335773974657059, 275=0.15757149457931519, 276=0.39792895317077637, 277=0.32700517773628235, 278=-0.04615457355976105, 279=0.25988349318504333, 280=-0.091347336769104, 281=0.2871307134628296, 282=0.32614269852638245, 283=-0.4779491126537323, 284=-0.12296654284000397, 285=-0.4846457242965698, 286=0.1063489317893982, 287=0.15479019284248352, 288=0.1782737523317337, 289=-0.5864168405532837, 290=0.33551904559135437, 291=0.4467630386352539, 292=0.2523404061794281, 293=-0.30433759093284607, 294=-0.31645768880844116, 295=-0.051132768392562866, 296=0.005848266184329987, 297=-0.14090265333652496, 298=0.24667660892009735, 299=0.08129002153873444, 300=0.2592727541923523, 301=0.35248568654060364, 302=-0.34920844435691833, 303=-0.0091690793633461, 304=-0.03573458641767502, 305=0.1647346019744873, 306=0.42030903697013855, 307=0.28396928310394287, 308=0.7061797380447388, 309=-0.30408960580825806, 310=0.15801116824150085, 311=0.13584789633750916, 312=0.12860912084579468, 313=0.5205117464065552, 314=0.006947316229343414, 315=0.11243520677089691, 316=0.38559335470199585, 317=-0.0665978416800499, 318=0.337490439414978, 319=0.1829356849193573, 320=0.5160303115844727, 321=-0.1299796849489212, 322=0.4660511612892151, 323=0.41570669412612915, 324=0.058356672525405884, 325=0.018694676458835602, 326=-0.7225120663642883, 327=0.5632511377334595, 328=0.23511403799057007, 329=0.27776089310646057, 330=-0.006380133330821991, 331=-0.3943710923194885, 332=0.5627593994140625, 333=-0.051022909581661224, 334=0.5315565466880798, 335=0.2589815855026245, 336=0.10288769006729126, 337=0.11095601320266724, 338=0.30779096484184265, 339=0.1882854700088501, 340=0.25513216853141785, 341=0.5006116628646851, 342=0.31353211402893066, 343=0.2064826637506485, 344=0.20337191224098206, 345=0.37518882751464844, 346=-0.5109310746192932, 347=0.15530312061309814, 348=-0.014532919973134995, 349=-0.4100968539714813, 350=-0.41702941060066223, 351=-0.03720090165734291, 352=0.4377273619174957, 353=-0.11275497078895569, 354=-0.028397336602211, 355=0.12688791751861572, 356=0.3089796304702759, 357=0.1022544652223587, 358=-0.007881700992584229, 359=-0.12737412750720978, 360=0.2407739758491516, 361=0.22673989832401276, 362=-0.26267820596694946, 363=-0.1985563188791275, 364=0.2883371114730835, 365=0.3122367262840271, 366=0.07638287544250488, 367=0.07954242080450058, 368=-0.04913613200187683, 369=0.8656174540519714, 370=0.14963215589523315, 371=0.09622099995613098, 372=0.2065020203590393, 373=0.42023926973342896, 374=-0.10268233716487885, 375=0.4896998703479767, 376=-0.4853786826133728, 377=-0.14072881639003754, 378=-0.5173507928848267, 379=0.38132721185684204, 380=0.25262725353240967, 381=-0.32924580574035645, 382=-1.0359116792678833, 383=-0.17485319077968597, 384=-0.3701465129852295, 385=-0.8697187304496765, 386=0.5144445896148682, 387=0.4708768129348755, 388=-0.6273230314254761, 389=-0.5959116220474243, 390=0.775299072265625, 391=0.24602915346622467, 392=0.22195833921432495, 393=0.0023524751886725426, 394=-0.03728947043418884, 395=-0.03094223141670227, 396=-0.15462444722652435, 397=-0.034669749438762665, 398=0.4292915463447571, 399=0.32169803977012634, 400=0.16544222831726074, 401=-0.3007698059082031, 402=0.36534780263900757, 403=-0.5253899097442627, 404=0.16559278964996338, 405=0.3338485658168793, 406=0.11777622997760773, 407=0.38854655623435974, 408=0.45733317732810974, 409=-0.24424320459365845, 410=0.0915689617395401, 411=0.36092060804367065, 412=-0.19449380040168762, 413=-0.34770578145980835, 414=-0.39052504301071167, 415=-0.17414671182632446, 416=0.12289627641439438, 417=0.11486253142356873, 418=0.019910935312509537, 419=0.10526599735021591, 420=0.23396717011928558, 421=-0.09883125126361847, 422=-0.009286921471357346, 423=0.8904801607131958, 424=0.11475704610347748, 425=0.345291405916214, 426=-0.47301366925239563, 427=-0.24833618104457855, 428=0.3506036698818207, 429=0.30501192808151245, 430=-0.062260523438453674, 431=-0.24678128957748413, 432=-0.24240989983081818, 433=-0.1763041615486145, 434=-0.6848266124725342, 435=0.3886722922325134, 436=-0.300096333026886, 437=-0.38859665393829346, 438=-0.3443761467933655, 439=0.34774649143218994, 440=-0.444487065076828, 441=0.6925027370452881, 442=0.2934395670890808, 443=0.7522571086883545, 444=0.04532145708799362, 445=6.469404697418213, 446=-0.4607214331626892, 447=-0.11063402146100998, 448=0.4674191176891327, 449=0.03281670808792114, 450=-0.18718792498111725, 451=0.631888210773468, 452=-0.018420681357383728, 453=-0.5640164613723755, 454=-0.5204675197601318, 455=-0.2229187786579132, 456=0.36513474583625793, 457=0.09127238392829895, 458=-0.31675559282302856, 459=0.25613102316856384, 460=0.04067419469356537, 461=-0.49958667159080505, 462=0.004863567650318146, 463=0.37281471490859985, 464=-0.12441016733646393, 465=0.009451359510421753, 466=0.21275663375854492, 467=0.1525597870349884, 468=-0.2588292360305786, 469=-0.0727156475186348, 470=-0.1658860743045807, 471=0.03745807707309723, 472=0.2686837911605835, 473=0.10170812904834747, 474=0.008957238867878914, 475=0.22339260578155518, 476=-0.11310940235853195, 477=-0.013431794941425323, 478=0.03361571580171585, 479=-0.10842948406934738, 480=-0.11378055065870285, 481=0.36763235926628113, 482=-0.21335536241531372, 483=-0.5552292466163635, 484=0.20042705535888672, 485=0.29235607385635376, 486=-0.16682162880897522, 487=-0.6830902099609375, 488=-0.7897211909294128, 489=-0.3452065587043762, 490=0.3262530267238617, 491=0.14846180379390717, 492=0.1995244175195694, 493=-0.19814562797546387, 494=0.3111087679862976, 495=-0.09076923131942749, 496=-0.3196336030960083, 497=-0.2874402105808258, 498=-0.4715084433555603, 499=0.07187982648611069, 500=0.0868721678853035, 501=0.018129687756299973, 502=0.40888115763664246, 503=0.06677349656820297, 504=0.31802651286125183, 505=0.44461026787757874, 506=0.06925699859857559, 507=-0.39630845189094543, 508=0.5758805871009827, 509=-0.48720282316207886, 510=0.21655049920082092, 511=0.18041738867759705, 512=-0.37001532316207886, 513=0.30423882603645325, 514=0.4165676236152649, 515=0.5890618562698364, 516=-0.09354093670845032, 517=-0.490866482257843, 518=0.005985692143440247, 519=0.2342604696750641, 520=-0.29090598225593567, 521=0.25867366790771484, 522=0.020116716623306274, 523=-0.4834479093551636, 524=-0.38326889276504517, 525=0.030562326312065125, 526=-0.28143012523651123, 527=0.047483958303928375, 528=0.03131657838821411, 529=0.14539261162281036, 530=0.2657688856124878, 531=-0.30514922738075256, 532=0.20335280895233154, 533=0.9238275289535522, 534=0.26777851581573486, 535=0.07666008919477463, 536=-0.2004360556602478, 537=0.07873768359422684, 538=-0.22396880388259888, 539=0.016511525958776474, 540=-0.0645843967795372, 541=0.041655030101537704, 542=-0.31202131509780884, 543=-0.16237586736679077, 544=-0.018756203353405, 545=-0.6770877242088318, 546=-0.36256980895996094, 547=-0.0001107826828956604, 548=-0.21042238175868988, 549=0.14332248270511627, 550=0.4511985182762146, 551=-0.2126290649175644, 552=0.16274899244308472, 553=0.2696540355682373, 554=0.24619798362255096, 555=-0.27679193019866943, 556=0.282692015171051, 557=-0.04539155215024948, 558=-0.03358479216694832, 559=-0.2293872833251953, 560=-0.27398476004600525, 561=0.1605578362941742, 562=-0.04925669729709625, 563=-0.8379406929016113, 564=-0.2562994956970215, 565=0.10627657920122147, 566=-0.23925571143627167, 567=0.2988191545009613, 568=-0.33967524766921997, 569=-0.5148581266403198, 570=0.2767416536808014, 571=-0.2846421003341675, 572=0.11819150298833847, 573=0.01820939965546131, 574=0.1320335865020752, 575=0.40425556898117065, 576=-0.2290542721748352, 577=0.2540421187877655, 578=-0.07735625654459, 579=0.17901620268821716, 580=0.4665515720844269, 581=0.04234086722135544, 582=-0.007519880309700966, 583=0.019908905029296875, 584=-0.47425276041030884, 585=0.4152224063873291, 586=-0.3330836296081543, 587=-0.02608952485024929, 588=-0.4902445673942566, 589=-0.18680687248706818, 590=0.15022170543670654, 591=0.49135512113571167, 592=-0.0025474634021520615, 593=-0.057231973856687546, 594=-0.025098828598856926, 595=-0.20455726981163025, 596=0.05164807289838791, 597=0.25541508197784424, 598=0.25040099024772644, 599=-0.7788197994232178, 600=0.24559934437274933, 601=-0.4419705271720886, 602=-0.24329061806201935, 603=0.08764069527387619, 604=-0.3053707480430603, 605=-0.3855486512184143, 606=-0.12006847560405731, 607=-0.11367455124855042, 608=-0.21191062033176422, 609=0.3248484432697296, 610=-0.3784434199333191, 611=-0.14282730221748352, 612=-0.24174684286117554, 613=0.05087311193346977, 614=0.004864200949668884, 615=-0.1897895634174347, 616=0.40971019864082336, 617=0.14054784178733826, 618=-0.01762159913778305, 619=-0.14712519943714142, 620=0.15746773779392242, 621=0.5074120163917542, 622=0.2645655572414398, 623=-0.11579076945781708, 624=0.025874730199575424, 625=-0.1005643755197525, 626=0.10092177987098694, 627=-0.26683509349823, 628=0.2171652615070343, 629=-7.491466522216797, 630=0.15933790802955627, 631=-0.5703866481781006, 632=-0.1324462890625, 633=-0.24147555232048035, 634=-0.2548695504665375, 635=0.3617456555366516, 636=-0.3048914670944214, 637=-0.12735946476459503, 638=0.19228529930114746, 639=-0.17645305395126343, 640=-0.03264733776450157, 641=0.02849065139889717, 642=0.48743507266044617, 643=-0.006416557356715202, 644=0.19153225421905518, 645=0.3393341898918152, 646=-0.42886412143707275, 647=-0.5044686794281006, 648=0.017152875661849976, 649=0.450178861618042, 650=0.10408412665128708, 651=-0.37451794743537903, 652=0.3647468388080597, 653=-0.1729431450366974, 654=-0.022314993664622307, 655=0.07218004763126373, 656=-0.2202787846326828, 657=0.37946537137031555, 658=-0.34831735491752625, 659=0.0005491501651704311, 660=-0.46094444394111633, 661=-0.008464418351650238, 662=0.31013303995132446, 663=-0.40341243147850037, 664=-0.19397953152656555, 665=0.3725287616252899, 666=0.47799381613731384, 667=-0.11735855042934418, 668=0.3080187141895294, 669=-0.7572708129882812, 670=0.2692417502403259, 671=-0.2783730626106262, 672=0.1584850251674652, 673=0.6216636896133423, 674=0.059204451739788055, 675=-0.5437134504318237, 676=0.345933198928833, 677=-0.37469154596328735, 678=-0.12409835308790207, 679=-0.01588243618607521, 680=0.25664180517196655, 681=0.01004074141383171, 682=-0.33959072828292847, 683=-0.6478890180587769, 684=0.07243404537439346, 685=0.09472053498029709, 686=-0.0037708990275859833, 687=0.4530909061431885, 688=0.514091432094574, 689=-2.8486554622650146, 690=0.22625623643398285, 691=-0.2233259379863739, 692=0.43105655908584595, 693=-0.18853747844696045, 694=-0.15312513709068298, 695=-0.3631364107131958, 696=0.23635566234588623, 697=-0.45348837971687317, 698=0.2951396703720093, 699=0.2870984673500061, 700=0.10346721112728119, 701=-0.08410540223121643, 702=0.23930275440216064, 703=0.10277339816093445, 704=-0.18653425574302673, 705=-0.16459563374519348, 706=0.21484853327274323, 707=0.5792280435562134, 708=0.4287334680557251, 709=-0.08481064438819885, 710=-0.44990074634552, 711=-0.21747061610221863, 712=0.027091773226857185, 713=0.0522066205739975, 714=-0.2510857582092285, 715=-0.18713971972465515, 716=-0.06101231649518013, 717=0.32859501242637634, 718=0.22905907034873962, 719=0.04920406639575958, 720=-0.04036427661776543, 721=0.8666608333587646, 722=-0.08779007196426392, 723=-0.18075817823410034, 724=-0.2269165813922882, 725=0.07553339749574661, 726=0.39757049083709717, 727=-0.08612340688705444, 728=0.11958380043506622, 729=0.22096823155879974, 730=0.45666128396987915, 731=-0.6605594754219055, 732=0.357834130525589, 733=0.5312210917472839, 734=-0.020172499120235443, 735=0.34068599343299866, 736=-0.7552745938301086, 737=0.40271398425102234, 738=-0.28172922134399414, 739=-0.33636292815208435, 740=0.23783975839614868, 741=-0.4733569025993347, 742=0.007202818989753723, 743=-0.4340314567089081, 744=-0.41992348432540894, 745=-0.12622782588005066, 746=0.4354759156703949, 747=0.06646571308374405, 748=-0.05462207272648811, 749=-0.06405144929885864, 750=-0.0672161653637886, 751=0.26126396656036377, 752=0.19557398557662964, 753=-0.15154574811458588, 754=0.21077761054039001, 755=0.3276965618133545, 756=-0.08911502361297607, 757=-0.3373878002166748, 758=-0.2528281807899475, 759=0.2551825940608978, 760=-0.16991230845451355, 761=-0.2709953486919403, 762=-0.33592551946640015, 763=-0.20697307586669922, 764=0.1233106479048729, 765=0.28270789980888367, 766=-0.49243953824043274, 767=-0.028941281139850616), Row(0=-0.3828985393047333, 1=0.12982381880283356, 2=-0.15237723290920258, 3=-0.30286455154418945, 4=0.07984059303998947, 5=0.24567967653274536, 6=-0.4774293005466461, 7=0.11368699371814728, 8=-0.43600544333457947, 9=0.0015827585011720657, 10=-0.18087996542453766, 11=0.012072999961674213, 12=0.44286948442459106, 13=0.3444838225841522, 14=-0.10974393784999847, 15=0.2576451003551483, 16=0.34980666637420654, 17=0.008537352085113525, 18=0.3259563148021698, 19=-0.05231372267007828, 20=0.2902263402938843, 21=0.46089696884155273, 22=-0.1315918117761612, 23=-0.6931538581848145, 24=-0.12955236434936523, 25=0.12681958079338074, 26=-0.3282795250415802, 27=0.3309135138988495, 28=0.18025650084018707, 29=-0.17974373698234558, 30=-0.12690448760986328, 31=0.2799723744392395, 32=-0.30506038665771484, 33=0.2364874929189682, 34=-0.06421627104282379, 35=-0.4623970091342926, 36=-0.4120138883590698, 37=0.29458779096603394, 38=0.22279199957847595, 39=0.13119202852249146, 40=0.23632945120334625, 41=0.3276778757572174, 42=-0.09378167986869812, 43=-0.4410403370857239, 44=-0.27837029099464417, 45=0.687725305557251, 46=-0.13885533809661865, 47=0.2768166661262512, 48=0.9898761510848999, 49=0.29929932951927185, 50=-0.2958138585090637, 51=-0.22763559222221375, 52=-0.5000461339950562, 53=-0.456381231546402, 54=0.5633571743965149, 55=-0.0806773453950882, 56=-0.22764116525650024, 57=-0.352817177772522, 58=-0.22321543097496033, 59=-0.10916472971439362, 60=-0.29375842213630676, 61=0.049102891236543655, 62=-0.35317322611808777, 63=-0.2738627791404724, 64=0.40873318910598755, 65=-0.05459516867995262, 66=0.46635472774505615, 67=-0.32608309388160706, 68=-0.4124443531036377, 69=-0.36642986536026, 70=0.11626649647951126, 71=0.09299281239509583, 72=0.06576022505760193, 73=0.37549731135368347, 74=-0.18591205775737762, 75=0.2776002585887909, 76=-0.07947368919849396, 77=0.005565340630710125, 78=-0.12807142734527588, 79=0.6194952130317688, 80=-0.11788779497146606, 81=0.3837911784648895, 82=0.07781966775655746, 83=0.38846638798713684, 84=-0.3825984597206116, 85=0.15459533035755157, 86=-0.11893496662378311, 87=-0.0008560251444578171, 88=-0.5020004510879517, 89=0.3710005581378937, 90=-0.21979665756225586, 91=-0.13489516079425812, 92=-0.36333250999450684, 93=0.12563833594322205, 94=-0.39627963304519653, 95=0.1846710443496704, 96=0.23816588521003723, 97=0.504469633102417, 98=0.39792609214782715, 99=0.10264253616333008, 100=-0.16383711993694305, 101=0.3081285357475281, 102=-0.2959822714328766, 103=0.38059693574905396, 104=-0.0192202627658844, 105=0.08322116732597351, 106=0.14358305931091309, 107=-0.37024572491645813, 108=0.00025602802634239197, 109=-0.11155793070793152, 110=-0.15934599936008453, 111=-0.2876470386981964, 112=-0.5796794891357422, 113=-0.045676134526729584, 114=0.13739138841629028, 115=-0.3791455924510956, 116=-0.2531186044216156, 117=-0.11568623036146164, 118=-0.18219803273677826, 119=0.3195190131664276, 120=-0.20271077752113342, 121=0.18304714560508728, 122=-0.23445013165473938, 123=-0.09943544864654541, 124=0.030131343752145767, 125=0.38270896673202515, 126=-0.0835808590054512, 127=0.19541577994823456, 128=0.05114699900150299, 129=0.059720203280448914, 130=0.08824637532234192, 131=0.15253619849681854, 132=0.16178305447101593, 133=0.41068974137306213, 134=0.33532848954200745, 135=0.4894946217536926, 136=-0.15127144753932953, 137=0.04980859160423279, 138=-0.032458383589982986, 139=0.3900485336780548, 140=0.28783345222473145, 141=0.003131020814180374, 142=-0.28225046396255493, 143=0.17181256413459778, 144=-0.4567558467388153, 145=-0.0842592641711235, 146=0.1949719488620758, 147=-0.19299033284187317, 148=-0.204234778881073, 149=-0.07661204040050507, 150=-0.13178813457489014, 151=-0.18955537676811218, 152=0.12464132905006409, 153=-0.5529391169548035, 154=0.2887957990169525, 155=0.003092937171459198, 156=-0.0037227813154459, 157=0.5580743551254272, 158=-0.1969381868839264, 159=0.11123789846897125, 160=0.25598040223121643, 161=0.47395527362823486, 162=-0.1801462471485138, 163=-0.4388729929924011, 164=-0.38223153352737427, 165=0.012956555932760239, 166=-0.24710333347320557, 167=0.43666723370552063, 168=0.24677318334579468, 169=0.23773133754730225, 170=0.444384902715683, 171=-0.22997356951236725, 172=0.27833858132362366, 173=-0.39479929208755493, 174=0.21081610023975372, 175=-0.4874005615711212, 176=-0.3916911482810974, 177=-0.4020669162273407, 178=0.25062501430511475, 179=0.48264938592910767, 180=0.09846019744873047, 181=-0.1328398436307907, 182=0.12488850951194763, 183=-0.24129866063594818, 184=0.019469425082206726, 185=-0.21942366659641266, 186=-0.46079862117767334, 187=0.0012602112255990505, 188=-0.4591357707977295, 189=0.45616742968559265, 190=0.4057249426841736, 191=-0.7133828401565552, 192=0.1098237931728363, 193=0.055745214223861694, 194=0.19739742577075958, 195=-5.053869247436523, 196=-0.30236685276031494, 197=0.2796938717365265, 198=-0.25181832909584045, 199=0.300956666469574, 200=-0.602235734462738, 201=0.22278019785881042, 202=0.08122294396162033, 203=0.10978507995605469, 204=-0.6756348609924316, 205=-0.23615959286689758, 206=0.009627088904380798, 207=0.6062808036804199, 208=0.10739795863628387, 209=0.1361292004585266, 210=-0.0867241844534874, 211=0.2764720320701599, 212=-0.25111475586891174, 213=-0.35227569937705994, 214=-0.0027881551068276167, 215=0.42142659425735474, 216=0.38106662034988403, 217=0.26074835658073425, 218=0.11948724836111069, 219=0.24488465487957, 220=0.16162556409835815, 221=-0.05234997719526291, 222=-0.045644309371709824, 223=0.12627844512462616, 224=-0.0780147835612297, 225=-0.2982993721961975, 226=0.3270186185836792, 227=-1.2103849649429321, 228=-0.30864593386650085, 229=-0.08962051570415497, 230=0.3299592137336731, 231=-0.15174178779125214, 232=0.27171388268470764, 233=-0.1554953157901764, 234=0.13880865275859833, 235=-0.2654488980770111, 236=0.019639015197753906, 237=0.23643316328525543, 238=0.25318774580955505, 239=0.11243179440498352, 240=0.09229936450719833, 241=-0.02133374661207199, 242=0.4456542730331421, 243=0.5731046795845032, 244=-0.10235664993524551, 245=0.0009454526007175446, 246=0.2619055211544037, 247=-0.1284642070531845, 248=0.31462305784225464, 249=0.4402744174003601, 250=0.1305093616247177, 251=0.11256440728902817, 252=0.507465124130249, 253=-0.5029115080833435, 254=0.0027040820568799973, 255=0.006506666541099548, 256=-0.28479498624801636, 257=-0.1944963037967682, 258=-0.10936438292264938, 259=0.327178418636322, 260=-0.20705845952033997, 261=-0.2635635435581207, 262=-0.19220000505447388, 263=-0.045894015580415726, 264=0.27590417861938477, 265=-0.24078702926635742, 266=0.3788028359413147, 267=0.08569777011871338, 268=-0.0017673298716545105, 269=-0.3724876046180725, 270=-0.5869436860084534, 271=-0.4695771336555481, 272=0.12816813588142395, 273=-0.3862929046154022, 274=-0.10335773974657059, 275=0.15757149457931519, 276=0.39792895317077637, 277=0.32700517773628235, 278=-0.04615457355976105, 279=0.25988349318504333, 280=-0.091347336769104, 281=0.2871307134628296, 282=0.32614269852638245, 283=-0.4779491126537323, 284=-0.12296654284000397, 285=-0.4846457242965698, 286=0.1063489317893982, 287=0.15479019284248352, 288=0.1782737523317337, 289=-0.5864168405532837, 290=0.33551904559135437, 291=0.4467630386352539, 292=0.2523404061794281, 293=-0.30433759093284607, 294=-0.31645768880844116, 295=-0.051132768392562866, 296=0.005848266184329987, 297=-0.14090265333652496, 298=0.24667660892009735, 299=0.08129002153873444, 300=0.2592727541923523, 301=0.35248568654060364, 302=-0.34920844435691833, 303=-0.0091690793633461, 304=-0.03573458641767502, 305=0.1647346019744873, 306=0.42030903697013855, 307=0.28396928310394287, 308=0.7061797380447388, 309=-0.30408960580825806, 310=0.15801116824150085, 311=0.13584789633750916, 312=0.12860912084579468, 313=0.5205117464065552, 314=0.006947316229343414, 315=0.11243520677089691, 316=0.38559335470199585, 317=-0.0665978416800499, 318=0.337490439414978, 319=0.1829356849193573, 320=0.5160303115844727, 321=-0.1299796849489212, 322=0.4660511612892151, 323=0.41570669412612915, 324=0.058356672525405884, 325=0.018694676458835602, 326=-0.7225120663642883, 327=0.5632511377334595, 328=0.23511403799057007, 329=0.27776089310646057, 330=-0.006380133330821991, 331=-0.3943710923194885, 332=0.5627593994140625, 333=-0.051022909581661224, 334=0.5315565466880798, 335=0.2589815855026245, 336=0.10288769006729126, 337=0.11095601320266724, 338=0.30779096484184265, 339=0.1882854700088501, 340=0.25513216853141785, 341=0.5006116628646851, 342=0.31353211402893066, 343=0.2064826637506485, 344=0.20337191224098206, 345=0.37518882751464844, 346=-0.5109310746192932, 347=0.15530312061309814, 348=-0.014532919973134995, 349=-0.4100968539714813, 350=-0.41702941060066223, 351=-0.03720090165734291, 352=0.4377273619174957, 353=-0.11275497078895569, 354=-0.028397336602211, 355=0.12688791751861572, 356=0.3089796304702759, 357=0.1022544652223587, 358=-0.007881700992584229, 359=-0.12737412750720978, 360=0.2407739758491516, 361=0.22673989832401276, 362=-0.26267820596694946, 363=-0.1985563188791275, 364=0.2883371114730835, 365=0.3122367262840271, 366=0.07638287544250488, 367=0.07954242080450058, 368=-0.04913613200187683, 369=0.8656174540519714, 370=0.14963215589523315, 371=0.09622099995613098, 372=0.2065020203590393, 373=0.42023926973342896, 374=-0.10268233716487885, 375=0.4896998703479767, 376=-0.4853786826133728, 377=-0.14072881639003754, 378=-0.5173507928848267, 379=0.38132721185684204, 380=0.25262725353240967, 381=-0.32924580574035645, 382=-1.0359116792678833, 383=-0.17485319077968597, 384=-0.3701465129852295, 385=-0.8697187304496765, 386=0.5144445896148682, 387=0.4708768129348755, 388=-0.6273230314254761, 389=-0.5959116220474243, 390=0.775299072265625, 391=0.24602915346622467, 392=0.22195833921432495, 393=0.0023524751886725426, 394=-0.03728947043418884, 395=-0.03094223141670227, 396=-0.15462444722652435, 397=-0.034669749438762665, 398=0.4292915463447571, 399=0.32169803977012634, 400=0.16544222831726074, 401=-0.3007698059082031, 402=0.36534780263900757, 403=-0.5253899097442627, 404=0.16559278964996338, 405=0.3338485658168793, 406=0.11777622997760773, 407=0.38854655623435974, 408=0.45733317732810974, 409=-0.24424320459365845, 410=0.0915689617395401, 411=0.36092060804367065, 412=-0.19449380040168762, 413=-0.34770578145980835, 414=-0.39052504301071167, 415=-0.17414671182632446, 416=0.12289627641439438, 417=0.11486253142356873, 418=0.019910935312509537, 419=0.10526599735021591, 420=0.23396717011928558, 421=-0.09883125126361847, 422=-0.009286921471357346, 423=0.8904801607131958, 424=0.11475704610347748, 425=0.345291405916214, 426=-0.47301366925239563, 427=-0.24833618104457855, 428=0.3506036698818207, 429=0.30501192808151245, 430=-0.062260523438453674, 431=-0.24678128957748413, 432=-0.24240989983081818, 433=-0.1763041615486145, 434=-0.6848266124725342, 435=0.3886722922325134, 436=-0.300096333026886, 437=-0.38859665393829346, 438=-0.3443761467933655, 439=0.34774649143218994, 440=-0.444487065076828, 441=0.6925027370452881, 442=0.2934395670890808, 443=0.7522571086883545, 444=0.04532145708799362, 445=6.469404697418213, 446=-0.4607214331626892, 447=-0.11063402146100998, 448=0.4674191176891327, 449=0.03281670808792114, 450=-0.18718792498111725, 451=0.631888210773468, 452=-0.018420681357383728, 453=-0.5640164613723755, 454=-0.5204675197601318, 455=-0.2229187786579132, 456=0.36513474583625793, 457=0.09127238392829895, 458=-0.31675559282302856, 459=0.25613102316856384, 460=0.04067419469356537, 461=-0.49958667159080505, 462=0.004863567650318146, 463=0.37281471490859985, 464=-0.12441016733646393, 465=0.009451359510421753, 466=0.21275663375854492, 467=0.1525597870349884, 468=-0.2588292360305786, 469=-0.0727156475186348, 470=-0.1658860743045807, 471=0.03745807707309723, 472=0.2686837911605835, 473=0.10170812904834747, 474=0.008957238867878914, 475=0.22339260578155518, 476=-0.11310940235853195, 477=-0.013431794941425323, 478=0.03361571580171585, 479=-0.10842948406934738, 480=-0.11378055065870285, 481=0.36763235926628113, 482=-0.21335536241531372, 483=-0.5552292466163635, 484=0.20042705535888672, 485=0.29235607385635376, 486=-0.16682162880897522, 487=-0.6830902099609375, 488=-0.7897211909294128, 489=-0.3452065587043762, 490=0.3262530267238617, 491=0.14846180379390717, 492=0.1995244175195694, 493=-0.19814562797546387, 494=0.3111087679862976, 495=-0.09076923131942749, 496=-0.3196336030960083, 497=-0.2874402105808258, 498=-0.4715084433555603, 499=0.07187982648611069, 500=0.0868721678853035, 501=0.018129687756299973, 502=0.40888115763664246, 503=0.06677349656820297, 504=0.31802651286125183, 505=0.44461026787757874, 506=0.06925699859857559, 507=-0.39630845189094543, 508=0.5758805871009827, 509=-0.48720282316207886, 510=0.21655049920082092, 511=0.18041738867759705, 512=-0.37001532316207886, 513=0.30423882603645325, 514=0.4165676236152649, 515=0.5890618562698364, 516=-0.09354093670845032, 517=-0.490866482257843, 518=0.005985692143440247, 519=0.2342604696750641, 520=-0.29090598225593567, 521=0.25867366790771484, 522=0.020116716623306274, 523=-0.4834479093551636, 524=-0.38326889276504517, 525=0.030562326312065125, 526=-0.28143012523651123, 527=0.047483958303928375, 528=0.03131657838821411, 529=0.14539261162281036, 530=0.2657688856124878, 531=-0.30514922738075256, 532=0.20335280895233154, 533=0.9238275289535522, 534=0.26777851581573486, 535=0.07666008919477463, 536=-0.2004360556602478, 537=0.07873768359422684, 538=-0.22396880388259888, 539=0.016511525958776474, 540=-0.0645843967795372, 541=0.041655030101537704, 542=-0.31202131509780884, 543=-0.16237586736679077, 544=-0.018756203353405, 545=-0.6770877242088318, 546=-0.36256980895996094, 547=-0.0001107826828956604, 548=-0.21042238175868988, 549=0.14332248270511627, 550=0.4511985182762146, 551=-0.2126290649175644, 552=0.16274899244308472, 553=0.2696540355682373, 554=0.24619798362255096, 555=-0.27679193019866943, 556=0.282692015171051, 557=-0.04539155215024948, 558=-0.03358479216694832, 559=-0.2293872833251953, 560=-0.27398476004600525, 561=0.1605578362941742, 562=-0.04925669729709625, 563=-0.8379406929016113, 564=-0.2562994956970215, 565=0.10627657920122147, 566=-0.23925571143627167, 567=0.2988191545009613, 568=-0.33967524766921997, 569=-0.5148581266403198, 570=0.2767416536808014, 571=-0.2846421003341675, 572=0.11819150298833847, 573=0.01820939965546131, 574=0.1320335865020752, 575=0.40425556898117065, 576=-0.2290542721748352, 577=0.2540421187877655, 578=-0.07735625654459, 579=0.17901620268821716, 580=0.4665515720844269, 581=0.04234086722135544, 582=-0.007519880309700966, 583=0.019908905029296875, 584=-0.47425276041030884, 585=0.4152224063873291, 586=-0.3330836296081543, 587=-0.02608952485024929, 588=-0.4902445673942566, 589=-0.18680687248706818, 590=0.15022170543670654, 591=0.49135512113571167, 592=-0.0025474634021520615, 593=-0.057231973856687546, 594=-0.025098828598856926, 595=-0.20455726981163025, 596=0.05164807289838791, 597=0.25541508197784424, 598=0.25040099024772644, 599=-0.7788197994232178, 600=0.24559934437274933, 601=-0.4419705271720886, 602=-0.24329061806201935, 603=0.08764069527387619, 604=-0.3053707480430603, 605=-0.3855486512184143, 606=-0.12006847560405731, 607=-0.11367455124855042, 608=-0.21191062033176422, 609=0.3248484432697296, 610=-0.3784434199333191, 611=-0.14282730221748352, 612=-0.24174684286117554, 613=0.05087311193346977, 614=0.004864200949668884, 615=-0.1897895634174347, 616=0.40971019864082336, 617=0.14054784178733826, 618=-0.01762159913778305, 619=-0.14712519943714142, 620=0.15746773779392242, 621=0.5074120163917542, 622=0.2645655572414398, 623=-0.11579076945781708, 624=0.025874730199575424, 625=-0.1005643755197525, 626=0.10092177987098694, 627=-0.26683509349823, 628=0.2171652615070343, 629=-7.491466522216797, 630=0.15933790802955627, 631=-0.5703866481781006, 632=-0.1324462890625, 633=-0.24147555232048035, 634=-0.2548695504665375, 635=0.3617456555366516, 636=-0.3048914670944214, 637=-0.12735946476459503, 638=0.19228529930114746, 639=-0.17645305395126343, 640=-0.03264733776450157, 641=0.02849065139889717, 642=0.48743507266044617, 643=-0.006416557356715202, 644=0.19153225421905518, 645=0.3393341898918152, 646=-0.42886412143707275, 647=-0.5044686794281006, 648=0.017152875661849976, 649=0.450178861618042, 650=0.10408412665128708, 651=-0.37451794743537903, 652=0.3647468388080597, 653=-0.1729431450366974, 654=-0.022314993664622307, 655=0.07218004763126373, 656=-0.2202787846326828, 657=0.37946537137031555, 658=-0.34831735491752625, 659=0.0005491501651704311, 660=-0.46094444394111633, 661=-0.008464418351650238, 662=0.31013303995132446, 663=-0.40341243147850037, 664=-0.19397953152656555, 665=0.3725287616252899, 666=0.47799381613731384, 667=-0.11735855042934418, 668=0.3080187141895294, 669=-0.7572708129882812, 670=0.2692417502403259, 671=-0.2783730626106262, 672=0.1584850251674652, 673=0.6216636896133423, 674=0.059204451739788055, 675=-0.5437134504318237, 676=0.345933198928833, 677=-0.37469154596328735, 678=-0.12409835308790207, 679=-0.01588243618607521, 680=0.25664180517196655, 681=0.01004074141383171, 682=-0.33959072828292847, 683=-0.6478890180587769, 684=0.07243404537439346, 685=0.09472053498029709, 686=-0.0037708990275859833, 687=0.4530909061431885, 688=0.514091432094574, 689=-2.8486554622650146, 690=0.22625623643398285, 691=-0.2233259379863739, 692=0.43105655908584595, 693=-0.18853747844696045, 694=-0.15312513709068298, 695=-0.3631364107131958, 696=0.23635566234588623, 697=-0.45348837971687317, 698=0.2951396703720093, 699=0.2870984673500061, 700=0.10346721112728119, 701=-0.08410540223121643, 702=0.23930275440216064, 703=0.10277339816093445, 704=-0.18653425574302673, 705=-0.16459563374519348, 706=0.21484853327274323, 707=0.5792280435562134, 708=0.4287334680557251, 709=-0.08481064438819885, 710=-0.44990074634552, 711=-0.21747061610221863, 712=0.027091773226857185, 713=0.0522066205739975, 714=-0.2510857582092285, 715=-0.18713971972465515, 716=-0.06101231649518013, 717=0.32859501242637634, 718=0.22905907034873962, 719=0.04920406639575958, 720=-0.04036427661776543, 721=0.8666608333587646, 722=-0.08779007196426392, 723=-0.18075817823410034, 724=-0.2269165813922882, 725=0.07553339749574661, 726=0.39757049083709717, 727=-0.08612340688705444, 728=0.11958380043506622, 729=0.22096823155879974, 730=0.45666128396987915, 731=-0.6605594754219055, 732=0.357834130525589, 733=0.5312210917472839, 734=-0.020172499120235443, 735=0.34068599343299866, 736=-0.7552745938301086, 737=0.40271398425102234, 738=-0.28172922134399414, 739=-0.33636292815208435, 740=0.23783975839614868, 741=-0.4733569025993347, 742=0.007202818989753723, 743=-0.4340314567089081, 744=-0.41992348432540894, 745=-0.12622782588005066, 746=0.4354759156703949, 747=0.06646571308374405, 748=-0.05462207272648811, 749=-0.06405144929885864, 750=-0.0672161653637886, 751=0.26126396656036377, 752=0.19557398557662964, 753=-0.15154574811458588, 754=0.21077761054039001, 755=0.3276965618133545, 756=-0.08911502361297607, 757=-0.3373878002166748, 758=-0.2528281807899475, 759=0.2551825940608978, 760=-0.16991230845451355, 761=-0.2709953486919403, 762=-0.33592551946640015, 763=-0.20697307586669922, 764=0.1233106479048729, 765=0.28270789980888367, 766=-0.49243953824043274, 767=-0.028941281139850616), Row(0=-0.3828985393047333, 1=0.12982381880283356, 2=-0.15237723290920258, 3=-0.30286455154418945, 4=0.07984059303998947, 5=0.24567967653274536, 6=-0.4774293005466461, 7=0.11368699371814728, 8=-0.43600544333457947, 9=0.0015827585011720657, 10=-0.18087996542453766, 11=0.012072999961674213, 12=0.44286948442459106, 13=0.3444838225841522, 14=-0.10974393784999847, 15=0.2576451003551483, 16=0.34980666637420654, 17=0.008537352085113525, 18=0.3259563148021698, 19=-0.05231372267007828, 20=0.2902263402938843, 21=0.46089696884155273, 22=-0.1315918117761612, 23=-0.6931538581848145, 24=-0.12955236434936523, 25=0.12681958079338074, 26=-0.3282795250415802, 27=0.3309135138988495, 28=0.18025650084018707, 29=-0.17974373698234558, 30=-0.12690448760986328, 31=0.2799723744392395, 32=-0.30506038665771484, 33=0.2364874929189682, 34=-0.06421627104282379, 35=-0.4623970091342926, 36=-0.4120138883590698, 37=0.29458779096603394, 38=0.22279199957847595, 39=0.13119202852249146, 40=0.23632945120334625, 41=0.3276778757572174, 42=-0.09378167986869812, 43=-0.4410403370857239, 44=-0.27837029099464417, 45=0.687725305557251, 46=-0.13885533809661865, 47=0.2768166661262512, 48=0.9898761510848999, 49=0.29929932951927185, 50=-0.2958138585090637, 51=-0.22763559222221375, 52=-0.5000461339950562, 53=-0.456381231546402, 54=0.5633571743965149, 55=-0.0806773453950882, 56=-0.22764116525650024, 57=-0.352817177772522, 58=-0.22321543097496033, 59=-0.10916472971439362, 60=-0.29375842213630676, 61=0.049102891236543655, 62=-0.35317322611808777, 63=-0.2738627791404724, 64=0.40873318910598755, 65=-0.05459516867995262, 66=0.46635472774505615, 67=-0.32608309388160706, 68=-0.4124443531036377, 69=-0.36642986536026, 70=0.11626649647951126, 71=0.09299281239509583, 72=0.06576022505760193, 73=0.37549731135368347, 74=-0.18591205775737762, 75=0.2776002585887909, 76=-0.07947368919849396, 77=0.005565340630710125, 78=-0.12807142734527588, 79=0.6194952130317688, 80=-0.11788779497146606, 81=0.3837911784648895, 82=0.07781966775655746, 83=0.38846638798713684, 84=-0.3825984597206116, 85=0.15459533035755157, 86=-0.11893496662378311, 87=-0.0008560251444578171, 88=-0.5020004510879517, 89=0.3710005581378937, 90=-0.21979665756225586, 91=-0.13489516079425812, 92=-0.36333250999450684, 93=0.12563833594322205, 94=-0.39627963304519653, 95=0.1846710443496704, 96=0.23816588521003723, 97=0.504469633102417, 98=0.39792609214782715, 99=0.10264253616333008, 100=-0.16383711993694305, 101=0.3081285357475281, 102=-0.2959822714328766, 103=0.38059693574905396, 104=-0.0192202627658844, 105=0.08322116732597351, 106=0.14358305931091309, 107=-0.37024572491645813, 108=0.00025602802634239197, 109=-0.11155793070793152, 110=-0.15934599936008453, 111=-0.2876470386981964, 112=-0.5796794891357422, 113=-0.045676134526729584, 114=0.13739138841629028, 115=-0.3791455924510956, 116=-0.2531186044216156, 117=-0.11568623036146164, 118=-0.18219803273677826, 119=0.3195190131664276, 120=-0.20271077752113342, 121=0.18304714560508728, 122=-0.23445013165473938, 123=-0.09943544864654541, 124=0.030131343752145767, 125=0.38270896673202515, 126=-0.0835808590054512, 127=0.19541577994823456, 128=0.05114699900150299, 129=0.059720203280448914, 130=0.08824637532234192, 131=0.15253619849681854, 132=0.16178305447101593, 133=0.41068974137306213, 134=0.33532848954200745, 135=0.4894946217536926, 136=-0.15127144753932953, 137=0.04980859160423279, 138=-0.032458383589982986, 139=0.3900485336780548, 140=0.28783345222473145, 141=0.003131020814180374, 142=-0.28225046396255493, 143=0.17181256413459778, 144=-0.4567558467388153, 145=-0.0842592641711235, 146=0.1949719488620758, 147=-0.19299033284187317, 148=-0.204234778881073, 149=-0.07661204040050507, 150=-0.13178813457489014, 151=-0.18955537676811218, 152=0.12464132905006409, 153=-0.5529391169548035, 154=0.2887957990169525, 155=0.003092937171459198, 156=-0.0037227813154459, 157=0.5580743551254272, 158=-0.1969381868839264, 159=0.11123789846897125, 160=0.25598040223121643, 161=0.47395527362823486, 162=-0.1801462471485138, 163=-0.4388729929924011, 164=-0.38223153352737427, 165=0.012956555932760239, 166=-0.24710333347320557, 167=0.43666723370552063, 168=0.24677318334579468, 169=0.23773133754730225, 170=0.444384902715683, 171=-0.22997356951236725, 172=0.27833858132362366, 173=-0.39479929208755493, 174=0.21081610023975372, 175=-0.4874005615711212, 176=-0.3916911482810974, 177=-0.4020669162273407, 178=0.25062501430511475, 179=0.48264938592910767, 180=0.09846019744873047, 181=-0.1328398436307907, 182=0.12488850951194763, 183=-0.24129866063594818, 184=0.019469425082206726, 185=-0.21942366659641266, 186=-0.46079862117767334, 187=0.0012602112255990505, 188=-0.4591357707977295, 189=0.45616742968559265, 190=0.4057249426841736, 191=-0.7133828401565552, 192=0.1098237931728363, 193=0.055745214223861694, 194=0.19739742577075958, 195=-5.053869247436523, 196=-0.30236685276031494, 197=0.2796938717365265, 198=-0.25181832909584045, 199=0.300956666469574, 200=-0.602235734462738, 201=0.22278019785881042, 202=0.08122294396162033, 203=0.10978507995605469, 204=-0.6756348609924316, 205=-0.23615959286689758, 206=0.009627088904380798, 207=0.6062808036804199, 208=0.10739795863628387, 209=0.1361292004585266, 210=-0.0867241844534874, 211=0.2764720320701599, 212=-0.25111475586891174, 213=-0.35227569937705994, 214=-0.0027881551068276167, 215=0.42142659425735474, 216=0.38106662034988403, 217=0.26074835658073425, 218=0.11948724836111069, 219=0.24488465487957, 220=0.16162556409835815, 221=-0.05234997719526291, 222=-0.045644309371709824, 223=0.12627844512462616, 224=-0.0780147835612297, 225=-0.2982993721961975, 226=0.3270186185836792, 227=-1.2103849649429321, 228=-0.30864593386650085, 229=-0.08962051570415497, 230=0.3299592137336731, 231=-0.15174178779125214, 232=0.27171388268470764, 233=-0.1554953157901764, 234=0.13880865275859833, 235=-0.2654488980770111, 236=0.019639015197753906, 237=0.23643316328525543, 238=0.25318774580955505, 239=0.11243179440498352, 240=0.09229936450719833, 241=-0.02133374661207199, 242=0.4456542730331421, 243=0.5731046795845032, 244=-0.10235664993524551, 245=0.0009454526007175446, 246=0.2619055211544037, 247=-0.1284642070531845, 248=0.31462305784225464, 249=0.4402744174003601, 250=0.1305093616247177, 251=0.11256440728902817, 252=0.507465124130249, 253=-0.5029115080833435, 254=0.0027040820568799973, 255=0.006506666541099548, 256=-0.28479498624801636, 257=-0.1944963037967682, 258=-0.10936438292264938, 259=0.327178418636322, 260=-0.20705845952033997, 261=-0.2635635435581207, 262=-0.19220000505447388, 263=-0.045894015580415726, 264=0.27590417861938477, 265=-0.24078702926635742, 266=0.3788028359413147, 267=0.08569777011871338, 268=-0.0017673298716545105, 269=-0.3724876046180725, 270=-0.5869436860084534, 271=-0.4695771336555481, 272=0.12816813588142395, 273=-0.3862929046154022, 274=-0.10335773974657059, 275=0.15757149457931519, 276=0.39792895317077637, 277=0.32700517773628235, 278=-0.04615457355976105, 279=0.25988349318504333, 280=-0.091347336769104, 281=0.2871307134628296, 282=0.32614269852638245, 283=-0.4779491126537323, 284=-0.12296654284000397, 285=-0.4846457242965698, 286=0.1063489317893982, 287=0.15479019284248352, 288=0.1782737523317337, 289=-0.5864168405532837, 290=0.33551904559135437, 291=0.4467630386352539, 292=0.2523404061794281, 293=-0.30433759093284607, 294=-0.31645768880844116, 295=-0.051132768392562866, 296=0.005848266184329987, 297=-0.14090265333652496, 298=0.24667660892009735, 299=0.08129002153873444, 300=0.2592727541923523, 301=0.35248568654060364, 302=-0.34920844435691833, 303=-0.0091690793633461, 304=-0.03573458641767502, 305=0.1647346019744873, 306=0.42030903697013855, 307=0.28396928310394287, 308=0.7061797380447388, 309=-0.30408960580825806, 310=0.15801116824150085, 311=0.13584789633750916, 312=0.12860912084579468, 313=0.5205117464065552, 314=0.006947316229343414, 315=0.11243520677089691, 316=0.38559335470199585, 317=-0.0665978416800499, 318=0.337490439414978, 319=0.1829356849193573, 320=0.5160303115844727, 321=-0.1299796849489212, 322=0.4660511612892151, 323=0.41570669412612915, 324=0.058356672525405884, 325=0.018694676458835602, 326=-0.7225120663642883, 327=0.5632511377334595, 328=0.23511403799057007, 329=0.27776089310646057, 330=-0.006380133330821991, 331=-0.3943710923194885, 332=0.5627593994140625, 333=-0.051022909581661224, 334=0.5315565466880798, 335=0.2589815855026245, 336=0.10288769006729126, 337=0.11095601320266724, 338=0.30779096484184265, 339=0.1882854700088501, 340=0.25513216853141785, 341=0.5006116628646851, 342=0.31353211402893066, 343=0.2064826637506485, 344=0.20337191224098206, 345=0.37518882751464844, 346=-0.5109310746192932, 347=0.15530312061309814, 348=-0.014532919973134995, 349=-0.4100968539714813, 350=-0.41702941060066223, 351=-0.03720090165734291, 352=0.4377273619174957, 353=-0.11275497078895569, 354=-0.028397336602211, 355=0.12688791751861572, 356=0.3089796304702759, 357=0.1022544652223587, 358=-0.007881700992584229, 359=-0.12737412750720978, 360=0.2407739758491516, 361=0.22673989832401276, 362=-0.26267820596694946, 363=-0.1985563188791275, 364=0.2883371114730835, 365=0.3122367262840271, 366=0.07638287544250488, 367=0.07954242080450058, 368=-0.04913613200187683, 369=0.8656174540519714, 370=0.14963215589523315, 371=0.09622099995613098, 372=0.2065020203590393, 373=0.42023926973342896, 374=-0.10268233716487885, 375=0.4896998703479767, 376=-0.4853786826133728, 377=-0.14072881639003754, 378=-0.5173507928848267, 379=0.38132721185684204, 380=0.25262725353240967, 381=-0.32924580574035645, 382=-1.0359116792678833, 383=-0.17485319077968597, 384=-0.3701465129852295, 385=-0.8697187304496765, 386=0.5144445896148682, 387=0.4708768129348755, 388=-0.6273230314254761, 389=-0.5959116220474243, 390=0.775299072265625, 391=0.24602915346622467, 392=0.22195833921432495, 393=0.0023524751886725426, 394=-0.03728947043418884, 395=-0.03094223141670227, 396=-0.15462444722652435, 397=-0.034669749438762665, 398=0.4292915463447571, 399=0.32169803977012634, 400=0.16544222831726074, 401=-0.3007698059082031, 402=0.36534780263900757, 403=-0.5253899097442627, 404=0.16559278964996338, 405=0.3338485658168793, 406=0.11777622997760773, 407=0.38854655623435974, 408=0.45733317732810974, 409=-0.24424320459365845, 410=0.0915689617395401, 411=0.36092060804367065, 412=-0.19449380040168762, 413=-0.34770578145980835, 414=-0.39052504301071167, 415=-0.17414671182632446, 416=0.12289627641439438, 417=0.11486253142356873, 418=0.019910935312509537, 419=0.10526599735021591, 420=0.23396717011928558, 421=-0.09883125126361847, 422=-0.009286921471357346, 423=0.8904801607131958, 424=0.11475704610347748, 425=0.345291405916214, 426=-0.47301366925239563, 427=-0.24833618104457855, 428=0.3506036698818207, 429=0.30501192808151245, 430=-0.062260523438453674, 431=-0.24678128957748413, 432=-0.24240989983081818, 433=-0.1763041615486145, 434=-0.6848266124725342, 435=0.3886722922325134, 436=-0.300096333026886, 437=-0.38859665393829346, 438=-0.3443761467933655, 439=0.34774649143218994, 440=-0.444487065076828, 441=0.6925027370452881, 442=0.2934395670890808, 443=0.7522571086883545, 444=0.04532145708799362, 445=6.469404697418213, 446=-0.4607214331626892, 447=-0.11063402146100998, 448=0.4674191176891327, 449=0.03281670808792114, 450=-0.18718792498111725, 451=0.631888210773468, 452=-0.018420681357383728, 453=-0.5640164613723755, 454=-0.5204675197601318, 455=-0.2229187786579132, 456=0.36513474583625793, 457=0.09127238392829895, 458=-0.31675559282302856, 459=0.25613102316856384, 460=0.04067419469356537, 461=-0.49958667159080505, 462=0.004863567650318146, 463=0.37281471490859985, 464=-0.12441
*** WARNING: skipped 7648 bytes of output ***

[saif-ellafi]

Will be taking a look!

[CyborgDroid]

Hi @saifjsl, Any update on this? Thank you.

[maziyarpanahi]

Hi @CyborgDroid
We have a member assigned to this issue who's debugging the TF graph to find the issue. We will update this thread with any new findings.

[CyborgDroid]

Hi @saifjsl , do you have a rough estimate on when this will be fixed? Any kind of estimate would help. Thank you.

[saif-ellafi]

Hi @CyborgDroid we have it in our roadmap, and researched a lot. However we couldn't yet find a proper solution, the issue is somewhere between the tensorflow graph and loading it on the java API of tensorflow. We're doing all we can to get it sorted out asap.

[maziyarpanahi]

Here is a simple explanation of why the vectors are the same:

We are using the hidden state of the first token as our strategy [CLS] for sentence_embeddings. This only works if your BERT model is fine-tuned on any downstream tasks. If it's not (which is the case for pre-trained models), the hidden state of [CLS] is not a good sentence representation. If later you fine-tune the model, you may use [CLS] as well.

The only solution for pre-trained models is to compute the sentence embedding as a mean of BERT word embeddings as same as Word2Vec even though it may not have a good cosines similarity if you are comparing sentences.

We have to either change our default strategy to computing the mean of all BERT word embeddings which users with fine-tuned BERT models won't be able to use [CLS] or keep it the same and users with pre-trained have to do this manually.

One solution could be adding a way to set whether the BERT is a pre-trained model or a fine-tuned model.

I'll add this to our todo list and meanwhile, you can average all the vectors in a sentence to have sentence_embeddings.

[CyborgDroid]

Thanks for the detailed explanation @maziyarpanahi! So BERT sentence embeddings in a model that has not been trained are just averages of word embeddings? Or is that just the workaround for pre-trained models?

I was hoping BERT did some more complex analysis like n-grams or similar in it's calculations of sentence embeddings.

[maziyarpanahi]

The one we have right now, it's the hidden state of the first token [CLS]. It's not useful at all for BERT pre-trained models which we are also using as BertEmbeddings.

Since the majority of the users won't be fine-tuning their BERT models, I would suggest to our team to implement average pooling which is a workaround for BERT pre-trained models. (we are working on the same thing in WordEmbeddings(GloVe, Word2Vec))

BERT did a great job for contextual representations in both unidirectional or bidirectional for each token/word piece. Compare to context-free ones like word2vec or GloVe it knows the difference between the word bank if it's after river or if it's before deposit.

The sentence embeddings is normally used for some cosines similarity which BERT is not designed for that purpose. BERT is That's why it takes more work to do that. There are similar questions about this in bert-as-a-service repo.

[maziyarpanahi]

Continuing our last conversation about [CLS] not having a perfect sentence embedding, it also shouldn't be the same for every sentence. I am working on BertEmbeddings and with a higher layer, it won't yield the same result for setence_embeddings.
These changes will be part of a pre-release and once everything is done will be in a final release. If you are interested you can either use this pull request: https://github.com/JohnSnowLabs/spark-nlp/pull/585 or wait for the first pre-release on PyPi, Maven, and Conda.

[maziyarpanahi]

I am training new Bert models and uploading them in an hour. In 2.2.0 you have the option to choose which layer you want to extract the embeddings from. There is a param setPoolingLayer and it has 3 options:

• 0 first layer - simple embeddings - sentence_embeddings are always the same as this layer has no idea about the context. The word bank has the same embeddings whether this is bank vault or river bank
• -1 and -2 the last layer and second-to-last-hidden layer - it has context- sentence_embeddings are unique to each sentence - it may not be good since it's not fine-tuned but I will be adding another feature to ask for mean of all embeddings as the authors of Bert mentioned in their paper.

In a few hours you can try 2.2.0 release with new BertEmbeddings pre-trained models.

[DataGracie]

That’s great, Maziyar! Thank you!!

On Fri, Aug 23, 2019 at 7:57 AM Maziyar Panahi [e-mail]
wrote:

I am training new Bert models and uploading them in an hour. In 2.2.0 you
have the option to choose which layer you want to extract the embeddings
from. There is a param setPoolingLayer and it has 3 options:

• 0 first layer - simple embeddings - sentence_embeddings are always
  the same as this layer has no idea about the context. The word bank
  has the same embeddings whether this is bank vault or river bank
• -1 and -2 the last layer and second-to-last-hidden layer - it has
  context- sentence_embeddings are unique to each sentence - it may not be
  good since it's not fine-tuned but I will be adding another feature to ask
  for mean of all embeddings as the authors of Bert mentioned in their
  paper.

In a few hours you can try 2.2.0 release with new BertEmbeddings
pre-trained models.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/JohnSnowLabs/spark-nlp/issues/539?email_source=notifications&email_token=AI5FGRIQMXGOJJGW7Y2L5WTQF7GBNA5CNFSM4HVYEXJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4777TQ#issuecomment-524287950,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AI5FGRPLVPQYYZG2RQDD6TLQF7GBNANCNFSM4HVYEXJQ
.

[maziyarpanahi]

This issue has been resolved with this pull request that introduces a new annotator SentenceEmbeddings. This annotator will aggregate the word embeddings inside a sentence/document into 1 array of embeddings by either averaging or summing:
https://github.com/JohnSnowLabs/spark-nlp/pull/638