Tfjs: AdadeltaOptimizer is numerically unstable on iOS devices (half-float devices)

[nsthorat]

I'm disabling this test until we understand where numerical instability is coming from.

Mobile Safari 11.0.0 (iOS 11.0.1) AdadeltaOptimizer {} basic FAILED
    Error: Arrays differ: actual[0] = NaN, expected[0] = 0.8.
    Actual:   NaN,NaN.
    Expected: 0.8,1.6. in src/test_util.js (line 140)
    expectArraysClose@src/test_util.ts:98:10 <- src/test_util.js:140:40
    src/optimizers/adadelta_optimizer_test.ts:52:22 <- src/optimizers/adadelta_optimizer_test.js:55:50
    <Jasmine>

[manrajgrover]

@nsthorat Not 100% sure since I don't have access to any iOS device to test this, but it might be because of the default value of small epsilon being added set as 1e-8 which converts to 0.0 in float16. Since accumulatedGrad in test is 0.0, and epsilon is 0.0 itself, numerical instability occurs on this line which leads to division by zero. I feel changing it something like 1e-5 should fix it.

[nsthorat]

Nice find! Would you want to make that change? You can check ENV.get('WEBGL_RENDER_FLOAT32_TEXTURE_ENABLED'), once you make a PR it will run on browserstack on a real iOS every time you push to that branch.