Njs: Crypto related functions failing randomly

[xbb123]

Our NJS script is expericing random exceptions and core dumps with Crypto realted functions after upgrading to 0.3.4.

2019/08/13 16:40:38 [error] 3664#3664: *3685 js exception: TypeError: (intermediate value)["createHash"] is not a function
at checkAdminCookie (antiddos.js:910)
at WAF (WAF.js:2)
2019/08/13 16:44:56 [alert] 983#983: worker process 3664 exited on signal 11 (core dumped)
2019/08/13 16:26:51 [error] 1834#1834: *478 js exception: Error: Cannot find module "md5"
at require (native)
at getIPHash (WAF.js:1059)
at request.send (native)
at sendTestPage1 (WAF.js:1072)
at advancedBrowserTest (WAF.js:1045)
at WAF (WAF.js:2)
`
function getIPHash(IP) {

var s = "IPTokenKEY";
s += IP;
var d = new Date();
var cr = require('crypto').createHash('md5')
    .update(s+d.getUTCFullYear()+d.getUTCMonth())
    .digest('base64url');
return cr.toString();

}`

[xeioex]

Hi @xbb123

Thank you for the report.

Can you reproduce the issue in njs cli?
BTW, do you use fs.readFileSync() in your code?

[xbb123]

@xeioex We are unable to reproduce the issue in cli. The issue appears to be very random. Most of the time getIPHash can output the correct hash. But we can observe a lot of js exception and "InternalError: lvlhsh insert failed".
We are not using fs.readFileSync()

[xbb123]

@xeioex One interesting finding. This commit can reliably produce js exception: TypeError: (intermediate value)["createHash"] is not a function

Nginx CPU usage also goes to 100%:

[xbb123]

We also have this under stress test (and we are not doing recursion in this function) :
js exception: RangeError: Maximum call stack size exceeded
at getIPHash (WAF.js:1059)

js exception: InternalError: unexpected property type "unknown" while getting
at getIPHash (WAF.js:1059)

And kern.log:
Aug 14 03:42:01 STAGE001 kernel: [30180.626953] nginx[2917]: segfault at 7f0000000001 ip 00007f0000000001 sp 00007fff26c9ca78 error 14 in librt-2.27.so[7f008120a000+7000] Aug 14 03:42:02 STAGE001 kernel: [30181.951424] traps: nginx[2924] general protection ip:7f00950cdce0 sp:7fff26c9c980 error:0 in ngx_http_js_module.so[7f0095096000+5d000] Aug 14 03:42:06 STAGE001 kernel: [30186.280305] traps: nginx[2925] general protection ip:7f00950cdce0 sp:7fff26c9c980 error:0 in ngx_http_js_module.so[7f0095096000+5d000] Aug 14 03:42:08 STAGE001 kernel: [30187.628303] traps: nginx[2926] general protection ip:7f00950cdce0 sp:7fff26c9c980 error:0 in ngx_http_js_module.so[7f0095096000+5d000] Aug 14 03:42:09 STAGE001 kernel: [30188.963341] traps: nginx[2927] general protection ip:7f00950cdce0 sp:7fff26c9c980 error:0 in ngx_http_js_module.so[7f0095096000+5d000] Aug 14 03:42:10 STAGE001 kernel: [30190.296473] nginx[2928]: segfault at 10000 ip 00007f00950cdce0 sp 00007fff26c9c980 error 4 in ngx_http_js_module.so[7f0095096000+5d000] Aug 14 03:42:12 STAGE001 kernel: [30191.632093] traps: nginx[2929] general protection ip:7f00950cdce0 sp:7fff26c9c980 error:0 in ngx_http_js_module.so[7f0095096000+5d000] Aug 14 03:42:13 STAGE001 kernel: [30192.967967] traps: nginx[2930] general protection ip:7f00950cdce0 sp:7fff26c9c980 error:0 in ngx_http_js_module.so[7f0095096000+5d000] Aug 14 03:42:14 STAGE001 kernel: [30194.303718] traps: nginx[2931] general protection ip:7f00950cdce0 sp:7fff26c9c980 error:0 in ngx_http_js_module.so[7f0095096000+5d000] Aug 14 03:42:16 STAGE001 kernel: [30195.635290] traps: nginx[2932] general protection ip:7f00950cdce0 sp:7fff26c9c980 error:0 in ngx_http_js_module.so[7f0095096000+5d000] Aug 14 03:42:17 STAGE001 kernel: [30196.967779] traps: nginx[2933] general protection ip:7f00950cdce0 sp:7fff26c9c980 error:0 in ngx_http_js_module.so[7f0095096000+5d000] Aug 14 03:42:24 STAGE001 kernel: [30204.282610] nginx[2934]: segfault at 15 ip 00007f00950ba7bb sp 00007fff26c9c9e0 error 4 in ngx_http_js_module.so[7f0095096000+5d000] Aug 14 03:42:38 STAGE001 kernel: [30217.762739] nginx[2936]: segfault at d ip 00007f00950ba7bb sp 00007fff26c9c9e0 error 4 in ngx_http_js_module.so[7f0095096000+5d000]

[xeioex]

@xbb123

I can confirm that https://hg.nginx.org/njs/rev/04d7a5d93ae6 introduced the regression.

js.js:
function github206(r) {                                                                                                                                                                                                    
    require('crypto').createHash('md5')                    
    // require('crypto').createHash also causes crash                                                                            
    r.return(200, "OK") 
}

nginx.conf:
server {   
    listen 8000;  
    location /github206 {                                                                                              
        js_content github206;                                                                                          
   }
}

wrk -d3 -c12 http://localhost:8000/github206 causes crash after ~1-30 requests

But

$ njs
> for (var i = 0; i < 1000000; i++) {require('crypto').createHash('md5')}
undefined

works just fine.

[xeioex]

@xbb123

Please try the following patch: https://gist.github.com/xeioex/eb4f578478dec8bff40dc5984a15f356

[xbb123]

@xeioex The patch does pass all our test cases. It looks good 👍
Given the nature of this issue, are we going to have a bug fix version soon?

[xeioex]

@xbb123

Given the nature of this issue, are we going to have a bug fix version soon?

Yes, the issue is quite serious because it impacts any configuration which uses require().

[xeioex]

@xbb123

BTW, as a workaround you can import crypto using import statement, it is not affected.

import crypto from 'crypto';

function getIPHash(IP) {
    crypto.createHash('md5') ...
}