Ray: [ray] Critical security vulnerabilities in Redis/Lua dependency

[aregm]

What is the problem?

• Ray version and other system information (Python version, TensorFlow version, OS): 1.x

3 Ray components have critical security vulnerabilities that prevent them from deploying into any enterprise or cloud infrastructures for production usage.

• c-ares
  Current version - 1.14.0 OUTDATED
  Latest version - 1.16.1
  CVE-2016-5180
  CVE-2007-3152
  CVE-2007-3153

• Lua
  Current version - 5.1.5 OUTDATED
  Latest version - 5.4.1
  CVE-2020-15889
  CVE-2020-24342
  CVE-2020-15888
  CVE-2014-5461
  CVE-2020-15945

• Redis
  Current version -5.0.9 OUTDATED
  Latest version - 6.0.9
  CVE-2020-14147

[wuisawesome]

Thanks, I believe all those dependencies come from Redis. @barakmich is already working on upgrading our redis dependency.

[rkooo567]

@wuisawesome We should include this update to 1.0.1?

[wuisawesome]

@barakmich your call.

[aregm]

@barakmich any response?

[edoakes]

Won't make the cut for 1.0.1

[ericl]

We have to fix a number of upgrade issues with the latest redis, but it should be available in nightlies soon.

[barakmich]

As regards Redis: We've been fine this whole time.

Just because a newer version is available (a major version bump, which means upgrade issues), doesn't mean the CVE applies to the version we currently have. The issue quotes CVE-2020-14147.

According to the folks at Gentoo, it's fixed as of Redis 5.0.9: https://security.gentoo.org/glsa/202008-17
And according to Debian, it's fixed upstream in 5.0.8: https://security-tracker.debian.org/tracker/CVE-2020-14147
(Specifically by commit: https://github.com/antirez/redis/commit/ef764dde1cca2f25d00686673d1bc89448819571)

We have 5.0.9 in 1.0.1, 1.0.0, and 0.8.7: https://github.com/ray-project/ray/blob/releases/1.0.0/bazel/ray_deps_setup.bzl -- even by the issue's own statement.

Now, to be fair, our Windows version currently lags. So we could bump that relatively painlessly, even for 1.0.1 -- 5.0.9 is available.

Lua is used to build redis, sure, but we don't use it directly, and c-ares -- for resolving addresses -- is also not a direct dependency, but in both cases I welcome finding the dependency we might need to bump.

So this hasn't truly been a problem this whole time, and we can bump Redis at our leisure.

[devin-petersohn]

So this hasn't truly been a problem this whole time, and we can bump Redis at our leisure.

@barakmich I am disappointed that this is your response, it is not an attitude that is friendly to enterprise users. Enterprise users are very sensitive to security issues, and, by extension, Ray developers will need to be sensitive too if enterprise adoption is expected. I am sure it is a big ask to update the Redis dependency, but we cannot suggest to our customers to use Ray when there are known vulnerabilities in the Redis version.

It does not matter if other groups have found that the vulnerabilities don't affect the version of Redis Ray depends on, it is in NVD as <6.0.3 (https://nvd.nist.gov/vuln/detail/CVE-2020-14147). As long as it continues to be flagged as a vulnerability in BDBA, we cannot recommend Ray to customers or release tools and products that build on Ray.

To be clear, we opened this issue so that we can recommend Ray to enterprise customers. Until these CVEs are resolved, we cannot.

[wuisawesome]

Hey just to recap this conversation:

It seems like from a purely technical perspective, Ray has a different threat model than Redis, so a RCE vulnerability for Redis isn't a a dealbreaker (since the attacker that can connect to Redis would have a much easier time executing remote code by connecting as a Ray driver).

Using an old version of Redis _is_ still a problem though because from a compliance perspective, downstream patches are not recognized by NVD. Therefore, we could think of upgrading to Redis >= 6.0.3 as a high priority enterprise compliance feature?

@devin-petersohn @barakmich does this sound right?

[aregm]

Therefore, we could think of upgrading to Redis >= 6.0.3 as a high priority enterprise compliance feature?
@wuisawesome Thanks, that will solve the problem.

[barakmich]

@wuisawesome +1 to upgrading to Redis >= 6.0.3 to unblock our enterprise users. Maybe we should break out upgrading Redis as a separate issue for tracking since we've already got #11371 in progress. Perhaps I was hasty earlier; my focus was on the security risks to Ray's users and didn't account for enterprise requirements.

To be clear for the community; this vulnerability (CVE-2020-14147) is not present in current Ray and has not been since 2020-07-03 nightly or release 0.8.6.

Here is the work we did to support that conclusion:

Q: When was this vulnerability fixed in Redis?:

• https://nvd.nist.gov/vuln/detail/CVE-2020-14147
  
  
  
  • Has references to the pull request that fixes this: https://github.com/redis/redis/pull/6875
  
  
  • And the specific commit that does as well: https://github.com/antirez/redis/commit/ef764dde1cca2f25d00686673d1bc89448819571
  
  
• This patch was backported to the 5.0 release branch on 2020-02-10.
  
  
  
  • Specifically this commit that exists in the 5.0.8 and 5.0.9 release branch: https://github.com/redis/redis/commit/16b2d07f0a9b58027611dab7f97788d37cb5ab84
  
  
• The release SHAs that contain that commit in Redis are:
  
  
  
  • 5.0.8: https://github.com/redis/redis/commit/1f7d08b76dc83fd2aa43ee7fc04ab4ee01016628
  
  
  • 5.0.9: https://github.com/redis/redis/commit/ce878b6ed5df1a7b3e87a8af2d1262de4c5bb828
  
  

Q: How about Ray?

• Ray was updated to build against Redis 5.0.9 on 2020-07-02
  
  
  
  • https://github.com/ray-project/ray/commit/a8e4ddbd57f2cdc9a274138904a8857f8427dc39
  
  
• Before that it was on 5.0.3 -- which may still be vulnerable.

Q: How do we know Ray built the right Redis?

• In the above commit, the SHA256 of the 5.0.9 source used was db9bf149e237126f9bb5f40fb72f33701819555d06f16e9a38b4949794214201 which matches the reported sha256sum of https://github.com/antirez/redis/archive/5.0.9.tar.gz (as of time of writing).
• We can look in that archive and verify that the sha256 of the file inside matches that needed to fix the vulnerability:
  
  
  
  • From the git commit referenced in 5.0.9 (ce878b6):
    
    
    
    
    
    • 57bc18e91a747a846ba76c070d9213452283c0ed9e652861ee7fabf87d590878 deps/lua/src/lua_struct.c
    
    
    
  
  
  • From the tarball with the exact sha256 that ray built against:
    
    
    
    
    
    • 57bc18e91a747a846ba76c070d9213452283c0ed9e652861ee7fabf87d590878 deps/lua/src/lua_struct.c
    
    
    
  
  
  • Commit from the NIST page (ef764dde1cca2f25d00686673d1bc89448819571):
    
    
    
    
    
    • 57bc18e91a747a846ba76c070d9213452283c0ed9e652861ee7fabf87d590878 deps/lua/src/lua_struct.c
    
    
    
  
  

[devin-petersohn]

To be clear for the community; this vulnerability (CVE-2020-14147) is not present in current Ray and has not been since 2020-07-03 nightly or release 0.8.6.

This has not been independently verified by NIST. The links on the NVD are there for information only and are links to third party resources. It should be up to the user whether to accept these third party opinions, and we should be careful not to obfuscate the fact that NIST has this CVE classified as up to (excluding) Redis 6.0.3, which means Ray is currently vulnerable (scroll to the bottom of the page to see it). We do not yet know if the only changes necessary to fix the vulnerability are those found in the PR, NIST is currently reanalyzing the CVE.

According to the NVD this vulnerability is still present in Ray.

I do not think we should close this issue in favor of opening an issue for tracking an update to Redis. The issue is not resolved.

[ericl]

@aregm #11371 fixes this in nightly Ray; can you verify the issue no longer exists in the latest wheels?

[devin-petersohn]

Thanks @ericl, we began testing earlier today. It takes some time to get results back and we will share once we have them.

[aregm]

@ericl , thanks for updating the dependencies, but only redis and c-ares are updated. Lua dependency vulnerability still persists. Can you please update Lua to version 5.4.1?

[devin-petersohn]

@ericl @richardliaw The Lua version is still a problem, updating Lua to 5.4.1 will resolve the issue.

[robertnishihara]

@devin-petersohn does the tool provide any more info (e.g., location or file or something)? We don't directly use Lua in the project (and I can't find references to it in the codebase right now). Redis uses Lua for some stuff, so it must have something to do with Redis.

[aregm]

@robertnishihara @zhe-thoughts The tool scans the binary and its dependencies, and as the Lua is a dynamic library loaded by redis-server and is a part of the Redis package dependencies, it triggers the scan and the list of vulnerabilities. Look for the file liblua.so and links liblua.so.5.1, liblua.so.5.1.5. It also could be installed in your build dependencies on a system level.

Output from the tool

Current version - 5.1.5 OUTDATED
Latest version - 5.4.1
CVE-2020-15889
CVE-2020-24342
CVE-2020-15888
CVE-2014-5461
CVE-2020-15945

So if you are not actively using it, then just the upgrade of the version should be sufficient. The scan tool needs to see a reference to the right version of the library and appropriate symbols.

[ericl]

@barakmich any idea why upgrading redis doesn't also pull in the newest Lua? I don't think we have a direct dependency.