Pants: Hangs and crashes in `./v2 fmt ...`

[jsirois]

I use ./v2 fmt src/python/pants/backend/python/rules: src/python/pants/backend/python/subsystems: for the command on linux with no changed files and I've observed 2 forms of crash as well in hangs in the swim-lanes UI for 6 digests and 2 snapshots. All runs fail in one of these 3 ways.

In order of highest frequency 1st:

1. Panic Snapshotting:

   19:34:56 [ERROR] panic at 'state = Snapshot { is_running: true, is_notified: false, is_released: false, is_complete: false, is_canceled: false, is_join_interested: false, has_join_waker: false, is_final_ref: false }', /home/jsirois/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.13/src/task/state.rs:227
   

2. Panic starting a tokio runtime from within another one:

   19:33:16 [ERROR] panic at 'Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks.', /home/jsirois/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.13/src/runtime/enter.rs:19
   

3. A hang in the v2 ui with 8 swimlanes, 6 DigestFile(...) and 2 Scandir(...) (one for "src/python/pants/backend/python/rules" and one for "src/python/pants/backend/python/subsystems")

A git bisect leads to 6db9ab4becf5b520c12db297b10c9e6e78aa994e:

jsirois@gill ~/dev/pantsbuild/jsirois-pants ((6db9ab4be...)|BISECTING) $ git bisect bad
6db9ab4becf5b520c12db297b10c9e6e78aa994e is the first bad commit
commit 6db9ab4becf5b520c12db297b10c9e6e78aa994e
Author: Henry Fuller <[email protected]>
Date:   Thu Mar 26 12:01:39 2020 -0700

    Add notify fs watcher to engine. (#9318)

...

[jsirois]

Noting that formatting is an intersting case in the context of file watches since it modifies files in the workspace at the completion of the goal:
https://github.com/pantsbuild/pants/blob/9de623801ae82497e582359040087a7de5ec73bc/src/python/pants/rules/core/fmt.py#L174

It does this via the engine, so presumably there is a crease there to remedy if that is part of the problem.

[hrfuller]

Interesting, you are seeing some different failures modes than me and Eric. We see the black binary run multiple times occasionally ending with a long exception about computing the Select for Fmt goal. I'm going to try to repro what you're seeing a bit more.

Noting that formatting is an intersting case in the context of file watches since it modifies files in the workspace at the completion of the goal:

My hunch is that invalidation of the files which were formatted in the build root cause the goal to run again, but I'm not sure exactly why, or where that happens. Are all nodes in the build graph required to be in the completed state for the goal to finish successfully?

[hrfuller]

@jsirois are you using a linux machine?

[jsirois]

Yes. That was buried in the 1st sentence of the description.

[jsirois]

I also get a large Select but elided that output figuring it was not relevant.

So, looks like this in full for me (I hit the most common of the three cases above this time):

$ ./v2 fmt src/python/pants/backend/python/rules: src/python/pants/backend/python/subsystems:

All done! ✨ 🍰 ✨
28 files left unchanged.

All done! ✨ 🍰 ✨
28 files left unchanged.

All done! ✨ 🍰 ✨
28 files left unchanged.

All done! ✨ 🍰 ✨
28 files left unchanged.

All done! ✨ 🍰 ✨
28 files left unchanged.

All done! ✨ 🍰 ✨
28 files left unchanged.

All done! ✨ 🍰 ✨
28 files left unchanged.

All done! ✨ 🍰 ✨
28 files left unchanged.

All done! ✨ 🍰 ✨
28 files left unchanged.

22:30:30 [WARN] /home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/base/exception_sink.py:431: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  process_title=setproctitle.getproctitle(),

timestamp: 2020-03-29T15:30:30.940304
Exception caught: (pants.engine.scheduler.ExecutionError)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_loader.py", line 98, in <module>
    main()
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_loader.py", line 94, in main
    PantsLoader.run()
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_loader.py", line 90, in run
    cls.load_and_execute(entrypoint)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_loader.py", line 83, in load_and_execute
    entrypoint_main()
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_exe.py", line 33, in main
    runner.run(start_time)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_runner.py", line 102, in run
    runner.run()
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/local_pants_runner.py", line 344, in run
    engine_result = self._maybe_run_v2(v2)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/local_pants_runner.py", line 292, in _maybe_run_v2
    return self.graph_session.run_goal_rules(
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/init/engine_initializer.py", line 252, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(goal_product, params)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/scheduler.py", line 518, in run_goal_rule
    self._trace_on_error([exc], request)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/engine/scheduler.py", line 483, in _trace_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:
Computing Select((<pants.engine.console.Console object at 0x7d7dd75a9c40>, AddressSpecs(dependencies=(SiblingAddresses(directory='src/python/pants/backend/python/rules'), SiblingAddresses(directory='src/python/pants/backend/python/subsystems')), matcher=AddressSpecsMatcher(tags=(), exclude_patterns=())), OptionsBootstrapper(env_tuples=(('PANTS_DEV', '1'),), bootstrap_args=('--no-v1', '--v2'), args=('/home/jsirois/dev/pantsbuild/jsirois-pants/src/python/pants/bin/pants_loader.py', '--no-v1', '--v2', '--v2-ui', 'fmt', 'src/python/pants/backend/python/rules:', 'src/python/pants/backend/python/subsystems:'), config=_ChainedConfig(chained_configs=(_SingleFileConfig(config_path='/home/jsirois/.pants.rc', content_digest='a27c7d82405ec33f3f693569e578dffa1fb6d55a', values=_IniValues(parser=<configparser.ConfigParser object at 0x7f36038152e0>)), _SingleFileConfig(config_path='/home/jsirois/dev/pantsbuild/jsirois-pants/pants.toml', content_digest='ad65e29fa6a212c2756cb902e4d0a84adf30dcec', values=_TomlValues(values={'DEFAULT': {'buildroot': '/home/jsirois/dev/pantsbuild/jsirois-pants', 'homedir': '/home/jsirois', 'user': 'jsirois', 'pants_bootstrapdir': '/home/jsirois/.cache/pants', 'pants_configdir': '/home/jsirois/.config/pants', 'pants_workdir': '/home/jsirois/dev/pantsbuild/jsirois-pants/.pants.d', 'pants_supportdir': '/home/jsirois/dev/pantsbuild/jsirois-pants/build-support', 'pants_distdir': '/home/jsirois/dev/pantsbuild/jsirois-pants/dist', 'jvm_options': ['-Xmx1g'], 'local_artifact_cache': '%(pants_bootstrapdir)s/artifact_cache'}, 'GLOBAL': {'print_exception_stacktrace': True, 'v2': False, 'pythonpath': ['%(buildroot)s/contrib/avro/src/python', '%(buildroot)s/contrib/awslambda/python/src/python', '%(buildroot)s/contrib/buildrefactor/src/python', '%(buildroot)s/contrib/codeanalysis/src/python', '%(buildroot)s/contrib/cpp/src/python', '%(buildroot)s/contrib/confluence/src/python', '%(buildroot)s/contrib/errorprone/src/python', '%(buildroot)s/contrib/findbugs/src/python', '%(buildroot)s/contrib/go/src/python', '%(buildroot)s/contrib/googlejavaformat/src/python', '%(buildroot)s/contrib/jax_ws/src/python', '%(buildroot)s/contrib/mypy/src/python', '%(buildroot)s/contrib/node/src/python', '%(buildroot)s/contrib/scalajs/src/python', '%(buildroot)s/contrib/scrooge/src/python', '%(buildroot)s/contrib/thrifty/src/python', '%(buildroot)s/pants-plugins/src/python'], 'backend_packages': {'add': ['pants.backend.docgen', 'pants.contrib.avro', 'pants.contrib.awslambda.python', 'pants.contrib.buildrefactor', 'pants.contrib.codeanalysis', 'pants.contrib.cpp', 'pants.contrib.confluence', 'pants.contrib.errorprone', 'pants.contrib.findbugs', 'pants.contrib.go', 'pants.contrib.googlejavaformat', 'pants.contrib.jax_ws', 'pants.contrib.scalajs', 'pants.contrib.node', 'pants.contrib.scrooge', 'pants.contrib.thrifty', 'internal_backend.repositories', 'internal_backend.sitegen', 'internal_backend.utilities'], 'remove': ['pants.backend.python.lint.isort']}, 'backend_packages2': ['pants.backend.project_info', 'pants.backend.python', 'pants.backend.python.lint.black', 'pants.backend.python.lint.docformatter', 'pants.backend.python.lint.flake8', 'pants.backend.python.lint.isort', 'pants.backend.native', 'internal_backend.rules_for_testing'], 'pantsd_invalidation_globs': {'add': ['src/python/**/*.py']}, 'pants_ignore': {'add': ['/build-support/virtualenvs/', '/build-support/*.venv/', '/build-support/bin/native/src', '/src/rust/engine/target']}}, 'cache': {'read_from': [], 'write_to': [], 'bootstrap': {'read_from': ['%(local_artifact_cache)s'], 'write_to': ['%(local_artifact_cache)s']}, 'resolve': {'coursier': {'read_from': ['%(local_artifact_cache)s'], 'write_to': ['%(local_artifact_cache)s']}}}, 'ivy': {'ivy_settings': '%(pants_supportdir)s/ivy/ivysettings.xml', 'ivy_profile': '%(pants_supportdir)s/ivy/ivy.xml'}, 'gen': {'scrooge': {'service_deps': "{\n  'java': [\n    '3rdparty:slf4j-api',\n    '3rdparty:thrift',\n    '3rdparty/jvm/com/twitter:finagle-thrift',\n    '3rdparty/jvm/com/twitter:scrooge-core',\n  ],\n  'scala': [\n    '3rdparty:thrift',\n    '3rdparty/jvm/com/twitter:finagle-thrift',\n    '3rdparty/jvm/com/twitter:scrooge-core',\n  ],\n}\n", 'structs_deps': "{\n  'java': [\n    '3rdparty:thrift',\n    '3rdparty/jvm/com/twitter:scrooge-core',\n  ],\n  'scala': [\n    '3rdparty:thrift',\n    '3rdparty/jvm/com/twitter:scrooge-core',\n  ],\n}\n", 'service_exports': "{\n  'java': [\n    '3rdparty:thrift',\n  ],\n  'scala': [\n    '3rdparty:thrift',\n    '3rdparty/jvm/com/twitter:finagle-thrift',\n    '3rdparty/jvm/com/twitter:scrooge-core',\n  ],\n}\n", 'structs_exports': "{\n  'java': [\n    '3rdparty:thrift',\n  ],\n  'scala': [\n    '3rdparty:thrift',\n    '3rdparty/jvm/com/twitter:scrooge-core',\n  ],\n}\n"}, 'thrift-java': {'deps': ['3rdparty:thrift']}, 'thrift-py': {'deps': ['examples/3rdparty/python:thrift']}, 'thrifty': {'allow_dups': True}, 'antlr-py': {'antlr3_deps': ['3rdparty/python:antlr-3.1.3']}, 'go-protobuf': {'import_target': 'contrib/go/examples/src/protobuf/org/pantsbuild/example:protobuf-deps'}}, 'compile': {'errorprone': {'command_line_options': ['-Xep:CatchAndPrintStackTrace:OFF', '-Xep:StringSplitter:OFF'], 'exclude_patterns': ['contrib/errorprone/tests/java/org/pantsbuild/contrib/errorprone:error', 'testprojects/src/java/org/pantsbuild/testproject/.*']}, 'findbugs': {'max_rank': 4, 'fail_on_error': True, 'exclude_patterns': ['contrib/findbugs/tests/java/org/pantsbuild/contrib/findbugs:high', 'testprojects/src/java/org/pantsbuild/testproject/.*']}, 'rsc': {'jvm_options': ['-Xmx4g', '-XX:+UseConcMarkSweepGC', '-XX:ParallelGCThreads=4'], 'args': ['-C-encoding', '-CUTF-8', '-S-encoding', '-SUTF-8', '-S-g:vars', '-S-target:jvm-1.8'], 'warning_args': ['-S-deprecation', '-S-unchecked', '-S-feature'], 'no_warning_args': ['-S-nowarn']}, 'javac': {'args': ['-encoding', 'UTF-8', '-J-Xmx2g'], 'warning_args': ['-Xlint:deprecation', '-Xlint:empty', '-Xlint:finally', '-Xlint:overrides', '-Xlint:static', '-Xlint:unchecked', '-Xlint:try'], 'no_warning_args': ['-Xlint:none']}}, 'black': {'config': 'pyproject.toml'}, 'checkstyle': {'config': '%(pants_supportdir)s/checkstyle/coding_style.xml'}, 'docformatter': {'args': ['--wrap-summaries=100', '--wrap-descriptions=100']}, 'eslint': {'setupdir': '%(pants_supportdir)s/eslint', 'config': '%(pants_supportdir)s/eslint/.eslintrc', 'ignore': '%(pants_supportdir)s/eslint/.eslintignore'}, 'flake8': {'config': 'build-support/flake8/.flake8', 'extra_requirements': {'add': ['flake8-pantsbuild>=2.0,<3']}}, 'google-java-format': {'skip': True}, 'isort': {'config': ['.isort.cfg', 'contrib/.isort.cfg', 'examples/.isort.cfg']}, 'scalafmt': {'skip': True}, 'scalafix': {'skip': True}, 'scalastyle': {'config': '%(buildroot)s/build-support/scalastyle/scalastyle_config.xml'}, 'lint': {'scalastyle': {'excludes': '%(buildroot)s/build-support/scalastyle/excludes.txt'}}, 'protoc': {'gen': {'go-protobuf': {'version': '3.4.1'}}}, 'scala': {'version': 2.12}, 'java': {'strict_deps': True}, 'jvm': {'options': ['-Xmx1g'], 'bench': {'options': ['-Xmx1g']}, 'run': {'jvm': {'options': ['-Xmx1g']}}, 'test': {'junit': {'options': ['-Djava.awt.headless=true', '-Xmx1g']}}}, 'jvm-platform': {'default_platform': 'java8', 'platforms': "{\n  'java7': {'source': '7', 'target': '7', 'args': [] },\n  'java8': {'source': '8', 'target': '8', 'args': [] },\n  'java9': {'source': '9', 'target': '9', 'args': [] },\n}\n"}, 'pants-releases': {'branch_notes': "{\n  'master': 'src/python/pants/notes/master.rst',\n  '1.0.x': 'src/python/pants/notes/1.0.x.rst',\n  '1.1.x': 'src/python/pants/notes/1.1.x.rst',\n  '1.2.x': 'src/python/pants/notes/1.2.x.rst',\n  '1.3.x': 'src/python/pants/notes/1.3.x.rst',\n  '1.4.x': 'src/python/pants/notes/1.4.x.rst',\n  '1.5.x': 'src/python/pants/notes/1.5.x.rst',\n  '1.6.x': 'src/python/pants/notes/1.6.x.rst',\n  '1.7.x': 'src/python/pants/notes/1.7.x.rst',\n  '1.8.x': 'src/python/pants/notes/1.8.x.rst',\n  '1.9.x': 'src/python/pants/notes/1.9.x.rst',\n  '1.10.x': 'src/python/pants/notes/1.10.x.rst',\n  '1.11.x': 'src/python/pants/notes/1.11.x.rst',\n  '1.12.x': 'src/python/pants/notes/1.12.x.rst',\n  '1.13.x': 'src/python/pants/notes/1.13.x.rst',\n  '1.14.x': 'src/python/pants/notes/1.14.x.rst',\n  '1.15.x': 'src/python/pants/notes/1.15.x.rst',\n  '1.16.x': 'src/python/pants/notes/1.16.x.rst',\n  '1.17.x': 'src/python/pants/notes/1.17.x.rst',\n  '1.18.x': 'src/python/pants/notes/1.18.x.rst',\n  '1.19.x': 'src/python/pants/notes/1.19.x.rst',\n  '1.20.x': 'src/python/pants/notes/1.20.x.rst',\n  '1.21.x': 'src/python/pants/notes/1.21.x.rst',\n  '1.22.x': 'src/python/pants/notes/1.22.x.rst',\n  '1.23.x': 'src/python/pants/notes/1.23.x.rst',\n  '1.24.x': 'src/python/pants/notes/1.24.x.rst',\n  '1.25.x': 'src/python/pants/notes/1.25.x.rst',\n  '1.26.x': 'src/python/pants/notes/1.26.x.rst',\n}\n"}, 'publish': {'jar': {'ivy_settings': '%(pants_supportdir)s/ivy/publish.ivysettings.xml', 'push_postscript': '# Prevent Travis CI from running for this automated JAR publish commit:\n#  https://docs.travis-ci.com/user/customizing-the-build/#skipping-a-build\n[ci skip]\n', 'repos': "{\n  'public': {  # must match the name of the `Repository` object that you defined in your plugin.\n    'resolver': 'oss.sonatype.org', # must match hostname in ~/.netrc and the <url> parameter\n                                    # in your custom ivysettings.xml.\n    'auth': 'build-support/ivy:netrc',  # Pants spec to a 'credentials()' object.\n    'help': 'Configure your ~/.netrc for oss.sonatype.org access.'\n  }\n}\n", 'restrict_push_branches': ['master'], 'restrict_push_urls': ['[email protected]:pantsbuild/pants.git', 'https://github.com/pantsbuild/pants.git'], 'verify_commit': False}}, 'python-setup': {'interpreter_cache_dir': '%(pants_bootstrapdir)s/python_cache/interpreters', 'resolver_cache_dir': '%(pants_bootstrapdir)s/python_cache/requirements'}, 'pytest': {'pytest_plugins': {'add': ['setuptools==44.0.0', 'ipdb']}}, 'test': {'pytest': {'timeouts': True, 'timeout_default': 60}, 'junit': {'timeouts': True, 'timeout_default': 60}, 'go': {'chroot': False}}, 'buildgen': {'go': {'materialize': True, 'remote': True, 'fail_floating': True}}, 'reference': {'pants_reference_template': 'reference/pants_reference_body.html', 'build_dictionary_template': 'reference/build_dictionary_body.html'}, 'markdown': {'fragment': True}, 'sitegen': {'config_path': 'src/docs/docsite.json'}, 'native-build-step': {'cpp-compile-settings': {'default_compiler_option_sets': ['glibcxx_use_old_abi'], 'compiler_option_sets_enabled_args': "{'glibcxx_use_old_abi': ['-D_GLIBCXX_USE_CXX11_ABI=0']}", 'compiler_option_sets_disabled_args': "{'glibcxx_use_old_abi': ['-D_GLIBCXX_USE_CXX11_ABI=1']}"}}, 'libc': {'enable_libc_search': True}, 'sourcefile-validation': {'config': '@build-support/regexes/config.yaml'}, 'coursier': {'repos': ['https://maven-central.storage-download.googleapis.com/maven2', 'https://repo1.maven.org/maven2', 'https://raw.githubusercontent.com/toolchainlabs/binhost/master/', 'file://%(homedir)s/.m2/repository'], 'fetch_options': {'add': ['--retry-count', '3', '--no-default']}}}))))), Workspace(_scheduler=<pants.engine.scheduler.SchedulerSession object at 0x7d7dd759b730>)), Fmt)
  <None>



22:30:31 [ERROR] panic at 'state = Snapshot { is_running: true, is_notified: false, is_released: false, is_complete: false, is_canceled: false, is_join_interested: false, has_join_waker: false, is_final_ref: false }', /home/jsirois/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.13/src/task/state.rs:227
22:30:31 [ERROR] Please file a bug at https://github.com/pantsbuild/pants/issues.

[hrfuller]

Thanks, that panic is useful info, which I'm not getting on OSX. It would be helpful if you could provide a run with RUST_BACKTRACE=full also.

[hrfuller]

Hey @jsirois re:

It would be helpful if you could provide a run with RUST_BACKTRACE=full also.

This would still be helpful as I work on a long term solution for this issue.

[jsirois]

@hrfuller here are 3 slight variations from the runtime within runtime - which becomes the most frequent when using RUST_BACKTRACE:
1.
thread 'tokio-runtime-worker' panicked at 'Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks.', /home/jsirois/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.13/src/runtime/enter.rs:19:5 stack backtrace: 0: 0x7efe26c140a4 - <unknown> 1: 0x7efe26c39a9c - <unknown> 2: 0x7efe26c0d527 - <unknown> 3: 0x7efe26c16985 - <unknown> 4: 0x7efe26c16671 - <unknown> 5: 0x7efe26c16fca - <unknown> 6: 0x7efe26b22bfe - <unknown> 7: 0x7efe26b3615d - <unknown> 8: 0x7efe26b36204 - <unknown> 9: 0x7efe264f8ef4 - <unknown> 10: 0x7efe26512e42 - <unknown> 11: 0x7efe265272d0 - <unknown> 12: 0x7efe26516125 - <unknown> 13: 0x7efe2650c2cf - <unknown> 14: 0x7efe26556853 - <unknown> 15: 0x7efe264f497c - <unknown> 16: 0x7efe26c1f857 - <unknown> 17: 0x7efe26539cd9 - <unknown> 18: 0x7efe26b39c2f - <unknown> 19: 0x7efe26b387db - <unknown> 20: 0x7efe26b22e51 - <unknown> 21: 0x7efe26c1f857 - <unknown> 22: 0x7efe26b3dff4 - <unknown> 23: 0x7efe26b20f5f - <unknown> 24: 0x7efe26b22f85 - <unknown> 25: 0x7efe26c1f857 - <unknown> 26: 0x7efe26b2325a - <unknown> 27: 0x7efe26c060cf - <unknown> 28: 0x7efe26c1ea40 - <unknown> 29: 0x7efe2a71846f - start_thread 30: 0x7efe2abf03d3 - clone 31: 0x0 - <unknown> free(): double free detected in tcache 2
2.
thread 'tokio-runtime-worker' panicked at 'Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks.', /home/jsirois/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.13/src/runtime/enter.rs:19:5 stack backtrace: 0: 0x7f7c167ff0a4 - <unknown> 1: 0x7f7c16824a9c - <unknown> 2: 0x7f7c167f8527 - <unknown> 3: 0x7f7c16801985 - <unknown> 4: 0x7f7c16801671 - <unknown> 5: 0x7f7c16801fca - <unknown> 6: 0x7f7c1670dbfe - <unknown> 7: 0x7f7c1672115d - <unknown> 8: 0x7f7c16721204 - <unknown> 9: 0x7f7c160e3ef4 - <unknown> 10: 0x7f7c160fde42 - <unknown> 11: 0x7f7c161122d0 - <unknown> 12: 0x7f7c16101125 - <unknown> 13: 0x7f7c160f72cf - <unknown> 14: 0x7f7c16141853 - <unknown> 15: 0x7f7c160df97c - <unknown> 16: 0x7f7c1680a857 - <unknown> 17: 0x7f7c16124cd9 - <unknown> 18: 0x7f7c16724c2f - <unknown> 19: 0x7f7c167237db - <unknown> 20: 0x7f7c1670de51 - <unknown> 21: 0x7f7c1680a857 - <unknown> 22: 0x7f7c16728ff4 - <unknown> 23: 0x7f7c1670bf5f - <unknown> 24: 0x7f7c1670df85 - <unknown> 25: 0x7f7c1680a857 - <unknown> 26: 0x7f7c1670e25a - <unknown> 27: 0x7f7c167f10cf - <unknown> 28: 0x7f7c16809a40 - <unknown> 29: 0x7f7c1a30346f - start_thread 30: 0x7f7c1a7db3d3 - clone 31: 0x0 - <unknown> double free or corruption (fasttop)
3.
thread 'tokio-runtime-worker' panicked at 'Cannot start a runtime from within a runtime. This happens because a function (like `block_on`) attempted to block the current thread while the thread is being used to drive asynchronous tasks.', /home/jsirois/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.13/src/runtime/enter.rs:19:5 stack backtrace: 0: 0x7fba4f22f0a4 - <unknown> 1: 0x7fba4f254a9c - <unknown> 2: 0x7fba4f228527 - <unknown> 3: 0x7fba4f231985 - <unknown> 4: 0x7fba4f231671 - <unknown> 5: 0x7fba4f231fca - <unknown> 6: 0x7fba4f13dbfe - <unknown> 7: 0x7fba4f15115d - <unknown> 8: 0x7fba4f151204 - <unknown> 9: 0x7fba4eb13ef4 - <unknown> 10: 0x7fba4eb2de42 - <unknown> 11: 0x7fba4eb422d0 - <unknown> 12: 0x7fba4eb31125 - <unknown> 13: 0x7fba4eb272cf - <unknown> 14: 0x7fba4eb71853 - <unknown> 15: 0x7fba4eb0f97c - <unknown> 16: 0x7fba4f23a857 - <unknown> 17: 0x7fba4eb54cd9 - <unknown> 18: 0x7fba4f154c2f - <unknown> 19: 0x7fba4f1537db - <unknown> 20: 0x7fba4f13de51 - <unknown> 21: 0x7fba4f23a857 - <unknown> 22: 0x7fba4f158ff4 - <unknown> 23: 0x7fba4f13bf5f - <unknown> 24: 0x7fba4f13df85 - <unknown> 25: 0x7fba4f23a857 - <unknown> 26: 0x7fba4f13e25a - <unknown> 27: 0x7fba4f2210cf - <unknown> 28: 0x7fba4f239a40 - <unknown> 29: 0x7fba52d3346f - start_thread 30: 0x7fba5320b3d3 - clone 31: 0x0 - <unknown> free(): invalid pointer

And the single variant seen in the Snapshot case:

thread 'tokio-runtime-worker' panicked at 'state = Snapshot { is_running: true, is_notified: false, is_released: false, is_complete: false, is_canceled: false, is_join_interested: false, has_join_waker: false, is_final_ref: false }', /home/jsirois/.cache/pants/rust/cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.13/src/task/state.rs:227:9
stack backtrace:
   0:     0x7f4c8d2640a4 - <unknown>
   1:     0x7f4c8d289a9c - <unknown>
   2:     0x7f4c8d25d527 - <unknown>
   3:     0x7f4c8d266985 - <unknown>
   4:     0x7f4c8d266671 - <unknown>
   5:     0x7f4c8d266fca - <unknown>
   6:     0x7f4c8d266bb0 - <unknown>
   7:     0x7f4c8d266b2b - <unknown>
   8:     0x7f4c8d18f699 - <unknown>
   9:     0x7f4c8cb8a671 - <unknown>
  10:     0x7f4c8d189e07 - <unknown>
  11:     0x7f4c8d1887db - <unknown>
  12:     0x7f4c8d172e51 - <unknown>
  13:     0x7f4c8d26f857 - <unknown>
  14:     0x7f4c8d18dff4 - <unknown>
  15:     0x7f4c8d170f5f - <unknown>
  16:     0x7f4c8d172f85 - <unknown>
  17:     0x7f4c8d26f857 - <unknown>
  18:     0x7f4c8d17325a - <unknown>
  19:     0x7f4c8d2560cf - <unknown>
  20:     0x7f4c8d26ea40 - <unknown>
  21:     0x7f4c90d6846f - start_thread
  22:     0x7f4c912403d3 - clone
  23:                0x0 - <unknown>

This all looks pretty unhelpful though - is there something I can do to enable better backtrace info? I tried this naive thing and no different:

$ rm -rf ~/.cache/pants/bin/native-engine/
$ rm -rf ~/.cache/pants/lmdb_store/
$ git clean -fdx
$ MODE_FLAG=debug RUST_BACKTRACE=full ./v2 fmt src/python/pants/backend/python/rules: src/python/pants/backend/python/subsystems: