Gltf-blender-io: Image compositor crash / temporary mesh invalidation

[emackey]

This is a strange crash, and I had some trouble creating a minimal reproduce case for it.

To reproduce it, I needed to:

• Merge two textures in such a way that they call @scurest's (awesome) image compositor.
• Turn on "Smooth Shading" and "Auto Normals" for the mesh, such that it will need normal splitting.
• Turn on "Apply Modifiers" such that the normals will get split.

Here's a Blender 2.81a project that will reproduce the problem, but only on newer (2.82) versions of this glTF addon.

crashy_cube.zip

Unfortunately, this crash now happens in the stable release of 2.82. It's new there, it didn't happen in the stable release of Blender 2.81a.

But, the problem can be reproduced in 2.81a, if one manually applies a newer version of this addon to it. That means the problem is localized to the addon, not a change in Blender itself. From there I was able to run git bisect to get an exact ID for which commit introduces the problem.

Git bisect says: 4b460489cdba12bb01954a2748cd7209e6549959 is the first bad commit.

The crash writes this to the console:

Traceback (most recent call last):
  File "D:\emackey\Git_Large\glTF-Blender-IO\addons\io_scene_gltf2\__init__.py", line 487, in execute
    return gltf2_blender_export.save(context, export_settings)
  File "D:\emackey\Git_Large\glTF-Blender-IO\addons\io_scene_gltf2\blender\exp\gltf2_blender_export.py", line 40, in save
    json, buffer = __export(export_settings)
  File "D:\emackey\Git_Large\glTF-Blender-IO\addons\io_scene_gltf2\blender\exp\gltf2_blender_export.py", line 53, in __export
    __gather_gltf(exporter, export_settings)
  File "D:\emackey\Git_Large\glTF-Blender-IO\addons\io_scene_gltf2\blender\exp\gltf2_blender_export.py", line 62, in __gather_gltf
    active_scene_idx, scenes, animations = gltf2_blender_gather.gather_gltf2(export_settings)
  File "D:\emackey\Git_Large\glTF-Blender-IO\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather.py", line 37, in gather_gltf2
    scenes.append(__gather_scene(blender_scene, export_settings))
  File "D:\emackey\Git_Large\glTF-Blender-IO\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_cache.py", line 65, in wrapper_cached
    result = func(*args)
  File "D:\emackey\Git_Large\glTF-Blender-IO\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather.py", line 56, in __gather_scene
    node = gltf2_blender_gather_nodes.gather_node(blender_object, blender_scene, export_settings)
  File "D:\emackey\Git_Large\glTF-Blender-IO\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_nodes.py", line 46, in gather_node
    node = __gather_node(blender_object, blender_scene, export_settings)
  File "D:\emackey\Git_Large\glTF-Blender-IO\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_cache.py", line 65, in wrapper_cached
    result = func(*args)
  File "D:\emackey\Git_Large\glTF-Blender-IO\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_nodes.py", line 64, in __gather_node
    mesh=__gather_mesh(blender_object, export_settings),
  File "D:\emackey\Git_Large\glTF-Blender-IO\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_nodes.py", line 337, in __gather_mesh
    export_settings)
  File "D:\emackey\Git_Large\glTF-Blender-IO\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_cache.py", line 65, in wrapper_cached
    result = func(*args)
  File "D:\emackey\Git_Large\glTF-Blender-IO\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_mesh.py", line 43, in gather_mesh
    weights=__gather_weights(blender_mesh, vertex_groups, modifiers, export_settings)
  File "D:\emackey\Git_Large\glTF-Blender-IO\addons\io_scene_gltf2\blender\exp\gltf2_blender_gather_mesh.py", line 138, in __gather_weights
    if not export_settings[MORPH] or not blender_mesh.shape_keys:
ReferenceError: StructRNA of type Mesh has been removed

[julienduroure]

Same error than #930
I pushed a fix/workaround yesterday

[julienduroure]

Cf https://github.com/KhronosGroup/glTF-Blender-IO/commit/f2f3b83a0bef5fd1640742f94d9469238c561a2b

[emackey]

@julienduroure Thanks! Confirmed fix.

[emackey]

I wonder if this means shape keys are broken on objects such as my reproduce case here... I don't have time to test right now though.

[emackey]

And the fact that this was introduced by the new image compositor, which has nothing that I know of to do with shape keys, seems very suspicious that something deeper is wrong...

[julienduroure]

When using Apply Modifiers, Blender can't export SK. In most cases, mesh.shape_keys return None, but in some cases, it crashes.
Yes, we need to investigate deeper. It may be a Blender API bug or a bug in compositor code.

[scurest]

Nice find.

Some background: a temporary mesh is created with to_mesh (Blender docs) when using Apply Modifiers.

• If this line is commented out, there's no error.

  https://github.com/KhronosGroup/glTF-Blender-IO/blob/606f05f7ee544a90b1ed2a8235d305c9467d5252/addons/io_scene_gltf2/blender/exp/gltf2_blender_image.py#L339

• If the gltf2_blender_gather_materials.gather_material call is delayed until after we destroy the to_mesh mesh with to_mesh_clear(), there's no error.
• Accessing blender_mesh.shape_keys works until just after the call to gltf2_blender_gather_materials.gather_material.
• StructRNA of type Mesh has been removed is what happens if you try to access the temporary mesh after its explicitly been destroyed by to_mesh_clear.

So my working theory consistent with all these is: bpy.ops.render.render somehow invalidates the temporary mesh created by to_mesh :/

I can think of two solutions:

1. Check if "Apply Modifiers" is turned on in __encode_unhappy_with_compositor and fallback to the slow numpy code

2. Delay calling gather_material until after we're done with the to_mesh mesh. (ie. store a tuple of arguments for gather material in primitive.material, then replace it with the actuall call later)

   This is weirder but let's us continue to use the fast compositor path.

[julienduroure]

Let's reopen this ticket to keep this subject alive.
I will ask blender devs if invalidation of tmp mesh after render is a known issue/feature and if there is a way to avoid it

[scurest]

More info:

This appears to require two ingredients: the bpy.ops.render.render AND this block

https://github.com/KhronosGroup/glTF-Blender-IO/blob/606f05f7ee544a90b1ed2a8235d305c9467d5252/addons/io_scene_gltf2/blender/exp/gltf2_blender_gather_nodes.py#L304-L306

It's not until after this block that bpy.op.render.render invalidates the mesh. Look

````diff

• # This block is OK!
• scn = bpy.data.scenes.new('temp scene')
• scn.use_nodes = True
• scn.node_tree.nodes.remove(scn.node_tree.nodes['Render Layers'])
• bpy.ops.render.render(scene=scn.name, write_still=True)
• print(blender_mesh.name)
  +
  if auto_smooth and not some_normals_modifier:
  blender_object.data.use_auto_smooth = True
  blender_object.modifiers.remove(edge_split)
  +
• print(blender_mesh.name) # <- this is still OK!
  +
• scn = bpy.data.scenes.new('temp scene')
• scn.use_nodes = True
• scn.node_tree.nodes.remove(scn.node_tree.nodes['Render Layers'])
• bpy.ops.render.render(scene=scn.name, write_still=True)
  +
• print(blender_mesh.name) # <- ERROR! StructRNA of type Mesh has been removed
  ````

[scurest]

Script to reproduce this without needing the whole addon (maybe useful for asking upstream).

Open @emackey's crashy_cube.blend, put this in the text editor and run it. It will crash.
If at least one of the if Trues is changed to if False, the crash goes away.

````py
import bpy

ob = bpy.data.objects['Cube']

edge_split = ob.modifiers.new('Temporary_Auto_Smooth', 'EDGE_SPLIT')
ob.data.use_auto_smooth = False

depsgraph = bpy.context.evaluated_depsgraph_get()
mesh_owner = ob.evaluated_get(depsgraph)
mesh = mesh_owner.to_mesh(preserve_all_data_layers=True, depsgraph=depsgraph)

if True:
ob.data.use_auto_smooth = True
ob.modifiers.remove(edge_split)

if True:
scn = bpy.data.scenes.new('temp scene')
scn.use_nodes = True
scn.node_tree.nodes.remove(scn.node_tree.nodes['Render Layers'])
bpy.ops.render.render(scene=scn.name, write_still=True)

print(mesh.name)
````

[julienduroure]

Thanks a lot @scurest , it was my plan to get this minimal script to reproduce the bug, but you did it first! Thanks :)

[julienduroure]

Bug report on blender tracker: https://developer.blender.org/T74092

[julienduroure]

Seems rendering invalidate the temporary mesh.
The proposed solution is to factorize the depsgraph evaluation, and if render is used, call the evaluation again.
Checking with the devs and waiting the final words, but seems this is only a reference, so no impact on performance and on memory usage.

[scurest]

That doesn't explain why it doesn't happen if the first if True block doesn't run though.

Btw, for a work-around, can we just switch the order of __gather_weights and __gather_primitives in the gather_mesh function so that __gather_weights comes first?

[julienduroure]

That doesn't explain why it doesn't happen if the first if True block doesn't run though.

This can be an exaplanation ( it's my own supposition):
If you don't execute the first True block, you don't change the state of the mesh data that you reference in your temporary mesh variable, and the reference is still valid

[scurest]

That's plausible. Some kind of dirty flag gets set when the state is changed that gets checked when rendering?

[julienduroure]

Note that a workaround is now comitted (6a6fa8402a698937c02439476b962093532f872e)
Keeping this ticket open until final response on blender ticket https://developer.blender.org/T74092