Spleeter: [Bug] Train on the basis of github releases model，Error：Key batch_normalization/beta/Adam not found in checkpoint

[baiyu8534]

Description

Hi, All

I want restore github releases model，continue training on this basis。
I use ‘spleeter train’ command train model

But this error occurred：tensorflow.python.framework.errors_impl.NotFoundError: Key batch_normalization/beta/Adam not found in checkpoint [[{{node save/RestoreV2}}]]

When not using github releases model，not copy github releases model to ’ "model_dir": "musdb_model" ’，training is normal

Step to reproduce

1.I copy github releases model in to ’ "model_dir": "musdb_model" ’
2.Run ‘python -m spleeter train -p /Users/baiyu/train-all-data/musdb_config.json -d /Users/baiyu/train-all-data’

Output

% python -m spleeter train -p /Users/baiyu/train-all-data/musdb_config_local.json -d /Users/baiyu/train-all-data
INFO:spleeter:Start model training
WARNING:tensorflow:From /Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/spleeter/commands/train.py:117: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

INFO:tensorflow:Not using Distribute Coordinator.
INFO:tensorflow:Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 300 or save_checkpoints_secs None.
WARNING:tensorflow:From /Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From /Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/data/util/random_seed.py:58: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.string_join is deprecated. Please use tf.strings.join instead.

INFO:tensorflow:Calling model_fn.
INFO:tensorflow:Apply unet for vocals_spectrogram
WARNING:tensorflow:From /Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
INFO:tensorflow:Apply unet for drums_spectrogram
INFO:tensorflow:Apply unet for bass_spectrogram
INFO:tensorflow:Apply unet for other_spectrogram
INFO:tensorflow:Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from musdb_model/model
Traceback (most recent call last):
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: Key batch_normalization/beta/Adam not found in checkpoint
     [[{{node save/RestoreV2}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1290, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key batch_normalization/beta/Adam not found in checkpoint
     [[node save/RestoreV2 (defined at /site-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'save/RestoreV2':
  File "/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/site-packages/spleeter/__main__.py", line 58, in <module>
    entrypoint()
  File "/site-packages/spleeter/__main__.py", line 54, in entrypoint
    main(sys.argv)
  File "/site-packages/spleeter/__main__.py", line 46, in main
    entrypoint(arguments, params)
  File "/site-packages/spleeter/commands/train.py", line 121, in entrypoint
    evaluation_spec)
  File "/site-packages/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate
    return executor.run()
  File "/site-packages/tensorflow_estimator/python/estimator/training.py", line 613, in run
    return self.run_local()
  File "/site-packages/tensorflow_estimator/python/estimator/training.py", line 714, in run_local
    saving_listeners=saving_listeners)
  File "/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default
    saving_listeners)
  File "/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1490, in _train_with_estimator_spec
    log_step_count_steps=log_step_count_steps) as mon_sess:
  File "/site-packages/tensorflow_core/python/training/monitored_session.py", line 584, in MonitoredTrainingSession
    stop_grace_period_secs=stop_grace_period_secs)
  File "/site-packages/tensorflow_core/python/training/monitored_session.py", line 1014, in __init__
    stop_grace_period_secs=stop_grace_period_secs)
  File "/site-packages/tensorflow_core/python/training/monitored_session.py", line 725, in __init__
    self._sess = _RecoverableSession(self._coordinated_creator)
  File "/site-packages/tensorflow_core/python/training/monitored_session.py", line 1207, in __init__
    _WrappedSession.__init__(self, self._create_session())
  File "/site-packages/tensorflow_core/python/training/monitored_session.py", line 1212, in _create_session
    return self._sess_creator.create_session()
  File "/site-packages/tensorflow_core/python/training/monitored_session.py", line 878, in create_session
    self.tf_sess = self._session_creator.create_session()
  File "/site-packages/tensorflow_core/python/training/monitored_session.py", line 638, in create_session
    self._scaffold.finalize()
  File "/site-packages/tensorflow_core/python/training/monitored_session.py", line 237, in finalize
    self._saver.build()
  File "/site-packages/tensorflow_core/python/training/saver.py", line 840, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/site-packages/tensorflow_core/python/training/saver.py", line 878, in _build
    build_restore=build_restore)
  File "/site-packages/tensorflow_core/python/training/saver.py", line 502, in _build_internal
    restore_sequentially, reshape)
  File "/site-packages/tensorflow_core/python/training/saver.py", line 381, in _AddShardedRestoreOps
    name="restore_shard"))
  File "/site-packages/tensorflow_core/python/training/saver.py", line 328, in _AddRestoreOps
    restore_sequentially)
  File "/site-packages/tensorflow_core/python/training/saver.py", line 575, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/site-packages/tensorflow_core/python/ops/gen_io_ops.py", line 1696, in restore_v2
    name=name)
  File "/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1300, in restore
    names_to_keys = object_graph_key_mapping(save_path)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1618, in object_graph_key_mapping
    object_graph_string = reader.get_tensor(trackable.OBJECT_GRAPH_PROTO_KEY)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py", line 915, in get_tensor
    return CheckpointReader_GetTensor(self, compat.as_bytes(tensor_str))
tensorflow.python.framework.errors_impl.NotFoundError: Key _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/spleeter/__main__.py", line 58, in <module>
    entrypoint()
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/spleeter/__main__.py", line 54, in entrypoint
    main(sys.argv)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/spleeter/__main__.py", line 46, in main
    entrypoint(arguments, params)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/spleeter/commands/train.py", line 121, in entrypoint
    evaluation_spec)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate
    return executor.run()
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 613, in run
    return self.run_local()
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/training.py", line 714, in run_local
    saving_listeners=saving_listeners)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default
    saving_listeners)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1490, in _train_with_estimator_spec
    log_step_count_steps=log_step_count_steps) as mon_sess:
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 584, in MonitoredTrainingSession
    stop_grace_period_secs=stop_grace_period_secs)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1014, in __init__
    stop_grace_period_secs=stop_grace_period_secs)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 725, in __init__
    self._sess = _RecoverableSession(self._coordinated_creator)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1207, in __init__
    _WrappedSession.__init__(self, self._create_session())
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 1212, in _create_session
    return self._sess_creator.create_session()
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 878, in create_session
    self.tf_sess = self._session_creator.create_session()
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/training/monitored_session.py", line 647, in create_session
    init_fn=self._scaffold.init_fn)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/training/session_manager.py", line 290, in prepare_session
    config=config)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/training/session_manager.py", line 220, in _restore_checkpoint
    saver.restore(sess, ckpt.model_checkpoint_path)
  File "/Users/baiyu/opt/anaconda3/envs/spleeter/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1306, in restore
    err, "a Variable name or other graph key that is missing")
tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key batch_normalization/beta/Adam not found in checkpoint
     [[node save/RestoreV2 (defined at /site-packages/tensorflow_core/python/framework/ops.py:1748) ]]

Original stack trace for 'save/RestoreV2':
  File "/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/site-packages/spleeter/__main__.py", line 58, in <module>
    entrypoint()
  File "/site-packages/spleeter/__main__.py", line 54, in entrypoint
    main(sys.argv)
  File "/site-packages/spleeter/__main__.py", line 46, in main
    entrypoint(arguments, params)
  File "/site-packages/spleeter/commands/train.py", line 121, in entrypoint
    evaluation_spec)
  File "/site-packages/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate
    return executor.run()
  File "/site-packages/tensorflow_estimator/python/estimator/training.py", line 613, in run
    return self.run_local()
  File "/site-packages/tensorflow_estimator/python/estimator/training.py", line 714, in run_local
    saving_listeners=saving_listeners)
  File "/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1195, in _train_model_default
    saving_listeners)
  File "/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1490, in _train_with_estimator_spec
    log_step_count_steps=log_step_count_steps) as mon_sess:
  File "/site-packages/tensorflow_core/python/training/monitored_session.py", line 584, in MonitoredTrainingSession
    stop_grace_period_secs=stop_grace_period_secs)
  File "/site-packages/tensorflow_core/python/training/monitored_session.py", line 1014, in __init__
    stop_grace_period_secs=stop_grace_period_secs)
  File "/site-packages/tensorflow_core/python/training/monitored_session.py", line 725, in __init__
    self._sess = _RecoverableSession(self._coordinated_creator)
  File "/site-packages/tensorflow_core/python/training/monitored_session.py", line 1207, in __init__
    _WrappedSession.__init__(self, self._create_session())
  File "/site-packages/tensorflow_core/python/training/monitored_session.py", line 1212, in _create_session
    return self._sess_creator.create_session()
  File "/site-packages/tensorflow_core/python/training/monitored_session.py", line 878, in create_session
    self.tf_sess = self._session_creator.create_session()
  File "/site-packages/tensorflow_core/python/training/monitored_session.py", line 638, in create_session
    self._scaffold.finalize()
  File "/site-packages/tensorflow_core/python/training/monitored_session.py", line 237, in finalize
    self._saver.build()
  File "/site-packages/tensorflow_core/python/training/saver.py", line 840, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/site-packages/tensorflow_core/python/training/saver.py", line 878, in _build
    build_restore=build_restore)
  File "/site-packages/tensorflow_core/python/training/saver.py", line 502, in _build_internal
    restore_sequentially, reshape)
  File "/site-packages/tensorflow_core/python/training/saver.py", line 381, in _AddShardedRestoreOps
    name="restore_shard"))
  File "/site-packages/tensorflow_core/python/training/saver.py", line 328, in _AddRestoreOps
    restore_sequentially)
  File "/site-packages/tensorflow_core/python/training/saver.py", line 575, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/site-packages/tensorflow_core/python/ops/gen_io_ops.py", line 1696, in restore_v2
    name=name)
  File "/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

Environment

| | |
| ----------------- | ------------------------------- |
| OS | MacOS 10.15.1 |
| Installation type | pip |
| RAM available | 8G |
| Hardware spec | Intel Core i5 2.8 GHz |

Additional context

[baiyu8534]

I fixed. Use GradientDescentOptimizer

    def _build_optimizer(self):
        """ Builds an optimizer instance from internal parameter values.

        Default to AdamOptimizer if not specified.

        :returns: Optimizer instance from internal configuration.
        """
        name = self._params.get('optimizer')
        if name == self.ADADELTA:
            return tf.compat.v1.train.AdadeltaOptimizer()
        rate = self._params['learning_rate']
        if name == self.SGD:
            return tf.compat.v1.train.GradientDescentOptimizer(rate)
        # return tf.compat.v1.train.AdamOptimizer(rate)
        return tf.compat.v1.train.GradientDescentOptimizer(rate)