Error occupty when i restore two model

[xuliang5115]

Complex prediction is divided into two networks,''Re'' and ''Im'' which sharing the same structure.
I trained two networks separately and the training was successful.
And MIONetIn4CartesianProd is MIONet with four branch-net, i complete it for my project.
But i got a error when i restore two model at a run(restore separately is OK).

from solver import *
from main import Dataset, periodic


m, p = 1000, 1000
epochs = 1000000
_lr = 0.001
dataRe = Dataset(m, p, 1, Flag='Re').reshape2tuple()
dataIm = Dataset(m, p, 1, Flag='Im').reshape2tuple()
activation = "tanh"
net = MIONetIn4CartesianProd(
   [m, 200, 200],
   [m, 200, 200],
   [m, 200, 200],
   [m, 200, 200],
   [2, 200, 200],
   activation,
   "Glorot normal",)

net.apply_feature_transform(periodic)

# model.restore, restore successful.
try:
   modelRe = dde.Model(dataRe, net)
   modelRe.compile(
       "adam",
       lr=_lr,
       decay=("inverse time", 100, 0.01),
       # decay=("cosine", 1000, 0.5),
       metrics=["mse"],
   )
   modelRe.restore("save/ReModel.ckpt-{epoch}.ckpt".format(epoch=epochs), verbose=1)
except NameError:
   print(1)
# I got some error when i restore another model.
try:
   modelIm = dde.Model(dataIm, net)
   modelIm.compile(
       "adam",
       lr=_lr,
       decay=("inverse time", 100, 0.01),
       # decay=("cosine", 1000, 0.5),
       metrics=["mse"],
   )
   # model.restore （not work）
   modelIm.restore("save/ImModel.ckpt-{epoch}.ckpt".format(epoch=epochs), verbose=1)
except NameError:
   print(2)

There are parts of Error report:

Restoring model from save/ReModel.ckpt-1000000.ckpt ...
INFO:tensorflow:Restoring parameters from save/ReModel.ckpt-1000000.ckpt
Compiling model...
2022-06-21 11:25:11.428989: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 2151 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1650 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5
'compile' took 0.349466 s
Restoring model from save/ImModel.ckpt-1000000.ckpt ...
INFO:tensorflow:Restoring parameters from save/ImModel.ckpt-1000000.ckpt
2022-06-21 11:25:11.836096: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at save_restore_v2_ops.cc:222 : NOT_FOUND: Key Variable/Adam not found in checkpoint
Traceback (most recent call last):
  File "C:\Users\51158\anaconda3\envs\DeepxdeProject\lib\site-packages\tensorflow\python\client\session.py", line 1377, in _do_call
    return fn(*args)
  File "C:\Users\51158\anaconda3\envs\DeepxdeProject\lib\site-packages\tensorflow\python\client\session.py", line 1360, in _run_fn
    return self._call_tf_sessionrun(options, feed_dict, fetch_list,
  File "C:\Users\51158\anaconda3\envs\DeepxdeProject\lib\site-packages\tensorflow\python\client\session.py", line 1453, in _call_tf_sessionrun
    return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found.
  (0) NOT_FOUND: Key Variable/Adam not found in checkpoint
	 [[{{node save_1/RestoreV2}}]]
	 [[save_1/RestoreV2/_41]]
  (1) NOT_FOUND: Key Variable/Adam not found in checkpoint
	 [[{{node save_1/RestoreV2}}]]
0 successful operations.
0 derived errors ignored.
During handling of the above exception, another exception occurred:

Looking forward to your reply.

[forxltk]

#54 (comment)

[xuliang5115]

I define dde.Model(data, net) twice in the same code.
I will try put model1 and model2 in two files.
Thx.