[Usage] We cannot use model(**input) outside the trainer.train()

[MarkDeng1]

Describe the issue

Issue:
I have original model: model
then, i trained it within the trainier.train.

I use deepcopy to copy model called ref_model. then I use this model forward pass by ref_model(**input) within trainer.train .

However, we can use model(**input) within trainer.train. we cannot use ref_model(**input).

Why?

Command:

ref_model(**input)

Log:
Traceback (most recent call last):
File "/fred/oz337/zdeng/prompt_stealing_ours/LLaVA/llava/train/train_mem.py", line 4, in
train(attn_implementation="flash_attention_2")
File "/fred/oz337/zdeng/prompt_stealing_ours/LLaVA/llava/train/train.py", line 1101, in train
trainer.train()
File "/fred/oz337/zdeng/.conda/envs/llava/lib/python3.10/site-packages/transformers/trainer.py", line 1539, in train
return inner_training_loop(
File "/fred/oz337/zdeng/.conda/envs/llava/lib/python3.10/site-packages/transformers/trainer.py", line 1869, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs)
File "/fred/oz337/zdeng/prompt_stealing_ours/LLaVA/llava/train/llava_trainer.py", line 397, in training_step
loss = super().training_step(model, inputs)
File "/fred/oz337/zdeng/.conda/envs/llava/lib/python3.10/site-packages/transformers/trainer.py", line 2772, in training_step
loss = self.compute_loss(model, inputs)
File "/fred/oz337/zdeng/prompt_stealing_ours/LLaVA/llava/train/llava_trainer.py", line 422, in compute_loss
loss = self.compute_loss_func(batch=inputs,
File "/fred/oz337/zdeng/prompt_stealing_ours/LLaVA/llava/train/llava_trainer.py", line 171, in compute_loss_func
ref0_output = ref0_model(**new_batch)
File "/fred/oz337/zdeng/.conda/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/fred/oz337/zdeng/.conda/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/fred/oz337/zdeng/prompt_stealing_ours/LLaVA/llava/model/language_model/llava_llama.py", line 106, in forward
return super().forward(
File "/fred/oz337/zdeng/.conda/envs/llava/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1183, in forward
outputs = self.model(
File "/fred/oz337/zdeng/.conda/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/fred/oz337/zdeng/.conda/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/fred/oz337/zdeng/.conda/envs/llava/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1070, in forward
layer_outputs = decoder_layer(
File "/fred/oz337/zdeng/.conda/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/fred/oz337/zdeng/.conda/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/fred/oz337/zdeng/.conda/envs/llava/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 795, in forward
hidden_states = self.input_layernorm(hidden_states)
File "/fred/oz337/zdeng/.conda/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/fred/oz337/zdeng/.conda/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/fred/oz337/zdeng/.conda/envs/llava/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 117, in forward
return self.weight * hidden_states.to(input_dtype)
RuntimeError: The size of tensor a (0) must match the size of tensor b (4096) at non-singleton dimension 2

Screenshots:
You may attach screenshots if it better explains the issue.