Does it support llava-video?

[zjukop]

No description provided.

[Yangsenqiao]

In my opinion, LLaVA-Video introduces a high-quality dataset, and the model is still based on the LLaVA architecture. Therefore, I think VisionZip can be applied to LLaVA-Video.

[zjukop]

I encountered an error, Could u help me?
File "lmms-eval-main/VisionZip/visionzip/llava_arch.py", line 43, in encode_images_visionzip_multi
image_features, keep_idx = self.get_model().get_vision_tower().forward(images)
File "/root/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1688, in getattr
raise AttributeError(f"'{type(self).name}' object has no attribute '{name}'")
AttributeError: 'ClipEncoderLayer' object has no attribute 'metric

[Yangsenqiao]

Hi 🤗,

The metric is calculated in this [line](

VisionZip/visionzip/clip_encoder.py

Line 45 in 641be94

metric = self.vision_tower.vision_model.encoder.layers[-2].metric

).

Please check if the model correctly executed this line.

If you encounter any further errors, feel free to ask me.

Best,
Senqiao