[BUG] OpenBMB/llama.cpp build error and segmentation fault on Mac M3

[hkr04]

是否已有关于该错误的issue或讨论？ | Is there an existing issue / discussion for this?

• 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答？ | Is there an existing answer for this in FAQ?

• 我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior

1. 若不使用-DLLAMA_METAL=OFF选项，会出现后端编译问题：

[  0%] Compiling Metal kernels
xcrun: error: unable to find utility "metal", not a developer tool or in PATH

2. 开启-DLLAMA_METAL=OFF选项绕开 metal 后端编译失败后，显示库找不到

fatal error: 'libavcodec/avcodec.h' file not found
    #include <libavcodec/avcodec.h>
             ^~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make[2]: *** [examples/llava/CMakeFiles/llama-minicpmv-cli.dir/minicpmv-cli.cpp.o] Error 1
make[1]: *** [examples/llava/CMakeFiles/llama-minicpmv-cli.dir/all] Error 2

3. 手动添加 FFmpeg 依赖后，在 clip_model_load 中出现段错误：

./build/bin/llama-minicpmv-cli -m Model-7.6B-Q4_K_M.gguf --mmproj mmproj-model-f16.gguf --image image.png --temp 0.1 -p "描述这张图片"
Log start
clip_model_load: description:  image encoder for MiniCPM-V
clip_model_load: GGUF version: 3
clip_model_load: alignment:    32
clip_model_load: n_tensors:    455
clip_model_load: n_kv:         19
clip_model_load: ftype:        f16

clip_model_load: loaded meta data with 19 key-value pairs and 455 tensors from /Users/hyde/Desktop/workspace/agent.cpp/3rdparty/llama.cpp/models/mmproj-model-f16.gguf
clip_model_load: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
clip_model_load: - kv   0:                       general.architecture str              = clip
clip_model_load: - kv   1:                      clip.has_text_encoder bool             = false
clip_model_load: - kv   2:                    clip.has_vision_encoder bool             = true
clip_model_load: - kv   3:                clip.has_minicpmv_projector bool             = true
clip_model_load: - kv   4:                          general.file_type u32              = 1
clip_model_load: - kv   5:                        general.description str              = image encoder for MiniCPM-V
clip_model_load: - kv   6:                        clip.projector_type str              = resampler
clip_model_load: - kv   7:                      clip.minicpmv_version i32              = 4
clip_model_load: - kv   8:                     clip.vision.image_size u32              = 448
clip_model_load: - kv   9:                     clip.vision.patch_size u32              = 14
clip_model_load: - kv  10:               clip.vision.embedding_length u32              = 1152
clip_model_load: - kv  11:            clip.vision.feed_forward_length u32              = 4304
clip_model_load: - kv  12:                 clip.vision.projection_dim u32              = 0
clip_model_load: - kv  13:           clip.vision.attention.head_count u32              = 16
clip_model_load: - kv  14:   clip.vision.attention.layer_norm_epsilon f32              = 0.000001
clip_model_load: - kv  15:                    clip.vision.block_count u32              = 27
clip_model_load: - kv  16:                     clip.vision.image_mean arr[f32,3]       = [0.500000, 0.500000, 0.500000]
clip_model_load: - kv  17:                      clip.vision.image_std arr[f32,3]       = [0.500000, 0.500000, 0.500000]
clip_model_load: - kv  18:                              clip.use_gelu bool             = true
clip_model_load: - type  f32:  285 tensors
clip_model_load: - type  f16:  170 tensors
clip_model_load: CLIP using CPU backend
clip_model_load: text_encoder:   0
clip_model_load: vision_encoder: 1
clip_model_load: llava_projector:  0
clip_model_load: minicpmv_projector:  1
clip_model_load: model size:     996.02 MB
clip_model_load: metadata size:  0.16 MB
clip_model_load: params backend buffer size =  996.02 MB (455 tensors)
key clip.vision.image_grid_pinpoints not found in file
key clip.vision.mm_patch_merge_type not found in file
key clip.vision.image_crop_resolution not found in file
clip_image_build_graph: 448 448 # segmentation fault after here
zsh: segmentation fault  ./build/bin/llama-minicpmv-cli -m  --mmproj  --image  --temp 0.1 -p 

比较MiniCPM-V-2.6可以发现是在 load 的过程中发生段错误：

key clip.vision.image_grid_pinpoints not found in file
key clip.vision.mm_patch_merge_type not found in file
key clip.vision.image_crop_resolution not found in file
clip_image_build_graph: 448 448 # segmentation fault after here (MiniCPM-o)
clip_model_load: compute allocated memory: 102.80 MB
uhd_slice_image: multiple 3
uhd_slice_image: image_size: 959 550; source_image size: 588 336
uhd_slice_image: image_size: 959 550; best_grid: 2 1
uhd_slice_image: refine_image_size: 840 476; refine_size: 840 476

换成MiniCPM-V-2.6用的mmproj-model-f16.gguf就可以正常推理了。另，llama.cpp中a813bad的minicpmv-cli使用这个视觉模型加上Minicpmv-o可以正常推理。

期望行为 | Expected Behavior

1. 正确在 Mac metal 后端上编译
2. 添加配置文件或相关说明正确处理 FFmpeg 依赖：

extern "C" {
    #include <libavcodec/avcodec.h>
    #include <libavformat/avformat.h>
    #include <libavutil/imgutils.h>
    #include <libswscale/swscale.h>
}

在本机通过添加list(APPEND CMAKE_PREFIX_PATH "/path/to/ffmpeg")及

find_package(PkgConfig REQUIRED)
pkg_check_modules(AVCODEC REQUIRED IMPORTED_TARGET libavcodec)
pkg_check_modules(AVFORMAT REQUIRED IMPORTED_TARGET libavformat)
pkg_check_modules(AVUTIL REQUIRED IMPORTED_TARGET libavutil)
pkg_check_modules(SWSCALE REQUIRED IMPORTED_TARGET libswscale)

target_link_libraries(${TARGET} PRIVATE PkgConfig::AVCODEC PkgConfig::AVFORMAT PkgConfig::AVUTIL PkgConfig::SWSCALE)

后可解决该依赖问题
3. 查看mmproj-model-f16.gguf文件是否有损坏问题（可能是我操作不当损坏，待确认，似乎在其他 issue 中也有人提出）

复现方法 | Steps To Reproduce

No response

运行环境 | Environment

- OS: MacOS 14.6 (23G80)
- CMake: 3.31.0
- FFmpeg: 7.1 (built with Apple clang)

备注 | Anything else?

No response

[hkr04]

Hello, I have never used LM Studio. If it is based on llama.cpp and can use gguf to inference, then my changes have not been merged into the official yet, and you may need to wait until the merge.

I am using the gguf version downloaded, and LM Studio is built-in with llama.cpp. The visual model of minicpm-v-2.6 is normal, and the language model of minicpm-o-2.6 is also normal. Only the visual model of minicpm-o-2.6 has an issue.

Originally posted by @AmintaCCCP in #717

[tc-mb]

你好，minicpm-o2.6还没有合并到llama.cpp官方分支中，请不要混用minicpm-v2.6和minicpm-o2.6文件。

[hkr04]

你好，minicpm-o2.6还没有合并到llama.cpp官方分支中，请不要混用minicpm-v2.6和minicpm-o2.6文件。

感谢百忙之中的回复！但需要注意，这里使用的 llama.cpp 为 minicpmv-main 的分支（0ae6bdc），官方分支仅为视觉塔有效性的验证。

[tc-mb]

嗯嗯，我也还没把minicpm-o2.6合并到minicpmv-main分支，我会在这几天尽快整合一下代码。请耐心等待一下。

[Doraemonwei]

我在树莓派5上遇到了相同的问题，将MiniCPM-o-2.6中的mmproj-model-f16.gguf模型替换为MiniCPM-V-2.6中的同名视觉模型即可正常运行