[Feature] Support minicpmv v2.6

[mickqian]

Motivation

Addressing #2461

Modifications

1. Add new model MiniCPMV and corresponding processor MiniCPMVImageProcessor
2. Register new chat and conversation templates

Checklist

• Format your code according to the Contributor Guide.
• Add unit tests as outlined in the Contributor Guide.
• Update documentation as needed, including docstrings or example tutorials.

[mickqian]

Questions

1. From my understanding of current implementation of sglang, it does not support two kind of attentions at the same time, thus the attention module in Resampler can't be replaced with RadixAttention. Is that correct?

[zhaochenyang20]

Questions

1. From my understanding on current implementation of sglang, it does not support two kind of attentions at the same time, thus the attention module in Resampler and Idefics2VisionAttention can't be replaced with RadixAttention. Is that correct?
2. Current implementation of Idefics2VisionAttention in this PR relies on vllm.attention.MultiheadAttention, which is introduced in version vllm==0.6.5. If that is not acceptable, I will rewrite the attention module

@Ying1123 @merrymercy

[zhaochenyang20]

@mickqian Looks nice! Are there anything left to do?

[mickqian]

@mickqian Looks nice! Are there anything left to do?看起来不错！还有什么要做的吗？

1. The number of frames processed from the video is hard-coded for the moment, which should be calculated. I added a TODO on it.
2. A CI Test failed because of this:

[2025-01-10 03:24:51 TP0] Multimodal prompt is too long after expanding multimodal tokens. After expanding len(req.origin_input_ids_unpadded)=29 => 308 >= 294.

from an assertion of checking the input_ids' length. The token limit (from my understanding) is calculated from the GPU:

 if len(req.origin_input_ids) >= self.max_req_input_len:
        logger.error(
            "Multimodal prompt is too long after expanding multimodal tokens. "
            f"After expanding {len(req.origin_input_ids_unpadded)=} => {len(req.origin_input_ids)} >= {self.max_req_input_len}. "
                )

3. Some mp4 file failed the decode process in decode_video_base64

[zhaochenyang20]

@hnyls2002 Could you help to review. Thanks!

[zhaochenyang20]

@mickqian Thanks. We will update the API key as soon as possible. Before that, could you fix the conflicts of qwen2_vl.py.

[yizhang2077]

LGTM cc @merrymercy

[merrymercy]

1. Can you add some unit tests like this to compare the logtis against HF implementation? https://github.com/sgl-project/sglang/pull/2365/files#diff-6e52783df34170e0b3d9aadbd7c338d9c16f0303c18846afc1d298c32d4a4eb2R1
2. Can you help update the docs here for VLM?

   sglang/docs/references/supported_models.md

   Line 50 in 83452db

   ## How to Support a New Model

[merrymercy]

Do not import anything from vllm.

• SamplingMetadata, SamplerOutput are not used.
• Use sglang.srt.distributed
• copy over small utility functions.

[zhaochenyang20]

@mickqian Also, remove Chinese comments in the PR. Thanks so much. Could we try not import any vllm dependency, rather rewrite it ourselves.

[zhaochenyang20]

Under the models files, we prefer not to import anything from vllm. We will remove them all later.

[mickqian]

Trying to address these in #2977