[Fix] Address remain issues of supporting MiniCPMV

[mickqian]

Motivation

Address remaining issues of #2785

Modifications

1. Update document of implementing a new vision-llm
2. Add some test for comparing logits output of SGLang and HF
3. Code cleanup

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• [] Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling.

[zhaochenyang20]

Great work! Once ready, please ask us to review.

[merrymercy]

also remove these vllm dependency

sglang/python/sglang/srt/layers/attention/vision.py

Lines 8 to 9 in 2584f6d

	from vllm.distributed import parallel_state
	from vllm.distributed import utils as dist_utils

[mickqian]

context_attention_fwd generates different result with hf implementation: SiglipAttention, probably because:
1.SiglipAttention performs softmax in float32:

attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)

2. SiglipAttention performs full-sequence + mask, whereas context_attention_fwd skips padding tokens, leaving 0s in the attention weights.

[yizhang2077]

Does qwen2vl model have the same question? If so, I think maybe we can set qwen2vl use_context_forward to False or even remove use_context_forward branch.

[zhaochenyang20]

Cool. I will ask @yizhang2077 to help~

[yizhang2077]

Cool! I think maybe this test can be more general so that it can also be used by qwen2vl or more VLM?

[yizhang2077]

Why do we need add args here?