多张卡推理Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:3!

[lys791227]

Your current environment

ile "/u02/liuys/MiniCPM-V/vllm/vllm/worker/model_runner.py", line 1185, in execute_model
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] hidden_or_intermediate_states = model_executable(
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] File "/u01/liuys/anaconda3/envs/minicpm-v-env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] File "/u01/liuys/anaconda3/envs/minicpm-v-env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] File "/u02/liuys/MiniCPM-V/vllm/vllm/model_executor/models/minicpmv.py", line 583, in forward
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] vlm_embeddings, vision_hidden_states = self.get_embedding(inputs)
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] File "/u02/liuys/MiniCPM-V/vllm/vllm/model_executor/models/minicpmv.py", line 541, in get_embedding
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] vision_hidden_states = self.get_vision_hidden_states(data)
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] File "/u02/liuys/MiniCPM-V/vllm/vllm/model_executor/models/minicpmv.py", line 519, in get_vision_hidden_states
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] vision_embedding = self.vpm(all_pixel_values.type(dtype), patch_attention_mask=patch_attn_mask).last_hidden_state
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] File "/u01/liuys/anaconda3/envs/minicpm-v-env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] File "/u01/liuys/anaconda3/envs/minicpm-v-env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] File "/u01/liuys/anaconda3/envs/minicpm-v-env/lib/python3.10/site-packages/transformers/models/idefics2/modeling_idefics2.py", line 715, in forward
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] hidden_states = self.embeddings(pixel_values=pixel_values, patch_attention_mask=patch_attention_mask)
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] File "/u01/liuys/anaconda3/envs/minicpm-v-env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] File "/u01/liuys/anaconda3/envs/minicpm-v-env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] File "/u01/liuys/anaconda3/envs/minicpm-v-env/lib/python3.10/site-packages/transformers/models/idefics2/modeling_idefics2.py", line 167, in forward
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] patch_embeds = self.patch_embedding(pixel_values)
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] File "/u01/liuys/anaconda3/envs/minicpm-v-env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] return self._call_impl(*args, **kwargs)
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] File "/u01/liuys/anaconda3/envs/minicpm-v-env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] return forward_call(*args, **kwargs)
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] File "/u01/liuys/anaconda3/envs/minicpm-v-env/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 460, in forward
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] return self._conv_forward(input, self.weight, self.bias)
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] File "/u01/liuys/anaconda3/envs/minicpm-v-env/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 456, in _conv_forward
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] return F.conv2d(input, weight, bias, self.stride,
(VllmWorkerProcess pid=2398311) ERROR 07-22 13:28:17 multiproc_worker_utils.py:226] RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:3! (when checking argument for argument weight in method wrapper_CUDA__cudnn_convolution)

How would you like to use vllm

I want to run inference of a [specific model](put link here). I don't know how to integrate it with vllm.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!