[WIP] support for more vlms #390
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
n1ck-guo
commented
Dec 19, 2024
•
n1ck-guo
commented
- support for more vlms: solvlm, aria(wip), llama-3.2v-cot, deepseek-vl2
- vl 70b+ on single card
- new processor for all hf-model
- modify get_multimodal_block_names (find language/vision/...)
Signed-off-by: n1ck-guo <[email protected]>
n1ck-guo
requested review from
WeiweiZhang1 and
wenhuach21
and removed request for
WeiweiZhang1
wenhuach21
reviewed
Dec 24, 2024
| @@ -160,6 +160,9 @@ def __init__( | |||
| self.template, model=model, tokenizer=tokenizer, processor=processor, image_processor=image_processor) | |||
| dataset = self.template.default_dataset if dataset is None else dataset | |||
|
|
|||
| if model.config.model_type == "deepseek_vl_v2": | |||
There was a problem hiding this comment.
the setting here is a little tricky. Could the quantizing-non-text-module still be supported?
Signed-off-by: n1ck-guo <[email protected]>
Signed-off-by: n1ck-guo <[email protected]>
|
any chance to use auto rounder with Aria-UI or Aria? |
|
|
thanks for the answer. Surely Qwen2VLForConditionalGeneration is not the class to use for Aria-UI-base? Any example or saved model would be great. Thanks |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.