Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: Ensure model provided in vLLM inference #820

Merged
merged 23 commits into from
Jan 14, 2025
Merged

fix: Ensure model provided in vLLM inference #820

merged 23 commits into from
Jan 14, 2025

Conversation

ishaansehgal99
Copy link
Collaborator

@ishaansehgal99 ishaansehgal99 commented Jan 12, 2025 •
edited
Loading

Reason for Change:
Update to include model name in requests made to VLLM (has a /v1/models endpoint)

@zhuangqh
Copy link
Collaborator

zhuangqh commented Jan 12, 2025 •
edited
Loading

remember to update api spec json and docs here https://github.com/kaito-project/kaito/blob/main/docs/inference/README.md#transformers-inference-api

@ishaansehgal99 ishaansehgal99 changed the title fix: Standardize inference with vllm fix: Ensure model_name provided in vLLM inference Jan 14, 2025
@ishaansehgal99 ishaansehgal99 changed the title fix: Ensure model_name provided in vLLM inference fix: Ensure model provided in vLLM inference Jan 14, 2025
bangqipropel
bangqipropel reviewed Jan 14, 2025
View reviewed changes
presets/ragengine/inference/inference.py
**kwargs # Pass all kwargs directly; kwargs may include model, temperature, max_tokens, etc.
)
return llm.complete(prompt)
return OpenAI(api_key=LLM_ACCESS_SECRET, **kwargs).complete(prompt)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suppose this part do something similar to the vllm part but if you want to merge it now for efficiency, it is good

@ishaansehgal99 ishaansehgal99 merged commit c0f21fb into main Jan 14, 2025
6 of 7 checks passed
@ishaansehgal99 ishaansehgal99 deleted the rag-standardize-api branch January 14, 2025 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bangqipropel bangqipropel bangqipropel approved these changes

@zhuangqh zhuangqh zhuangqh approved these changes

@Fei-Guo Fei-Guo Awaiting requested review from Fei-Guo Fei-Guo is a code owner

@helayoty helayoty Awaiting requested review from helayoty helayoty is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ishaansehgal99 @zhuangqh @bangqipropel