[BUG] 用anything-llm链接openai api格式的本地API部署出现问题

[afezeriaWrnbbmm]

是否已有关于该错误的issue或讨论？ | Is there an existing issue / discussion for this?

• 我已经搜索过已有的issues和讨论 | I have searched the existing issues / discussions

该问题是否在FAQ中有解答？ | Is there an existing answer for this in FAQ?

• 我已经搜索过FAQ | I have searched FAQ

当前行为 | Current Behavior

使用提供的python openai_api.py命令本地部署
直接用API进行聊天是没有问题的
但是通过anything-llm聊天会报错
INFO: 172.30.0.2:55212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/sse_starlette/sse.py", line 269, in call
await wrap(partial(self.listen_for_disconnect, receive))
File "/usr/local/lib/python3.10/dist-packages/sse_starlette/sse.py", line 258, in wrap
await func()
File "/usr/local/lib/python3.10/dist-packages/sse_starlette/sse.py", line 215, in listen_for_disconnect
message = await receive()
File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 538, in receive
await self.message_event.wait()
File "/usr/lib/python3.10/asyncio/locks.py", line 214, in wait
await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7fcd84785690

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 84, in call
return await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 116, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 186, in call
raise exc
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 83, in call
await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 55, in wrapped_app
raise exc
File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 44, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 746, in call
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 75, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 55, in wrapped_app
raise exc
File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 44, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 73, in app
await response(scope, receive, send)
File "/usr/local/lib/python3.10/dist-packages/sse_starlette/sse.py", line 255, in call
async with anyio.create_task_group() as task_group:
File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 664, in aexit
raise BaseExceptionGroup(
exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)

使用的模型是72B-chat
ubuntu22.04
内存188G
显卡是2080ti22G*8

期望行为 | Expected Behavior

No response

复现方法 | Steps To Reproduce

No response

运行环境 | Environment

- OS:
- Python:
- Transformers:
- PyTorch:
- CUDA (`python -c 'import torch; print(torch.version.cuda)'`):

备注 | Anything else?

No response

[jklj077]

是否有更多的错误信息呢，这个错误不完整，看不到实际的error。

另外anything-llm使用的是openai的最新API接口，本repo中的openai_api.py生成的API与最新OpenAI的API接口定义并不兼容。

• 建议尝试使用第三方的OpenAI API工具，比如FastChat+vLLM
• 或者尝试这个开发者贡献的这个PR：1. support openai_api.py to the latest openai sdk(>=1.0.0); 2. add st… #814

[afezeriaWrnbbmm]

但是我按照readme里的方式用FastChat启动后 直接通过API进行聊天会报错
2024-02-19 09:30:25 | INFO | stdout | INFO: 192.168.14.4:44354 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
2024-02-19 09:30:25 | ERROR | stderr | ERROR: Exception in ASGI application
2024-02-19 09:30:25 | ERROR | stderr | Traceback (most recent call last):
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
2024-02-19 09:30:25 | ERROR | stderr | result = await app( # type: ignore[func-returns-value]
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/uvicorn/middleware/proxy_headers.py", line 84, in call
2024-02-19 09:30:25 | ERROR | stderr | return await self.app(scope, receive, send)
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/fastapi/applications.py", line 1054, in call
2024-02-19 09:30:25 | ERROR | stderr | await super().call(scope, receive, send)
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/starlette/applications.py", line 116, in call
2024-02-19 09:30:25 | ERROR | stderr | await self.middleware_stack(scope, receive, send)
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 186, in call
2024-02-19 09:30:25 | ERROR | stderr | raise exc
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/errors.py", line 164, in call
2024-02-19 09:30:25 | ERROR | stderr | await self.app(scope, receive, _send)
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/cors.py", line 83, in call
2024-02-19 09:30:25 | ERROR | stderr | await self.app(scope, receive, send)
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/starlette/middleware/exceptions.py", line 62, in call
2024-02-19 09:30:25 | ERROR | stderr | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 55, in wrapped_app
2024-02-19 09:30:25 | ERROR | stderr | raise exc
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 44, in wrapped_app
2024-02-19 09:30:25 | ERROR | stderr | await app(scope, receive, sender)
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 746, in call
2024-02-19 09:30:25 | ERROR | stderr | await route.handle(scope, receive, send)
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 288, in handle
2024-02-19 09:30:25 | ERROR | stderr | await self.app(scope, receive, send)
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 75, in app
2024-02-19 09:30:25 | ERROR | stderr | await wrap_app_handling_exceptions(app, request)(scope, receive, send)
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 55, in wrapped_app
2024-02-19 09:30:25 | ERROR | stderr | raise exc
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/starlette/_exception_handler.py", line 44, in wrapped_app
2024-02-19 09:30:25 | ERROR | stderr | await app(scope, receive, sender)
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/starlette/routing.py", line 70, in app
2024-02-19 09:30:25 | ERROR | stderr | response = await func(request)
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 299, in app
2024-02-19 09:30:25 | ERROR | stderr | raise e
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 294, in app
2024-02-19 09:30:25 | ERROR | stderr | raw_response = await run_endpoint_function(
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/fastapi/routing.py", line 191, in run_endpoint_function
2024-02-19 09:30:25 | ERROR | stderr | return await dependant.call(**values)
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/fastchat/serve/openai_api_server.py", line 438, in create_chat_completion
2024-02-19 09:30:25 | ERROR | stderr | max_new_tokens, error_check_ret = await check_length(
2024-02-19 09:30:25 | ERROR | stderr | File "/usr/local/lib/python3.10/dist-packages/fastchat/serve/openai_api_server.py", line 172, in check_length
2024-02-19 09:30:25 | ERROR | stderr | length = min(max_tokens, context_len - token_num)
2024-02-19 09:30:25 | ERROR | stderr | TypeError: unsupported operand type(s) for -: 'int' and 'str'

如果通过anything-llm的话会2024-02-19 09:30:11 | INFO | stdout | INFO: 192.168.14.4:31904 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request
然后没有任何其他报错信息
anything-llm那边显示
Request failed with status code 400

[jklj077]

这个错误是FastChat产生的, 具体是VLLMWorker.count_token返回了的结果中count字段类型错误，并不符合预期。建议寻求FastChat的帮助。