-
Notifications
You must be signed in to change notification settings - Fork 62
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Inference Failed Because of '500 Internal Server Error' #45
Comments
Hey @dawenxi-007 , can you print the distribution side logs if you see any. |
Here is what I saw:
It was asking for Wolfram API key. I didn't see this message if I run the flow from the llama agentic system git repo. |
Yeah we changed some things on how distributions manage keys ( its now in the config instead of using env variables ). |
Yes, I noticed that change. Now the ask for set up the keys is at the configuration stage. However, shouldn't the setting for the key should be optional? Like in this case, it only asks for Wolfram API key but not Brave Search API key. Also, there could be some inference scenario may not use the related tool APIs. |
I think it just fails as soon as it needs to use the Wolfram API key. See https://github.com/meta-llama/llama-stack/blob/main/llama_toolchain/agentic_system/meta_reference/agentic_system.py#L83 The next iteration of that loop (if you did provide the wolfram key but did not provide Brave Search key) will fail at the Brave Search step. |
After launching the distribution server by
"llama distribution start --name local-llama-8b --port 5000 --disable-ipv6 "
, running any inference example, for example"python examples/scripts/vacation.py localhost 5000 --disable-safety"
will give the following error:The text was updated successfully, but these errors were encountered: