generation usage should accept a list of dict for async generation #896
dove-young
started this conversation in
Ideas
Replies: 1 comment
-
|
Hi @dove-young thanks a lot for the pointer. This is indeed a bit of a mismatch. Could you create a generation for each of the prompts in the list? |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
So far
usageingenerationis a dictionary. There is a problem with it.Some LLM provide
asyncfeature in theirgenerationfunction. It accept a list of prompts, and process them in a batch. Then it gives a list of generated responses.In this case there are a list of prompts has been processed, and a list of generation and token usage count are returned. And there is a single generate/completion call. So there would be a single
generationobject in thetrace.But now it is not possible to record these list of usage data into a
usagerecord in thegenerationobjectBeta Was this translation helpful? Give feedback.
All reactions