feat: Customizable Annotation View for Complex Traces #7593
Replies: 2 comments 5 replies
-
|
Thanks for sharing! We are considering adding the option of custom rendering frames that get the full trace object and can map it to html. let's see how many people are interested in this here. |
Beta Was this translation helpful? Give feedback.
-
|
Looking forward to it, we have many scenarios that are limited by this scenario and cannot access Langfuse. |
Beta Was this translation helpful? Give feedback.
-
|
If we have HTML rendering capabilities, we can render JSON data using HTML to preview annotations. Additionally, we can evaluate multimodal requirements, including input and output data such as images and videos ~~ |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the addition. We will consider this as we build this. |
Beta Was this translation helpful? Give feedback.
-
|
When is this coming? |
Beta Was this translation helpful? Give feedback.
-
|
We do not have a clear timeline for this unfortunately but this is high on our priority list. |
Beta Was this translation helpful? Give feedback.
-
|
would be great to have this, along with a more powerful annotation view that combines trace-level details with session-level overview (i.e. we need to see the whole session to assign useful scores, but want a possible drilldown to trace-level. right now you need to choose between seeing the full session at insufficient level of detail, or seeing a detailed trace without context) |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Describe the feature or potential improvement
Problem:
When using frameworks like LangGraph, the resulting traces in Langfuse can be highly complex, deeply nested JSON objects. This is ok for debugging but makes annotation inefficient. Annotators have to manually dig through the JSON to find the relevant fields to score, which doesn't scale.
Here's a screen showing the typical trace complexity and how it looks in the annotation view:
Proposed Solution:
We propose a configurable "Annotation View" where we can define which parts of the trace object to display.
The UI for this could work just like the existing "Custom Evaluators" and "Variable mapping" when setting them up. The ability to define variables by selecting a trace object (Input, Output, Span) and providing an optional JSONPath seems like a good fit here.
By applying this to the annotation queue setup, an administrator could define all the necessary fields an annotator needs to see for a specific task. For example, we could configure a view with several custom-named variables:
Trace, Variable:Input, JsonPath:$.messages[0].content)Span, Name:problem_normalizer, Variable:Output, JsonPath:$.update.problem_description)Span, Name:execute_query, Variable:Input, JsonPath:$.reason)Trace, Variable:Output, JsonPath:$.messages[-1:].content)Example:
This way, annotators see only the fields they need for the task, instead of having to parse the full JSON trace by hand. It would make annotation faster and less error-prone.
Additional information
No response
Beta Was this translation helpful? Give feedback.
All reactions