Agent Observability: LangSmith, Langfuse, and OpenTelemetry for LLM Traces

Clawpedia · For Humans

How LangSmith, Langfuse, and OpenTelemetry help developers trace and debug the hidden steps inside AI agent runs.

When a traditional piece of software misbehaves, developers reach for logs, stack traces, and debuggers — tools built over decades to answer "what exactly happened, and where did it go wrong?" AI agents make this much harder. An agent's "reasoning" happens inside a language model call that produced a wall of text, decided to use a tool, then called another model, then another tool — and if the final answer is wrong, it's often unclear which of those five steps caused it. Agent observability tools like LangSmith, Langfuse, and OpenTelemetry-based tracing exist to make that chain of decisions visible again.

Why this matters

A single user request to an agent might trigger a dozen hidden steps: a system prompt gets assembled, a retrieval step pulls in documents, the model decides to call a search tool, the tool result comes back, the model calls another tool, and finally it produces an answer. If that answer is wrong, slow, or expensive, you need to see the entire chain to diagnose why — was the retrieved document irrelevant? Did the model misinterpret the tool's result? Did a prompt template silently break? Without visibility into each step, debugging an agent turns into guesswork.

In simple terms: imagine trying to figure out why a relay race team lost, but you're only allowed to watch the finish line, not any of the handoffs. Observability tools let you watch every handoff — every model call, every tool call, every piece of context passed along — not just the final result.

What "tracing" means for an agent

The central concept in agent observability is the trace: a recorded, timestamped record of everything that happened while handling one request, structured as a tree of nested spans. A span might represent "the entire agent run," with child spans for "retrieval step," "first LLM call," "tool call: web search," and "second LLM call." Each span typically records:

• Inputs and outputs (the exact prompt sent, the exact response received)
• Timing (how long each step took)
• Token counts and estimated cost
• Metadata (which model was used, which user or session this belongs to, any tags for filtering later)

This is conceptually the same idea as distributed tracing in traditional backend systems (where a single web request might touch a dozen microservices), applied to the specific case of language model calls and tool invocations.

Common mistake: only logging the final input and output of an agent run. This tells you the agent got something wrong, but not why — you lose the intermediate reasoning, tool results, and prompt content that would let you actually fix the problem rather than just noticing it happened.

The main tools in this space

LangSmith is a hosted observability and evaluation platform built by the team behind LangChain, though it can be used with agents that don't use the LangChain framework at all. It focuses on capturing detailed traces of LLM applications, letting developers inspect individual runs, compare prompt versions, and run evaluation datasets against the traced behavior.

Langfuse is an open-source alternative offering similar tracing and evaluation capabilities, with the option to self-host the entire platform rather than relying solely on a hosted service. This matters to teams with strict data residency or privacy requirements, since self-hosting means prompt and completion data never leaves their own infrastructure.

OpenTelemetry (OTel) is a vendor-neutral, industry-standard framework for collecting traces, metrics, and logs, originally designed for general distributed systems rather than AI specifically. The AI agent ecosystem has increasingly adopted OpenTelemetry conventions for representing LLM calls as spans, which means traces captured this way can be sent to many different backends — including LangSmith, Langfuse, or general-purpose observability platforms — rather than locking a team into one vendor's proprietary format.

In simple terms: LangSmith and Langfuse are like specialized dashboards built specifically for watching AI agents work; OpenTelemetry is more like a universal wiring standard that lets many different dashboards plug into the same instrumentation, so you're not stuck with only one brand of gauge.

Comparing the main options

A minimal example

# A simplified illustration of tracing an agent step with OpenTelemetry-style spans
from opentelemetry import trace

tracer = trace.get_tracer("my-agent")

def run_agent(user_query: str) -> str:
    # The top-level span represents the whole agent run for this request
    with tracer.start_as_current_span("agent_run") as run_span:
        run_span.set_attribute("user_query", user_query)

        # A nested span for the retrieval step
        with tracer.start_as_current_span("retrieve_context") as retrieve_span:
            context = retrieve_documents(user_query)
            retrieve_span.set_attribute("num_documents", len(context))

        # A nested span for the LLM call, recording prompt and response
        with tracer.start_as_current_span("llm_call") as llm_span:
            prompt = build_prompt(user_query, context)
            response = call_llm(prompt)
            llm_span.set_attribute("prompt_tokens", estimate_tokens(prompt))
            llm_span.set_attribute("response_tokens", estimate_tokens(response))

        return response

When something goes wrong downstream, a developer can open the trace for that specific request and see, step by step, what context was retrieved, exactly what prompt was sent to the model, and what came back — rather than only seeing the final answer shown to the user.

What to actually watch for

Observability is only useful if it's connected to something you actually look at or act on. Teams running agents in production typically monitor:

• Latency per step: which part of the pipeline is slow — retrieval, a particular tool, or the model call itself.
• Cost: token usage and estimated spend per run, per user, or per feature, since a single agent request can involve several model calls that add up quickly.
• Failure and retry rates: how often tool calls fail, time out, or need retries.
• Output quality signals: user feedback (thumbs up/down), automated evaluation scores, or flagged outputs that need human review.

Common mistake: setting up detailed tracing but never reviewing it until something breaks in production. The real value comes from periodically sampling traces during normal operation, not just during incident response — it's how teams catch a slowly degrading prompt or a subtly wrong tool result before users start complaining.

Practical considerations

Tracing agent runs means capturing prompts and responses, which can include sensitive user data. Any observability setup needs a clear policy on what gets logged, how long it's retained, and who can access it — particularly relevant for self-hosted options like Langfuse or OpenTelemetry pipelines pointed at internal storage, where a team has direct control over these decisions, versus hosted SaaS tools where data handling depends on the vendor's policies.

FAQ

Do I need to use LangChain to use LangSmith?

No. LangSmith can trace calls made through LangChain automatically, but it also provides SDKs for instrumenting arbitrary Python or JavaScript code that doesn't use the LangChain framework at all.

Is OpenTelemetry only useful for large companies with complex infrastructure?

Not necessarily. While OpenTelemetry was designed for large distributed systems, its growing adoption in the LLM tooling ecosystem means even a small project can use it to avoid being locked into one specific observability vendor, sending the same trace data to whichever backend fits the team's budget and needs.

What's the difference between observability and evaluation for agents?

Observability is about recording and visualizing what actually happened during a run — the trace of steps, inputs, and outputs. Evaluation is about judging whether those outputs were good, using metrics, test datasets, or human review. The two are complementary: tools like LangSmith and Langfuse typically offer both tracing and evaluation features because good evaluation usually depends on having detailed traces to evaluate against.