ADK.Eval.Scorer behaviour (ADK v0.0.1-alpha.1)

Copy Markdown View Source

Behaviour for evaluation scorers.

A scorer examines an agent's response (list of events) and returns a score between 0.0 and 1.0, along with a pass/fail determination and details.

Implementing a custom scorer

defmodule MyScorer do
  @behaviour ADK.Eval.Scorer

  @impl true
  def score(events, opts) do
    # ... analyze events ...
    %{score: 1.0, pass: true, details: "looks good"}
  end
end

Summary

Types

score_result()

Callbacks

score(events, opts)

Functions

function_calls(events)

Extract all function calls from events.

response_text(events)

Extract all text from agent response events (non-user, non-partial).

Types

score_result()

@type score_result() :: %{score: float(), pass: boolean(), details: String.t() | nil}

Callbacks

score(events, opts)

@callback score(events :: [ADK.Event.t()], opts :: keyword()) :: score_result()

Functions

function_calls(events)

@spec function_calls([ADK.Event.t()]) :: [map()]

Extract all function calls from events.

response_text(events)

@spec response_text([ADK.Event.t()]) :: String.t()

Extract all text from agent response events (non-user, non-partial).