Examples with the SDK

Examples are known data points (inputs and their labels) used for evaluation and alignment. They can include the ground truth decision and rationale during ingestion or be annotated later.

Setup

import { Modaic } from "modaic";

const modaic = new Modaic();
const model = await modaic.models.get("acme", "support-priority");

Ingest examples

Models returned by models.create, models.get, or models.update expose model.examples. The model ID is supplied automatically. With AsyncModaic, await each method. The top-level modaic.examples methods remain available.

const created = await model.examples.ingest([
  {
    state: { ticket: "I was charged twice." },
    annotation: {
      groundTruth: { needs_refund: true, priority: "high" },
      groundReasoning: "The ticket reports a duplicate charge.",
    },
  },
  { state: { ticket: "How do I change my profile picture?" } },
]);

const exampleId = created.examples[0].id;

Supply an id on each input when retries should identify the same example. When an example first receives ground truth, Modaic assigns a stable train or test split. Updating the annotation later preserves that split.

List and retrieve

const page = await model.examples.list({ page: 1, pageSize: 30 });
const example = await model.examples.get(page.items[0].id);

console.log(page.total, page.totalPages);
console.log(example.state, example.annotation, example.latestDecision);

Annotate and inspect history

const updated = await model.examples.annotate(exampleId, {
  groundTruth: { needs_refund: true, priority: "high" },
  groundReasoning: "Confirmed duplicate charge.",
});

const history = await model.examples.listDecisions(exampleId);
for (const decision of history.decisions) {
  console.log(decision.source, decision.answers, decision.error);
}

Ground-truth values are keyed by question ID: booleans for noul, option keys for choice, and zero-based numeric levels for score questions.

Was this page helpful?