Instruction optimization

Instruction optimization tests revised question instructions against labeled examples. Use it when the model misses distinctions you can describe in words, such as which support tickets need immediate attention.

1. Collect and label examples

Start with the support-priority model. Include typical tickets and borderline cases, and label each one with the expected answer. This example adds one labeled ticket; repeat for a representative set so the optimizer can learn and test on separate splits.

from modaic import Modaic

client = Modaic()

model = client.models.get(workspace="acme", model="support-priority")

model.examples.ingest(examples=[{
    "state": "Checkout is unavailable.",
    "annotation": {"ground_truth": {"urgent": True}},
}])

2. Optimize the current revision

Start from the current main commit. Give each saved question text instructions so the optimizer has a starting point.

job = model.jobs.alignments.create(
    branch="main",
    source_commit_sha=model.commit.commit_sha,
    idempotency_key="support-optimize-v1",
)
print(job.id)

3. Review the result

Check the job until it finishes. If the new instructions improve the evaluation score, the job records a new model revision. Compare the result with your starting revision on tickets that matter to your workflow.

finished = client.alignments.wait(job.id)
print(finished.status, finished.result_commit_sha)

See the API reference for job options and Fine-tuning for weight training.