Confidence estimation

Confidence helps you decide which Model decisions can proceed automatically and which deserve human review. It is most useful as a ranking and thresholding signal, not as a substitute for measuring agreement against labeled examples.

Confidence by question type

• A noul answer returns noul, the probability of true from 0 to 1.
• A choice answer returns probabilities for every configured option and a confidence value from 0 to 1.
• A score answer returns probabilities over the score legend and a confidence value from 0 to 1.

Use confidence for review

A practical workflow is:

1. Choose a confidence threshold for the decision's risk level.
2. Send decisions below the threshold to a human reviewer.
3. Add ground truth and concise reasoning to corrected examples.
4. Run alignment after you have representative feedback.
5. Re-measure agreement and calibration for the new checkpoint.

Higher-risk decisions should use stricter thresholds and appropriate human oversight. Review a sample above the threshold as well, so systematic errors do not remain invisible simply because the Model was confident.

Metrics

Model checkpoints may include:

• Accuracy — fraction of labeled question answers that match ground truth.
• AUROC — how well confidence separates correct from incorrect decisions.
• ECE — how closely stated confidence matches observed correctness.
• Samples — number of labeled example-question pairs used for measurement.

Compare metrics only when the evaluation set and question definitions are compatible. A schema or rubric change can make checkpoint-to-checkpoint numbers look comparable when they are measuring different tasks.