[Lanre Shittu (Shizoqua)]

Summary

Fixes #384.

OfficeQA scored the first <answer> tag in a response. When a model drafted an answer and then revised it in the same message, the draft was graded: a correct revision was marked wrong, and a wrong revision of a correct draft was credited. OfficeQA now scores the last tag, as SearchQA, LiveMathematicianBench and DocVQA already do.

User impact: hard and soft are computed from the extracted answer, so the training reward and reported OfficeQA scores followed the draft instead of the model's final answer.

Proof: the regression tests fail on upstream ccfdcdf and pass with this change.

Root cause

_extract_answer() in skillopt/envs/officeqa/rollout.py used _FINAL_RE.search(text), which returns the first match. The other tagged benchmarks take the last one: SearchQA and LiveMathematicianBench use re.findall(...)[-1], and DocVQA uses rfind("<answer>").

Changes

• _extract_answer() uses _FINAL_RE.findall(text) and takes the last match. The fallback to the last non-empty line when there is no tag is unchanged.
• All five call sites in rollout.py that read the final answer go through _extract_answer(), so this one change covers every OfficeQA rollout path.
• New tests/test_officeqa_answer_extraction.py. Two tests run the real rollout.process_one() with a fake model response, the same pattern as tests/test_officeqa_parallel_search.py; the rest check _extract_answer() directly, including single tags, case, whitespace and the no-tag fallback.
• CHANGELOG entry under Unreleased / Fixed.

Reproduction

from skillopt.envs.officeqa.evaluator import evaluate
from skillopt.envs.officeqa.rollout import _extract_answer

for response in [
    "<answer>1774</answer>\nChecking the date again.\n<answer>1776</answer>",
    "<answer>1776</answer> On reflection: <answer>1774</answer>",
]:
    answer = _extract_answer(response)
    print(repr(answer), int(evaluate(answer, "1776")["em"]))

Before: '1774' 0 and '1776' 1 (the draft is scored).

After: '1776' 1 and '1774' 0 (the revision is scored).

Testing

• python -m pytest tests/test_officeqa_answer_extraction.py -q: on upstream ccfdcdf, 3 failed (assert '1774' == '1776' and similar) and 4 passed; with the fix, 7 passed.
• python -m pytest tests/test_officeqa_answer_extraction.py tests/test_officeqa_parallel_search.py tests/test_officeqa_keenable_search.py tests/test_benchmark_task_output_paths.py -q: 84 passed.
• python -m pytest -q: 2105 passed, 12 skipped, 4 xfailed.
• python -m ruff check and python -m ruff format --check on the changed files: clean.
• python -m mkdocs build --strict: passed.

Risk and compatibility

Responses with zero or one <answer> tag are extracted exactly as before. Only responses with several tags change, and they now match the other benchmarks. OfficeQA scores can change for runs whose target model revises its answer within one response, so results from before and after this fix are not directly comparable. No backend, config key, file format or CLI change.