--- format: "aidr-story-markdown/v1" id: "7ff8deb6c282f363de1a7c06ee139aefc4b04e32bef922306b6d212fb5511dfc" canonical_url: "https://aidr.today/7ff8deb6?lang=en" title: "Parsewave Fixes 206 Zapier Benchmark Bugs to Revise 28% of Kimi Grades" lang: "en" requested_lang: "en" available_langs: ["en","vi"] translation_fallback: null fallback_fields: [] published_at: "2026-10-06T22:24:38.000Z" category: "Models" topics: ["llm"] source_urls: ["https://marketbrief.now/ai/parsewave-fixes-206-zapier-benchmark-bugs-to-revise-28percent-of-kimi-gr-ad46118c","https://huggingnews.com/ai/parsewave-fixes-206-zapier-benchmark-bugs-to-revise-28percent-of-kimi-gr-ad46118c"] summary: "Parsewave reviewed all 600 public tasks in Zapier's AutomationBench to test whether the verifiers could be fooled by incorrect answers. The auditor used agents to generate plausible but wrong responses, resulting in the discovery of 206 bugs that the firm subsequently fixed in a release called AutomationBench Verified. When Parsewave replayed 1,235 runs of the Kimi K3 model using the updated graders, 344 of those entries, or 27.9%, received a different score." --- # Parsewave Fixes 206 Zapier Benchmark Bugs to Revise 28% of Kimi Grades > [Open the canonical story]() **Published:** 2026-10-06T22:24:38.000Z **Category:** Models **Topics:** llm ## Summary Parsewave reviewed all 600 public tasks in Zapier's AutomationBench to test whether the verifiers could be fooled by incorrect answers\. The auditor used agents to generate plausible but wrong responses, resulting in the discovery of 206 bugs that the firm subsequently fixed in a release called AutomationBench Verified\. When Parsewave replayed 1,235 runs of the Kimi K3 model using the updated graders, 344 of those entries, or 27\.9%, received a different score\. ## Sources - [Story source]() - [Story source]()