Testing AI coding tools on visualization projects
We give agents the same visualization task and record the result, time spent, and help they need from a person. The aim is to see how much work they can take on.
We want to know how much help an agent needs along the way.
01 / Context
A fair comparison requires the same code, context, and permissions, with no answers left in the test project. Chart usability and layout also need inspection in the browser.
What we built
The platform prepares isolated projects, runs agents, and records their work. Reports include automated checks, browser captures, and anonymous review materials. Model samples are still limited, so findings need further testing.
02 / System
Builds minimal fixtures from real projects while excluding answers, prior patches, and unequal context.
Record time, tool calls, token use, recovery, and human help, alongside the files produced by each run.
Automated checks cover code and page behavior. Browser captures support anonymous human review of visuals and usability.
03 / Decisions
Agents can diagnose, edit, and test as they would in regular development. The evaluation records that whole process, including time spent fixing mistakes.
Reports include versions, run records, screenshots, and reviews so readers can retrace the findings.
04 / Our work
Role：We designed the evaluation method and built the platform, including task selection, runners, result checks, and reports.
