NotJust.Buzz
Working protocols

Test the work,
not just the output.

No experiments have been run yet. These plans are here so you can see what a useful comparison would need.

A
First protocol / Coding tools

Ten repository tasks.

Compare a baseline workflow with an agent-assisted one. The clock stops when the change is accepted, not when the agent stops.

Scope

Ten representative tasks
Repeat where feasible

Outcome

Accepted changes
Total time and cost

Materials

Test plan only
No dataset or result yet

Read the protocol →
Before you run it

Keep enough of the setup
for someone else to repeat it.

  1. Environment

    Repo commit, dependencies, tool and model versions, operating system and hardware.

  2. Task and acceptance

    Representative inputs and a pass/fail rubric set before the run.

  3. Procedure

    Baseline, prompts, permissions, time limit and failure handling.

  4. Measurements

    All attempts, total time, corrections, retries and cost.

  5. Limits

    Sample size, variation, selection bias and conditions you did not test.