Memdoor’s five-agent Qwen3.8-27B team solved 49 of 50 runs on ten hard coding problems, an average of 9.8 out of 10 per pass. Fable 5, in one call per problem, averaged 8.0 on the same ten, per Memdoor’s README.
That is a real result on a small, repeated sample. It shows the team matches the original study’s harness on those ten problems. It does not show that a 27B model is smarter than Fable 5, or cheaper per task.
The short answer
- Sample: the first ten problems from the GVS5H authors’ LiveCodeBench Hard list, run five times each. That is 50 runs over ten distinct problems.
- Scores: 9.8 for the Memdoor workflow, 9.6 for the GVS5H authors’ own harness, 8.0 for Fable 5 in a single call. Memdoor’s README calls 9.8 and 9.6 effectively tied.
- Fairness: each Memdoor run made 5 to 22 model calls, per its run file. Fable 5 made one call per problem. That is two systems with different budgets, not two models head to head.
- Cost: Memdoor’s 50 runs cost $35.79, per its notes. The cost section below explains why that does not settle cost against Fable 5.
- Logs: the run files are public on GitHub, so anyone can recount the scores.
What exactly did Memdoor run?
Memdoor is a coding and automation agent in a single CLI binary, licensed Apache 2.0, per its README. Its team-solve workflow rebuilds the method from GVS5H, a study from Persis Capital Inc. Research Lab. In that method, fresh copies of one model split a problem and coordinate through shared files.
The workflow uses five model roles, a planner, two ideators, a solver and a manager, plus a repair worker, per the workflow README. Each starts with a fresh context. A script, not a model, checks the public sample tests. The final score comes from hidden tests the loop does not see.
Memdoor’s notes say it matched the authors’ settings, including temperatures of 0.2 to 0.4, a 128K reply cap and up to 10 rounds. The authors served the model on their own vLLM server in FP8. Memdoor ran it through OpenRouter, not on a local machine. On Hacker News, the developer confirmed OpenRouter and said Memdoor was created months ago, so the news is the reproduction, not the tool.
The run IDs in Memdoor’s run file carry dates of 30 and 31 October 2026.
How do the three scores break down?
Each pass covers the same ten problems, so a score of 9 means nine of ten solved in that pass. Figures below are from Memdoor’s README table and the authors’ file it republished.
| Pass | Memdoor workflow | GVS5H authors’ harness | Fable 5, one call |
|---|---|---|---|
| 1 | 10 | 9 | 7 |
| 2 | 10 | 10 | 8 |
| 3 | 10 | 10 | 10 |
| 4 | 10 | 9 | 8 |
| 5 | 9 | 10 | 7 |
| Total of 50 | 49 | 48 | 40 |
| Mean | 9.8 | 9.6 | 8.0 |
Memdoor did not rerun the last two columns. Its README says they come from the authors’ regraded result files, and that Fable 5 was not run in Memdoor’s harness. The README says the authors’ harness ran on Qwen3.8-27B, the same model.
The authors’ file, authors.tsv, holds a third arm labeled single. It solved 8, 6, 4, 7 and 6 problems in passes 1 to 5, or 31 of 50. The file gives no model name for that arm, and Memdoor’s README table leaves it out.
Why ten problems five times is not fifty problems
Repeating a problem tells you how steady a system is on it. It does not add new kinds of problems. Our count from authors.tsv shows where Fable 5 lost its ten runs:
- Problem 3717: solved in 1 of 5 passes.
- Problem abc399_e: solved in 1 of 5 passes, the same pass as 3717.
- Problem arc196_a: solved in 3 of 5 passes.
- The other seven problems: solved in all five passes.
So the whole 1.8-point gap sits on three problems. Swap those three for three others and the gap could shrink or grow. Ten problems cannot tell you which.
The subset also differs from the full study. The GVS5H paper’s abstract reports Fable 5 at 90.4% on 100 of the latest hard LiveCodeBench problems. On Memdoor’s ten, Fable 5 solved 80% of runs, by our arithmetic. The two setups score differently, so read that as a warning about small subsets, not as a second result.
Age matters too. Memdoor’s README says these contests date from spring 2025, so the model may have seen them in training.
Is a team against one call a fair fight?
Not as a test of raw model skill. Memdoor’s run file lists 345 model calls across the 50 runs, between 5 and 22 per run. Fable 5 got one call per problem. Each team run also took up to 8 rounds in the file.
The same file shows 15,309,678 output tokens for the team. By our arithmetic, that is about 306,000 per run on average. The authors’ file has no cost or token columns for any arm, so the token and dollar gap against Fable 5 cannot be measured from these files.
A fairer reading: an organized open-weight setup can match a frontier model’s single answer on these problems, if you accept many more calls and long run times. Memdoor’s notes put the median run at 40 minutes of model time and the longest at 4 hours. That is a systems result, not a model ranking.
What does the team run cost?
These are Memdoor’s own figures, from its notes and run file:
| Measure | Figure |
|---|---|
| Total for 50 counted runs | $35.79 |
| Including 16 attempts cut by a clock, stopped for a fix or paused | $38.77 |
| Median run | $0.39 |
| Mean run (our arithmetic: $35.79 / 50) | about $0.72 |
| Two costliest runs (pass 4, arc196_d and arc196_c) | $3.44 and $4.24 |
The mean is almost double the median because a few long runs cost a lot. Budget from the mean, plus extra attempts, not from the median.
The GVS5H abstract says an orchestrated Qwen3.8 Flash Next reached 93.0% at 9% of Fable 5’s cost. That claim covers a different model and the full 100 problems, and its dollar charts are images we could not read. It does not transfer to this ten-problem run.
Where did the team lose, and why does it matter?
Memdoor’s notes list the failures. In pass 5, a correct first program for abc399_e was replaced after the manager invented a counterexample with a wrong expected answer. The new program passed the samples but 32 of 44 hidden tests.
In pass 4, correct programs for arc196_c and arc196_d were overwritten because the manager did not declare them done. Both came back later, which made these the two costliest runs. The notes name fixes but say the published numbers do not include them.
Per the notes, these losses came from coordination between roles. A team adds steps that can fail on their own.
Who should care, and who can skip it?
Worth your time if
You run coding agents and want to test whether an open-weight team can stand in for a frontier call on your own task set. The workflow README says the steps name no model, so you can point it at another one.
Skip it if
You want a ranking of models. Ten repeated problems, with a team on one side and a single call on the other, cannot give you one.
What to do next
- We suggest testing on your own problems, not on contest problems a model may have seen.
- We suggest giving both systems a similar budget before you compare scores, or reporting cost per solved problem for each.
- Count total tokens and wall time per finished task, not price per token. Our piece on why a cheaper model can cost more per finished task walks through the same idea.
- Read the licenses separately. Apache 2.0 covers Memdoor itself. The GVS5H GitHub page lists its code as MIT and its paper, figures and run data as CC BY 4.0. The model and the hosts you call have their own terms.
Questions readers ask
Did a Qwen3.8-27B team beat Fable 5?
On these ten problems, yes: 49 of 50 runs against Fable 5’s 40 of 50. That gap sits on three problems, and the team made 5 to 22 calls per run against one.
Did Memdoor beat the original GVS5H harness?
No. Memdoor’s own README calls 9.8 and 9.6 effectively tied and frames the claim as parity.
Did this run on a laptop or local GPU?
No. The model ran through OpenRouter, on hosts including Modal, CoreWeave and Wafer, per Memdoor’s notes.
How many problems did the original study use?
GVS5H used 100 of the latest hard LiveCodeBench problems. Memdoor used the first ten of the authors’ list.
Glossary
- Pass: one full run of a system over the same problem set.
- Harness: the code and prompts that drive a model through a task.
- Hidden tests: grading tests the system cannot see while it works.
- Contamination: when test problems may have appeared in a model’s training data.
Sources
- Memdoor, project README on GitHub
- Memdoor, team-solve workflow: README, notes and run files
- Memdoor, team-solve results summary
- Hacker News, Memdoor discussion with the developer’s comments
- Persis Capital Inc. Research Lab, GVS5H on GitHub
- Gao, Khosrowshahi, Khosrowshahi, Sun, Lee and Lee, GVS5H paper on arXiv
Facts read on 1 November 2026.
Part of the guide: AI model pricing and API cost guide.
