AboutA study of 28 defects reported in Spec Kit, GitHub’s toolkit for spec-driven development.
Spec Kit (opens in a new tab) is GitHub’s open-source toolkit for spec-driven development. It sets up a project so that an AI coding agent works from a written specification, a plan and a task list.
Its command-line tool, specify, also installs extensions, presets, bundles and workflows, and wires them into each supported coding agent.
On 23 October 2026, Harry filed 28 defect reports against Spec Kit 1.2.4.dev0. This study lists them, how each was checked, and what happened upstream.
Each piece of evidence has one of five strengths: Proved, Tested, Decided by Harry, Assumed or Not yet reached. Nothing here is Proved. The defects were tested, one reproduction at a time.
The workflow expressions study looks more closely at three of the reports.
Finding28 reproduced defects; by 10 November 2026, 19 had a merged fix and 8 more had a fix open.
Most of the defects fail quietly. A build reports success but leaves out a required file. A re-init resets a setting. A workflow condition takes the wrong branch.
Each report names the line of Spec Kit’s own documentation, docstring or tests that the behaviour contradicts.
The maintainers triaged all 28 as in scope. Other contributors wrote the fixes.
StudyEach report is a reproduction script with recorded output, checked against Spec Kit’s own documentation.
Claude Code ran CGV over Spec Kit’s source, then eight subagents searched one slice each with Crosscheck’s /reason method. All 28 defects came from that search; CGV found none of them.
A finding counted only when a reproduction script showed the wrong behaviour. The agent that found it ran the script, and the main agent ran it again.
A reproduction shows that the defect happens. It does not prove anything about inputs it does not try.
RepeatEvery reproduction runs against Spec Kit commit 27d14e2.
Clone Spec Kit, check out that commit, run uv sync, then follow the report’s steps.
Run the same steps on Spec Kit’s main branch to see whether a defect is fixed.
Questions
Each shows a short answer and the evidence for it.
Evidence strength
- Proved
- Tested
- Decided by Harry
- Assumed
- Not yet reached
Evidence
Overview
The reports at a glance
.
28 reports were filed on 2026-10-23. By 2026-11-10, 19 of 28 had a merged fix and 8 more had a fix open for review. One had neither.
| Upstream state | Reports |
|---|---|
| A fix has merged | 19 |
| A fix is proposed and open | 8 |
| No fix merged or open | 1 |
| Issue closed by the maintainers | 16 |
| Triaged “must have” | 15 |
| Triaged “can wait” | 11 |
| Triaged “nice to have” | 2 |
Spec Kit’s maintainers give each report a triage verdict. “must have” reads: Verdict: high-value, important work for Spec Kit — do first. “can wait” reads: Verdict: valid and in-scope but deprioritized; held behind the evidence gate. “nice to have” reads: Verdict: evidence-backed fix or greenlit feature — land after review. None of the 28 was triaged out of scope.
Findings
What was found?
28 defects in Spec Kit, GitHub’s toolkit for spec-driven development with AI coding agents. They cover workflows, extensions, presets, bundles, agent integrations and the command line.
Most are quiet failures. A command reports success, or a condition takes the wrong branch, with no error.
Each report names the line of Spec Kit’s own documentation, docstring, test or sibling code that the behaviour contradicts. Two reports infer the expected behaviour instead.
How were they found?
In one Claude Code session on 22 October 2026. CGV ran first, over Spec Kit’s source. Its 39 errors turned out to hold no real bug.
Then eight subagents each searched one part of the source, following Crosscheck’s /reason method. A finding counted only when a script reproduced it.
38 findings were reproduced. 31 became the 28 reports. Six were left out because an open issue or pull request already covered them, and one was reported privately to GitHub’s security team.
Are the defects real?
Each report has a reproduction script and the output it gave at Spec Kit commit 27d14e2. Two agents ran each script in the session that found it, and Harry re-ran every one before filing.
Spec Kit’s maintainers triaged all 28 as in scope. 19 now have a merged fix.
This website did not rerun the reproductions.
Workflows
.
7 reports on workflow commands, catalogs and the expression evaluator.
| Report | What goes wrong | What it contradicts | Upstream |
|---|---|---|---|
| #5071 (opens in a new tab) | Chained comparisons such as 0 <= x <= 10 in workflow expressions silently return the wrong result | The module docstring calls the evaluator “a safe Jinja2 subset”, and Jinja2 and Python both chain comparisons. | A fix, #5203 (opens in a new tab), is open. Issue open. Triaged “must have”. |
| #5072 (opens in a new tab) | == and != do not compare numeric strings as numbers, while <, >, <= and >= do | The _safe_compare docstring and a test in tests/test_workflows.py say numeric strings compare as numbers. | Fixed by #5310 (opens in a new tab). Issue closed 2026-11-09. Triaged “can wait”. |
| #5073 (opens in a new tab) | Unsupported expression syntax evaluates to None silently instead of raising | The module docstring calls the evaluator “a safe Jinja2 subset” and lists string and numeric literals as supported. | A fix, #5206 (opens in a new tab), was closed without merging. Issue open. Triaged “must have”. |
| #5074 (opens in a new tab) | catalog add gives the new catalog the same priority as the first existing catalog when priorities are implicit | A code comment says the new priority is the highest existing priority plus one. | A fix, #5123 (opens in a new tab), is open. Issue open. Triaged “must have”. |
| #5075 (opens in a new tab) | workflow update rejects an archive catalog release when its version has leading zeros (for example CalVer 2024.02.01) | The workflow version format accepts 2024.02.01, and the same catalog entry installs when its URL is a .yml file. | Fixed by #5155 (opens in a new tab). Issue closed 2026-10-28. Triaged “nice to have”. |
| #5076 (opens in a new tab) | Workflow commands crash with rich.errors.MarkupError when user or catalog text contains square brackets | command_info.py treats every workflow.yml field as untrusted, and workflow search already escapes the same text. | A fix, #5112 (opens in a new tab), is open. Issue open. Triaged “must have”. |
| #5077 (opens in a new tab) | workflow add <url> installs workflow.yml with mode 0600, while a catalog install uses the project file mode | A test asserts mode 0644 for a catalog install, and the _stage_workflow_file docstring describes the normal project mode. | A fix, #5150 (opens in a new tab), is open. Issue open. Triaged “can wait”. |
Extensions
.
6 reports on extension manifests, aliases, updates and configuration.
| Report | What goes wrong | What it contradicts | Upstream |
|---|---|---|---|
| #5080 (opens in a new tab) | A hook with a list-valued command passes manifest validation, then crashes extension add and leaves files behind | EXTENSION-API-REFERENCE.md documents command as a string, and the manifest promises a ValidationError for a bad manifest. | Fixed by #5146 (opens in a new tab). Issue open. Triaged “must have”. |
| #5081 (opens in a new tab) | Extension alias conflict check misses spellings that map to the same file, so one extension overwrites and then deletes another’s alias | The _validate_install_conflicts docstring says installs that would shadow installed extension commands are rejected. | A fix, #5134 (opens in a new tab), is open. Issue open. Triaged “must have”. |
| #5082 (opens in a new tab) | specify extension update crashes with KeyError when a catalog entry has no version, and stops checking the other extensions | The same function skips an invalid catalog version with a warning, and catalog loading tolerates one malformed entry. | Fixed by #5108 (opens in a new tab). Issue closed 2026-11-02. Triaged “must have”. |
| #5083 (opens in a new tab) | Local config override file name differs between the docs (<id>-config.local.yml) and ConfigManager (local-config.yml) | EXTENSION-USER-GUIDE.md names the override file <id>-config.local.yml. | A fix, #5159 (opens in a new tab), is open. Issue open. Triaged “must have”. |
| #5084 (opens in a new tab) | Top-level defaults: in extension.yml, as documented and used in the extension template, is ignored by ConfigManager | EXTENSION-API-REFERENCE.md and the extension template put defaults under a top-level defaults key. | Fixed by #5164 (opens in a new tab). Issue closed 2026-10-28. Triaged “must have”. |
| #5094 (opens in a new tab) | specify extension add --dev shows a traceback when .extensionignore is a directory | The _install_ignore.py docstring says an invalid ignore file raises a validation error before copying. | Fixed by #5131 (opens in a new tab). Issue closed 2026-10-26. Triaged “can wait”. |
Presets
.
3 reports on how presets compose and resolve commands.
| Report | What goes wrong | What it contradicts | Upstream |
|---|---|---|---|
| #5085 (opens in a new tab) | preset add crashes with AttributeError when a composing command’s frontmatter is not a YAML mapping | CommandRegistrar.parse_frontmatter treats the same non-mapping frontmatter as empty. | Fixed by #5125 (opens in a new tab). Issue closed 2026-10-28. Triaged “can wait”. |
| #5086 (opens in a new tab) | resolve_with_source reports a core template whose name starts with “overrides” as a project override | collect_all_layers reports the same file as a core template. | Fixed by #5129 (opens in a new tab). Issue closed 2026-10-26. Triaged “can wait”. |
| #5087 (opens in a new tab) | specify preset set-priority does not update agent command files that are already written | docs/reference/presets.md says the preset with the lowest priority number wins. | A fix, #5113 (opens in a new tab), is open. Issue open. Triaged “must have”. |
Bundles
.
4 reports on building, validating and installing bundles.
| Report | What goes wrong | What it contradicts | Upstream |
|---|---|---|---|
| #5088 (opens in a new tab) | bundle build reports success but omits a symlinked bundle.yml/README.md, producing an artifact that cannot be installed | The build validates bundle.yml and README.md, then leaves them out of the artifact it reports as built. | Fixed by #5169 (opens in a new tab). Issue closed 2026-10-29. Triaged “can wait”. |
| #5089 (opens in a new tab) | bundle init <bundle> initialises with the default integration instead of the bundle’s, then aborts with a clash | bundle install initialises with the bundle’s declared integration. | A fix, #5111 (opens in a new tab), is open. Issue open. Triaged “must have”. |
| #5090 (opens in a new tab) | bundle install rejects a local artifact with an upper-case .ZIP extension | docs/reference/bundles.md accepts a local .zip artifact, and the download path compares the extension case-insensitively. | Fixed by #5106 (opens in a new tab). Issue closed 2026-10-26. Triaged “can wait”. |
| #5091 (opens in a new tab) | integration: mapping without an id makes a bundle silently integration-agnostic | A comment in manifest.py guards against a declared integration being dropped silently. | Fixed by #5161 (opens in a new tab). Issue closed 2026-10-28. Triaged “can wait”. |
Integrations and events
.
5 reports on agent integrations and their event hooks.
| Report | What goes wrong | What it contradicts | Upstream |
|---|---|---|---|
| #5078 (opens in a new tab) | integration switch leaves the manifest out of date when a preset overrides a core command | The _resync_manifest_after_registration docstring says the manifest is re-hashed after preset registration. | Fixed by #5145 (opens in a new tab) and #5193 (opens in a new tab) and #5274 (opens in a new tab). Issue open. Triaged “must have”. |
| #5079 (opens in a new tab) | Removing Vibe event hooks deletes user comments that follow a Spec Kit hook block | The _without_managed_hooks docstring says it preserves every other source slice verbatim. | Fixed by #5109 (opens in a new tab). Issue open. Triaged “must have”. |
| #5092 (opens in a new tab) | integration-events.yml entry without an events key silently removes all event hooks | A code comment says only an explicitly present events field replaces the prior layers. | Fixed by #5102 (opens in a new tab). Issue closed 2026-10-26. Triaged “must have”. |
| #5093 (opens in a new tab) | Malformed events in an unregistered extension directory makes specify integration upgrade fail | validate_events rejects the same matcher and timeout values. | Fixed by #5133 (opens in a new tab). Issue closed 2026-10-28. Triaged “can wait”. |
| #5097 (opens in a new tab) | Cline extension and preset commands have speckit.*.md file names rewritten to speckit-*-md | Inferred from the file naming in extensions/git; no document states it. | Fixed by #5168 (opens in a new tab). Issue closed 2026-11-02. Triaged “can wait”. |
Command line and scripts
.
3 reports on init, self upgrade and the feature script.
| Report | What goes wrong | What it contradicts | Upstream |
|---|---|---|---|
| #5095 (opens in a new tab) | specify self upgrade reports “unsupported” for uv tool installs under XDG_DATA_HOME (and likely for current pipx installs) | docs/upgrade.md says self upgrade detects uv tool and pipx installs. | Fixed by #5103 (opens in a new tab). Issue closed 2026-11-03. Triaged “nice to have”. |
| #5096 (opens in a new tab) | create-new-feature.sh rejects the feature descriptions -n, -e and -E, which the Python script accepts | The Python version of the script accepts the same descriptions, and a comment in the bash script says it uses printf for this reason. | Fixed by #5245 (opens in a new tab) and #5247 (opens in a new tab). Another fix, #5124 (opens in a new tab), is open. Issue closed 2026-11-06. Triaged “can wait”. |
| #5098 (opens in a new tab) | Re-running specify init --here --force resets feature_numbering in init-options.json to “sequential” | Inferred from the option merge in command_init.py and the documentation of the setting; no document states it. | Fixed by #5101 (opens in a new tab). Issue closed 2026-10-26. Triaged “must have”. |
Checks
How the defects were found
.
Only the reproductions count as evidence. The agents’ reasoning pointed at where to look.
A Claude Code session on 2026-10-22, from 23:29 to 23:59 UTC, against Spec Kit at the filing commit. The transcript is not public. The requests that started it read:
- “Now clone https://github.com/github/spec-kit and run CGV and Crosscheck on it. Orchestrate parallel subagents.”
- “Draft them in a file”
- “Also check if our findings match any open issues.”
CGV ran first. Then eight subagents each searched one slice of Spec Kit’s source for defects: Workflows core; workflow commands and steps; integrations; extensions; presets; bundles and artifacts; security, events and authentication; top-level modules and scripts.
A finding counted only when a reproduction script’s actual output showed the wrong behaviour. A finding argued from reading the code alone was dropped or marked unconfirmed. Each subagent followed the certificate steps of Crosscheck’s /reason skill: premises with file and line, traces, and checks of alternative explanations. Those certificates are an AI agent’s reasoning, so they are not evidence; only the reproductions count.
The main agent re-ran every reproduction script from the eight subagents. When drafting, each subagent made its reproduction self-contained and ran it again as written, and the main agent then ran all of them once more. Four findings about unsupported expression syntax became one report, #5073, so 31 findings became 28 reports.
A tenth subagent listed the titles of all 81 open issues and 98 open pull requests, ran two to six full-text searches for each finding and read every candidate. GitHub search does not cover comments or diffs.
The container had no Docker daemon, so Crosscheck’s Dafny and Lean pipelines did not run.
Each report records its reproduction’s output, from a Linux container and from macOS:
- Linux 6.18 (Ubuntu container), Python 3.12.3
- macOS (Darwin 25.6, arm64), Python 3.14.4
Every report names Spec Kit 1.2.4.dev0 at 27d14e2 (opens in a new tab) and ends: “Found and reproduced with Claude Code, run autonomously under human supervision. The human reporter re-ran every reproduction above before filing.”
What CGV found
.
| Scope | Functions | Edges | Exit | Errors | Unverified |
|---|---|---|---|---|---|
src/ | 1,851 | 17,361 | 1 | 39 | 993 |
scripts/python/ | 61 | 392 | 0 | 0 | 17 |
tests/ | 7,568 | 22,713 | 0 | 0 | 1,248 |
| Verdict | Sites |
|---|---|
| A real bug | 0 |
| Only the type annotation is wrong | 11 |
| False positive | 26 |
All 39 errors are nullability errors, most where a workflow step passes a value that may be None to the expression evaluator. A triage agent sorted the 37 distinct sites: 0 real bugs, 11 where only the type annotation is wrong, and 26 false positives where CGV missed a check that rules out None. Four of the false positives come from a defect in CGV’s extractor, which binds a function-local import to the wrong function. CGV found none of the 28 reported defects. The triage verdicts are an AI agent’s judgement, so they are not evidence that any site is safe; only the counts come from CGV itself. An exit code of 1 means only that the extracted graph has these errors.
What each report is measured against
.
Where Spec Kit’s documentation and its code disagree, either could be the one that is wrong.
No report rests on an outside opinion of how Spec Kit should behave. Each one quotes Spec Kit’s own statement of the behaviour: a docstring, a page of its documentation, a test, a code comment, or another code path that handles the same input differently.
That is an assumption. A report shows that code and statement disagree. It does not show which one the maintainers meant. In #5083 the fix that is open changes the documentation, not the code.
Two reports, #5097 and #5098, find no written statement and infer the expected behaviour from nearby code. Each says so.
Upstream
What did Spec Kit do with them?
The maintainers triaged every report: 15 as “must have”, 11 as “can wait” and 2 as “nice to have”.
Other contributors then opened fix pull requests. By 10 November 2026, 19 reports had a merged fix and 8 had a fix open for review. 16 issues are closed.
Three open issues carry an “invalid” label from an automated workflow. It added the label after their fixes had merged.
Six findings not filed
.
| Existing | Finding | Why it was not filed |
|---|---|---|
| #4916 (opens in a new tab) | Command step options, and workflow-level options, are recorded but never passed to the integration | Already reported as an open issue. |
| #4540 (opens in a new tab) | integration scaffold accepts keys such as base that shadow core modules and break imports | An open pull request fixes it. |
| #4745 (opens in a new tab) | A preset id with a trailing newline passes validation, installs, and is then ignored by the resolver | An open pull request fixes it. |
| #4470 (opens in a new tab) | bundle install leaves components installed and unowned when writing the bundle record fails | An open pull request fixes it. |
| #3903 (opens in a new tab) | The generated .specify/events.py dispatcher prints a traceback on non-UTF-8 input | An open pull request fixes it, judged from its commit messages, not its diff. |
| #3922 (opens in a new tab) | A corrupt step-registry.json is replaced on the next step add, dropping every other installed step | An open pull request fixes it. |
These were reproduced like the others. They are not counted in the 28.
One more finding was reported privately to GitHub’s security team by email, as Spec Kit’s SECURITY.md asks. This study does not describe it.
Three “invalid” labels
.
Spec Kit’s automated bug-assessment workflow added the invalid label to #5078, #5079 and #5080 on 2026-10-29, after fixes for all three had merged on 2026-10-27 and 2026-10-28. Its comment on #5078 says the label means “not currently reproducible/actionable on current codebase”, not “that the original report lacked merit”. Its comment on #5079 says it could not run the reproduction and read the code and tests instead.
Limits
What wasn’t checked?
The constraints most of these defects involve. CGV has no way to state rules about strings, paths or the shape of a dictionary, and Dafny and Lean did not run.
Whether each defect still reproduces on Spec Kit’s main branch. Upstream merged fixes are the maintainers’ evidence, not this study’s.
How much any defect matters to users. The study has no usage data.
Not yet reached
.
| Behaviour | Blocking property | Open question |
|---|---|---|
| Constraints on strings, paths and the shape of dictionaries, which most of Spec Kit’s defects involve | CGV checks type, nullability and choice requirements, and has no requirement kind for these constraints. | Which requirement kinds would let CGV check string, path and dictionary-shape constraints, and can their checks be proved sound like the existing ones? |
| A proof or reference model of any Spec Kit component | Dafny and Lean run in Docker, and the container that ran the study had no Docker daemon. | Is the expression evaluator, the one component with a stated model in Jinja2, specified precisely enough to model in Lean? |
| Whether each defect still reproduces on Spec Kit’s main branch | Running Spec Kit’s code needs an environment approved for third-party code, and the session that built this page had none. | Should the replay run in the same sandbox as Crosscheck’s Dafny and Lean containers, so third-party code never runs on the host? |
| A differential test of the expression evaluator against Jinja2 | Spec Kit now documents deliberate departures from Jinja2, such as left-to-right comparison grouping and numeric-string equality, so Jinja2 alone is no longer a correct reference. | Which Jinja2 behaviours does Spec Kit intend to keep, and can they be written down as a reference model to test against? |
| Whether the defects matter to users | The study has no data on how often the affected commands and inputs are used. | How should a finding’s severity be estimated when the project publishes no usage data? |
Repeat
Can I run a reproduction myself?
Yes. Each report gives its steps. You need git, uv and Python 3.12 or later.
Check out the filing commit to see the defect. Check out Spec Kit’s main branch to see whether it is fixed.
Repeat a reproduction
.
git clone https://github.com/github/spec-kit
cd spec-kit
git checkout --detach 27d14e20c187c1f2916540ea201ecbdc9f8ed588
uv sync
# Save the script from the report’s Steps to Reproduce as repro.py, then:
uv run python -I repro.pyEach report gives its own steps and its expected output under Actual Behavior. A few reports drive the command line instead of a Python script; follow the steps in the report. To see whether a defect is fixed, run the same steps on Spec Kit’s main branch.
People
What did Harry decide, and what did the AI do?
Harry chose Spec Kit, asked for CGV and Crosscheck with parallel subagents, and asked for the drafts and the search for existing reports.
The agents found the defects, wrote and ran the reproductions, and drafted the reports. Harry re-ran every reproduction and filed the reports. The maintainers’ verdicts and fixes are their own.
Who did what
.
- AI assistance
- Claude Code ran CGV, split Spec Kit’s source into eight slices with one subagent each, and had a ninth triage CGV’s errors and a tenth search for existing reports. The agents found the defects, wrote and ran the reproductions, and drafted the reports.
- Harry
- Harry chose Spec Kit as the target, asked for CGV and Crosscheck with parallel subagents, and asked for the drafts and for the search for existing reports. Harry re-ran every reproduction before filing, which gave the macOS output in each report, filed the 28 reports, and emailed the security report.
- What the checks establish
- Each reproduction is a script with recorded output at the filing commit. The maintainers’ triage and the merged fixes are their own decisions and are reported here, not counted as Crosscheck’s evidence.