AboutA closer look at three Spec Kit reports about the evaluator for workflow conditions.
Spec Kit workflows decide what to run with conditions such as {{ inputs.n > 10 }}. A small evaluator in Spec Kit works out each condition.
At the filing commit, the evaluator’s docstring called it “a safe Jinja2 subset”. Reports #5071, #5072 and #5073 show expressions it answered differently from Jinja2, or differently from its own other operators.
This page belongs to the Spec Kit study, which lists all 28 reports.
FindingWith n = 50, the condition 0 <= n <= 10 came out true, and the workflow took the wrong branch without an error.
The evaluator split the expression at one operator and read the rest as a single operand, so it compared 0 with a true-or-false value.
Validation accepted the condition, and the run reported success.
A grammar has since replaced the parser. The documentation now says comparisons group left to right.
StudyThe reports’ recorded outputs, compared with what Jinja2 gives, then the upstream changes.
Each expression here comes from a report’s reproduction, with the output it recorded at Spec Kit commit 27d14e2.
The study has not run these expressions on Spec Kit’s main branch. An AI agent read main’s source, and that reading is marked Not yet reached.
Questions
Each shows a short answer and the evidence for it.
Evidence strength
- Proved
- Tested
- Decided by Harry
- Assumed
- Not yet reached
Evidence
Finding
What went wrong?
Some conditions gave a wrong answer instead of an error. 0 <= n <= 10 was true for n = 50. 1e3 was read as a missing variable.
Separately, == did not treat the text "10" as the number 10, while >= and <= did.
A workflow using such a condition took the wrong branch, and validation did not warn.
How do we know?
Each report has a reproduction script, and the output it gave at commit 27d14e2. Two agents ran each script in the session that found it, and Harry re-ran every one before filing.
The Jinja2 column is what Jinja2 and Python give for the same expression. Spec Kit’s own docstring named Jinja2 as the model. This website did not run either.
The expressions from the reports
.
| Report | Expression | Spec Kit gave | Jinja2 gives | Note |
|---|---|---|---|---|
| #5071 (opens in a new tab) | {{ 0 <= inputs.n <= 10 }}n = 50 | True | False | Read as 0 <= (inputs.n <= 10). An if step with this condition ran its then branch, and validate_workflow reported no errors. |
| #5071 (opens in a new tab) | {{ 1 < inputs.n < 10 }}n = 5 | False | True | The same split, the other way round. |
| #5072 (opens in a new tab) | {{ steps.count.output.stdout == 10 }}stdout = "10" | False | False | The same value was both >= 10 and <= 10. Jinja2 agrees with == here; the report is about the inconsistency inside Spec Kit. |
| #5073 (opens in a new tab) | {{ 1e3 }} | None | 1000.0 | The literal was looked up as a variable. |
| #5073 (opens in a new tab) | {{ inputs.name == 'it\'s' }}name = "it's" | False | True | An escaped quote was not recognised as part of the string. |
| #5073 (opens in a new tab) | {{ steps.plan.output.matrix[0][1] }}matrix = [["a", "b"], ["c", "d"]] | None | 'b' | Only one index was accepted per path segment. |
| #5073 (opens in a new tab) | {{ 'x'in inputs.a }}a = "x" | False | True | Word operators were found only with a space on both sides. |
The Spec Kit column is the output recorded in each report. The Jinja2 column is what Jinja2 and Python give for the same expression; this page did not run them. The module docstring at the filing commit describes the evaluator as “a safe Jinja2 subset”. Read the docstring at the filing commit (opens in a new tab)
Upstream
What changed upstream?
A Lark grammar replaced the hand-written parser on 2 November 2026. The documentation now says comparisons share one level and group left to right.
For #5072, the maintainers made == treat numeric text as numbers, as the ordering operators already did.
A fix for #5071 is open. The fix proposed for #5073 was closed without merging.
Upstream changes
.
- 2026-10-26 and 2026-10-28
- #5114 (opens in a new tab) split the evaluator into a package. #5130 (opens in a new tab) added tests that pin its existing behaviour, citing #5071 (opens in a new tab) and #5073 (opens in a new tab).
- 2026-11-02
- #5171 (opens in a new tab) replaced the hand-written parser with a Lark grammar. The documentation now says all comparisons share one level and group left to right, so a == b > 0 means (a == b) > 0.
- 2026-10-23 and 2026-11-09
- On #5072 (opens in a new tab) a maintainer first wrote that Python semantics, with no coercion of numeric strings, were the intended contract. #5310 (opens in a new tab) then made == and != coerce numeric strings, as the ordering operators do, and the maintainer closed the issue.
- 2026-11-10
- #5203 (opens in a new tab), which would evaluate chained comparisons pairwise, is open. #5206 (opens in a new tab), aimed at #5073 (opens in a new tab), was closed without merging.
Main
Is it fixed on main?
Not known. Nobody has run the reproductions on main for this study.
A reading of main’s source suggests a mixed picture. Some cases now raise a syntax error, and two now evaluate correctly. 0 <= 50 <= 10 still looks true.
That reading is not evidence until a reproduction runs.
A reading of main
.
| Report | Case | What the source suggests |
|---|---|---|
| #5071 (opens in a new tab) | 0 <= inputs.n <= 10 with n = 50 | Groups as (0 <= 50) <= 10, which is True <= 10. The coercion helper treats True as a number, so the result still looks true. The documentation now describes left-to-right grouping. |
| #5072 (opens in a new tab) | "10" == 10 | Now coerces both sides, so it looks true. That matches Spec Kit’s ordering operators and differs from Jinja2 and Python. |
| #5073 (opens in a new tab) | 1e3, 'it\'s' and matrix[0][1] | No longer match the grammar’s number, string or path terminals, so they look like syntax errors instead of None. |
| #5073 (opens in a new tab) | 'x'in inputs.a and inputs.a=='x'and inputs.b | The operator terminals accept a quote before the word, so these look correctly evaluated. |
| #5073 (opens in a new tab) | "1e3" > 100 | The coercion helper still reads “1e3” as text, so the comparison looks silently false. |
An AI agent read the grammar and operators on main at 1ed587a0 while preparing this page. It did not run them. A reading is not evidence, so every row here is Not yet reached until a reproduction runs on main.
Limits
What wasn’t checked?
No reproduction ran on main, and no generated inputs were compared with Jinja2.
Spec Kit now documents some departures from Jinja2 on purpose, so Jinja2 alone is no longer a correct reference.
Not yet reached
.
| Behaviour | Blocking property | Open question |
|---|---|---|
| Constraints on strings, paths and the shape of dictionaries, which most of Spec Kit’s defects involve | CGV checks type, nullability and choice requirements, and has no requirement kind for these constraints. | Which requirement kinds would let CGV check string, path and dictionary-shape constraints, and can their checks be proved sound like the existing ones? |
| A proof or reference model of any Spec Kit component | Dafny and Lean run in Docker, and the container that ran the study had no Docker daemon. | Is the expression evaluator, the one component with a stated model in Jinja2, specified precisely enough to model in Lean? |
| Whether each defect still reproduces on Spec Kit’s main branch | Running Spec Kit’s code needs an environment approved for third-party code, and the session that built this page had none. | Should the replay run in the same sandbox as Crosscheck’s Dafny and Lean containers, so third-party code never runs on the host? |
| A differential test of the expression evaluator against Jinja2 | Spec Kit now documents deliberate departures from Jinja2, such as left-to-right comparison grouping and numeric-string equality, so Jinja2 alone is no longer a correct reference. | Which Jinja2 behaviours does Spec Kit intend to keep, and can they be written down as a reference model to test against? |
| Whether the defects matter to users | The study has no data on how often the affected commands and inputs are used. | How should a finding’s severity be estimated when the project publishes no usage data? |