GarvitPrakash

01 — Framing

Fluid vs. crystallized

Reasoning benchmarks often conflate crystallized knowledge (g_c: interpolating patterns already in the training data) with fluid intelligence (g_f: inducing a new rule under genuine distribution shift). ConceptARC isolates the second: abstract grid tasks built on Core Knowledge priors such as objectness, containment and directionality.

02 — Four paradigms, one compute budget

What gets compared

1. P0Native chain-of-thought. Single-shot reasoning, answer emitted as a grid. No verification.
2. P1Program synthesis. The model writes a Python function, executed in a sandbox.
3. P2Iterative self-refinement. Self-critique against the demonstration pairs, over a fixed round budget.
4. P3Iterative program repair. Run the demonstrations as unit tests and repair from the stack trace. In progress

03 — Mechanism

Why latent reasoning breaks on grids

The paper argues the gap is structural, not prompt engineering. A spatial serialization bottleneck: flattening a 2D grid into 1D tokens pushes vertical neighbours W tokens apart. And open-loop errors: one wrong token poisons every later step, while a program is checked against the demonstrations in a closed loop.

04 — A new metric

Spatial Invariance Ratio

ρ_OOD =Accuracy(S_OOD)Accuracy(S_in)

Does an inferred rule survive when the grid grows past 20 × 20? Near 1 means the rule really was learned; near 0 means the skill was scale-specific pattern matching.

05 — Design

Controlled ablation

• Five ConceptARC categories: CenterInside / OutsideExtract ObjectsDirectionalitySame / Different
• A 4 × 2 factorial: execution arm × grid-scale regime (8 runs)
• Fixed budget of 16k tokens per task instance
• Metrics: exact match, pixel accuracy, ρ_OOD
• Generated code runs in an isolated subprocess with a wall-clock timeout and no network

06 — Status

Where it stands

• In-distribution runs, P0–P2
• Out-of-distribution scale runs (> 20 × 20)
• P3, iterative program repair

Results will be published with the preprint, not before.