Asset Forge

01 The Approach

LLMs output structured drawing commands. A server-side rasterizer turns those commands into indexed-color pixel grids.

The naive approach — asking an LLM to output raw pixel arrays — produces incoherent results. Models struggle with spatial reasoning at the individual pixel level. But shape primitives are a different story: rectangles, circles, and polygons are the kind of structured, compositional output that language models handle well.

Asset Forge gives the model a fixed palette and a canvas size, then asks it to compose a sprite from geometric primitives. The server rasterizes the commands into a pixel grid, producing genuine pixel art with hard edges and an indexed color palette.

Drawing Primitives

02 The Evaluation Harness

A calibrated LLM judge (Opus evaluating Sonnet's output) scoring across six quality dimensions, validated against human grading.

Automated evaluation of generative art is hard. Subjective quality, prompt fidelity, and pixel-art-specific conventions all matter. We built a judge that scores each sprite on a 1–5 Likert scale across six dimensions, then calibrated its rubrics against blind human scoring over five rounds.

Quality Dimensions

Each dimension has explicit anchor descriptions at every score level, so both the LLM judge and human graders share a common rubric. Component Separation, for example, distinguishes between multi-part subjects (where parts should be visually distinct) and simple subjects (where cohesion is the goal).

Calibration Journey

We ran five rounds of human grading, tightening rubrics and changing the judge's inputs after each round. The key breakthrough was giving the judge vision — both the rendered PNG and an ASCII grid representation — which dramatically improved Spatial Coverage and Prompt Adherence scoring.

Calibration Data by Round

MAD = Mean Absolute Deviation between LLM judge scores and blind human scores (lower is better). Color coding: green ≤ 0.65, gold ≤ 1.10, red > 1.10.

Sprites Used for Judge Calibration

The same prompts generated fresh each round. The sprites differ because LLM output varies, but the generation prompt stayed the same. What changed was the judge.

Calibration Metrics Across Rounds

MAD = Mean Absolute Difference (lower is better). Agreement = % of scores within 1 point of human.

Dimension	Round 1 Baseline	Round 2 Rubric rewrites	Round 3 Model separation	Round 4 Vision + preChecks	Round 5 LLM spatial coverage, 30 prompts
Comp Sep	1.40	0.70	1.00	0.50	0.93
Color	1.00	0.80	0.60	0.56	0.71
Detail	1.00	0.80	0.90	0.39	1.04
Spatial	0.20	1.20	0.70	0.61	0.61
Pixel Art	-	-	-	-	0.89
Prompt	1.30	0.80	1.60	1.00	1.32

03 What Worked

Tiles, simple objects, and the calibration methodology itself were clear successes.

• Tiles and simple objects score well — human scores of 4.0–4.8 out of 5 for subjects like boulders, crates, floor tiles, and simple vehicles. The drawing-command approach handles these naturally.
• The calibration methodology converged — five rounds of blind human scoring, rubric rewriting, model separation, and adding vision brought overall MAD from 0.98 down to 0.61.
• Spatial Coverage transformed — the worst-calibrated dimension (MAD 0.95 under code-based formula) became the best (MAD 0.61) once moved to LLM-judged with vision input.
• The eval harness is reusable infrastructure — prompt sets, multi-shot generation, automated judging with rubric anchors, human grading UI, and calibration statistics are all general-purpose tools that work regardless of the generation approach.

Round 5: Best and Worst Sprites

The latest round with 30 prompts, expanded categories, and LLM-judged spatial coverage.

Top Scoring

Lowest Scoring

04 What Didn't Work

Generation prompt iteration made things worse. Complex subjects hit a quality ceiling.

• Prompt iteration backfired — v2 of the generation prompt scored −0.16 vs. baseline, v3 scored −0.45. More prescriptive instructions constrain the model and reduce quality.
• Complex subjects produce recognizable blobs — dragons, knights, and detailed vehicles have too many components to resolve clearly at 32–64px with geometric primitives.
• Prompt Adherence remains hardest to calibrate — MAD 1.32 on the latest full round. The judge tends to be more generous than humans about whether a sprite "looks like" the requested subject.
• The drawing-commands approach may have a ceiling — geometric primitives produce clean, structured output but lack the organic detail needed for complex subjects.

Generation Prompt Version Comparison

Human-graded mean scores (1–5 Likert scale). Each version evaluated on the same prompt set. v2 added detail emphasis and stronger instructions; v3 added per-category guidance and lighter perspective rules.

Side-by-Side Comparison

The same prompts rendered by each generation prompt version. Human scores (1–5) shown below each sprite.

05 Where It's Going

The eval infrastructure is solid. The generation approach may need to change.

The drawing-commands approach works well for tiles and simple objects but hits a quality ceiling for complex subjects. One alternative under consideration: generate a high-resolution image with an image-generation model, then use custom tooling to convert it down to pixel art with proper palette constraints and hard edges.

Regardless of the generation approach, the evaluation harness and calibrated judge remain valuable. The rubrics, prompt sets, human grading UI, and calibration methodology are general-purpose infrastructure for assessing pixel art quality.