SkillOpt on skillmake.xyz

01What SkillOpt is

A SKILL.md is the markdown file an AI agent reads to learn how to do one specific task — install a library, refactor a file, render a video. The agent loads it on demand, treats it as authoritative, and behaves accordingly. skillmake.xyz is the public marketplace where these skills live; each one has an install URL like skillmake.xyz/i/superpowers that returns the raw markdown so any agent can fetch it.

The SkillOpt paper (Wang et al., arXiv 2605.23904) makes one simple move: treat the SKILL.md as a trainable parameter. The model stays frozen — we never fine-tune Claude or Cursor — but the markdown gets optimized over time the same way a neural net's weights would. Bounded edits per step. Held-out validation gate. Strict-improvement promotion. Sections you mark as "do not touch" are byte-locked against the edit machinery.

That framing matters because "self-improving agents" without those guardrails mostly produce slop. SkillOpt is the discipline that turns a freeform edit this skill button into a stable optimization loop.

Frozen model plus trained context window is the practical adaptation. We are not training Claude. We are training the document Claude reads.

02Three FAQs

What is a SKILL.md and what is skillmake.xyz?

A SKILL.md is a plain-text recipe an agent loads to do one job. skillmake.xyz hosts a curated marketplace of them — 94 skills today across engineers, creators, devops, design, AI, marketing, and general audiences. Each skill has a stable install URL (/i/<name>) that returns the raw markdown so any agent — Claude Code, Cursor, Codex, Aider, Cline — can install it in one curl. The site itself is a Next.js app on Cloudflare Workers (via OpenNext), with KV for storage and Analytics Engine for usage telemetry.

Are we collecting usage data?

Yes — every install hit, page view, and click writes a row to Cloudflare Workers Analytics Engine. No personally identifying info is stored. The daily visitor id is a SHA-256 of ip + ua + day truncated to 8 bytes — same person on the same day collapses to one id, but cannot be tracked across days. Search queries are hashed before write. There are 14 MetricEvent types in src/lib/metrics-core.ts:

What can go wrong?

Three failure modes are real enough that we built explicit mitigations:

• Cache poisoning the A/B split. Cloudflare's edge cache can pin a candidate or current response, breaking the per-visitor route. Mitigation: while a candidate exists for a skill, /i/<name> flips to Cache-Control: private, no-store. After promotion, headers restore to public, max-age=300, must-revalidate.
• Low-traffic skills never decide. A long-tail skill might never cross 200 installs per arm. Mitigation: any candidate that fails to reach the threshold in 14 days routes to an LLM-judge fallback against the skill-judge peer skill — it must score strictly higher than the current before promotion.
• Optimizer loops. The model proposing edits can get stuck in a local minimum, re-suggesting near-duplicate diffs that keep getting rejected. Mitigation: rejected candidates write to a negative-feedback store at skill:<id>:rejected:v<hash>. The optimizer prompt includes the last N rejections so it doesn't re-propose them.

03Glossary

04SkillOpt figure mapping

The paper's Figure 1 walks through the gradient-descent analogy. Here is the exact translation we use on skillmake:

05The four phases

Phase 1 — Foundation live

Versioned KV storage, bounded edits, static gate, protected sections, per-version Grafana panels.

• Versioned storage: skill:<id> per-version rows, plus family pointers family:<name>:current, family:<name>:candidate, family:<name>:history. Old reads keep working — the migration is additive.
• Version binding on install: install_hit.blob2 = name@hash8. Existing dashboards keep working via the splitByChar('@', blob2)[1] wrapper.
• Bounded-edit endpoint: structured SkillEdit[] payload, 9 op kinds, ≤ 8 ops per candidate.
• Static gate: 5 checks run synchronously on every candidate (see §09).
• Protected sections: byte-identity enforced at both schema and rendered-markdown level.

Phase 2 — A/B routing pending

Deterministic per-visitor split at /i/<name>:

const candidate = await getCandidateSkillByName(name);
const useCandidate = candidate && shouldRouteCandidate(req, candidate.abTrafficShare);
const served = useCandidate ? candidate : current;

shouldRouteCandidate hashes (cf-connecting-ip + name), so the same visitor always sees the same arm. Default share: 20% candidate / 80% current. While a candidate exists for a skill, cache headers flip to private, no-store.

Phase 3 — Dynamic validation gate cron · hourly

Hourly Cloudflare Cron at /api/cron/validate-candidates reads Analytics Engine for the last 7 days. For each candidate: query marketplace_view count (route-agnostic, shared denominator), query install_hit per arm, require ≥ 200 installs per arm, Welch's t-test on conversion ratios with p < 0.05 AND strict improvement. Pass → promoteCandidate(name) atomically. Regress → retireCandidate(name) + write the diff to the negative-feedback store.

Phase 4 — Optimizer loop cron · weekly

Weekly cron /api/cron/propose-edits: find approved skills below the 25th percentile in conversion with no active candidate, pull 14 days of telemetry + last N rejected candidates, render to a narrow prompt, require a Zod-validated SkillEdit[] of length 4–8, apply → static gate → save as candidate. An SKILLOPT_ENABLED=true|false env var gates every cron — kill switch is one flip.

06A/B evaluation playbook — 7 steps

Once Phase 2 lands, here is exactly what running an experiment looks like from the operator side.

07Live numbers — last 24h

Pulled live from Cloudflare Analytics Engine SQL API on 2026-05-27 against the skillmake_metrics dataset.

The 6.1% versioned share is the rate at which install rows now carry the name@hash8 suffix. That number will climb as edge caches expire and older clients fetch fresh markdown — by the time Phase 2 lands and we actually run a real candidate, every install row will be versioned.

Top 5 skills by install count (last 24h)

1. superpowers83
2. claude-api77
3. anthropic-frontend-design77
4. mcp-builder77
5. karpathy-claude-md72

Top 3 countries (last 24h)

1. United States · US611
2. India · IN607
3. United Arab Emirates · AE511

Deploy verification

The first row that landed in Analytics Engine right after the Phase 1b deploy proved the slug encoder was wired correctly end to end:

blob1: "install_hit"
blob2: "github-actions-docs@cf993479"   // slug@hash8 — Phase 1b live
blob3: "US"
blob5: "curl"

Before Phase 1b, blob2 would have been the bare slug github-actions-docs. The @cf993479 suffix is the first 8 hex chars of the rendered markdown's SHA-256 — exactly what encodeVersionedSlug produces (§11).

08The 9 SkillEdit ops

Each op is atomic and reversible. The edit budget (MAX_EDITS_PER_STEP = 8) is the textual analog of a learning rate — remove the cap and the optimizer destabilizes. Forbidden on this surface (would require a new submission, not an edit): replace_name (breaks /i/<name>), replace_apiReference.signature (changes the contract), replace_category, replace_audience (both change marketplace placement).

The Zod schema (verbatim from src/lib/skill-edit.ts):

export const SkillEditSchema = z.discriminatedUnion("op", [
  z.object({ op: z.literal("add_gotcha"), value: z.string().min(8).max(280) }),
  z.object({ op: z.literal("delete_gotcha"), index: z.number().int().nonnegative() }),
  z.object({
    op: z.literal("replace_gotcha"),
    index: z.number().int().nonnegative(),
    value: z.string().min(8).max(280),
  }),
  z.object({
    op: z.literal("add_keyConcept"),
    value: z.object({
      term: z.string().min(2).max(80),
      explanation: z.string().min(10).max(500),
    }),
  }),
  z.object({
    op: z.literal("replace_keyConcept"),
    index: z.number().int().nonnegative(),
    value: z.object({
      term: z.string().min(2).max(80),
      explanation: z.string().min(10).max(500),
    }),
  }),
  z.object({ op: z.literal("delete_keyConcept"), index: z.number().int().nonnegative() }),
  z.object({ op: z.literal("replace_description"), value: z.string().min(20).max(220) }),
  z.object({
    op: z.literal("replace_whenToUse_item"),
    index: z.number().int().nonnegative(),
    value: z.string().min(8).max(200),
  }),
  z.object({
    op: z.literal("replace_apiReference_example"),
    index: z.number().int().nonnegative(),
    value: z.string().max(2400),
  }),
]);

export const SkillEditsPayload = z
  .array(SkillEditSchema)
  .min(1, "at least one edit required")
  .max(MAX_EDITS_PER_STEP, `bounded to 8 ops per step (learning rate)`);

09The 5 validator checks

Runs synchronously on every candidate insert. Returns { pass, reasons, warnings, tokenCount }. A fail can still be force-merged by a curator (the override is logged), matching the paper's researcher-override pattern. From src/lib/skill-validator.ts:

The token estimator is intentionally cheap — chars/4, accurate to ~15% — fast enough to run on every candidate without an extra dependency:

function estimateTokens(markdown: string): number {
  // chars/4 ≈ tokens. Cheap, close enough for gating.
  return Math.ceil(markdown.length / 4);
}

And the protected-section enforcement (verbatim, the part that matters):

// 6. Protected-section byte-identity at the rendered-markdown level.
if (protectedIds.size > 0) {
  const parentBlocks = extractProtectedBlocks(parent.markdown);
  const candidateBlocks = extractProtectedBlocks(candidate.markdown);
  for (const id of protectedIds) {
    if (parentBlocks.get(id) !== candidateBlocks.get(id)) {
      reasons.push(
        `protected_byte_identity: section '${id}' bytes differ between parent and candidate.`
      );
    }
  }
}

10KV layout

The Phase 1a migration is additive — old reads keep working through the legacy findApprovedByName path. New writes also populate the family namespace.

Read paths (from src/lib/storage.ts):

• getCurrentSkillByName(name) — reads family:<name>:current, falls back to legacy name scan if no pointer exists.
• getCandidateSkillByName(name) — reads family:<name>:candidate, returns null if no A/B is in flight.
• listVersionsByName(name) — reads the history pointer, returns every recorded version oldest-first.

Write paths:

• saveSkillVersion(parent, edits, input) — writes a new skill:<id> with versionStatus: "candidate", appends to history, sets the candidate pointer.
• promoteCandidate(name) — flips candidate → current atomically, demotes previous current to retired, clears the candidate pointer.
• retireCandidate(name) — marks candidate as retired and clears the candidate pointer. The row stays queryable for audit and negative-feedback reuse.
• ensureFamilyIndexed(entry) — lazy migration: the first read of a legacy entry repairs the family index in place. No separate migration job.

11The slug@hash encoding

Every install_hit row's blob2 column carries <skill-name>@<contentHash[:8]>. The Analytics Engine schema is unchanged; the version is encoded inside the existing column by convention. Old slug-only dashboards still work via splitByChar('@', blob2)[1] — which returns the whole string on legacy rows written before versioning, so old data attributes correctly. From src/lib/metrics-core.ts:

export function encodeVersionedSlug(name: string, contentHash: string): string {
  return `${name}@${contentHash.slice(0, 8)}`;
}

export function parseVersionedSlug(slug: string): { name: string; versionHash: string | null } {
  const at = slug.indexOf("@");
  if (at < 0) return { name: slug, versionHash: null };
  return { name: slug.slice(0, at), versionHash: slug.slice(at + 1) };
}

Round-trip property: parseVersionedSlug(encodeVersionedSlug(name, hash)).versionHash === hash.slice(0, 8). Legacy slugs round-trip too — parseVersionedSlug("legacy-skill").versionHash === null. The metrics.test.ts suite covers both directions and the legacy fallback (7 cases, all green).

12Protected sections in action

Rendered SKILL.md output wraps protected sections in HTML-comment markers the validator can find with a regex. Here is what a candidate of the caveman skill would render if protectedSections: ["gotchas", "keyConcepts"] were set (synthesized from scripts/seeds/caveman.json via the renderSkillMarkdown logic in src/lib/skill-schema.ts):

---
name: caveman
description: "Use when you want Claude Code (or 30+ agents) to cut ~75% of output tokens..."
source: https://github.com/JuliusBrussee/caveman/blob/main/README.md
generated: 2026-05-27T12:00:00Z
category: tool
audience: engineers
---

<!-- @protected:keyConcepts -->
## Key concepts

### talk like caveman

Skill instructs the agent to drop filler words, hedging, and pleasantries —
keep substance in telegraphic fragments. Same fix, ~75% fewer output tokens,
100% technical accuracy. Brain still big, mouth small.

### four grunt levels

/caveman accepts a level argument: `lite`, `full`, `ultra`, `wenyan`.
Level persists until end of session or `normal mode`.
<!-- /@protected:keyConcepts -->

## API reference

```
One-line install (Mac / Linux / WSL / Git Bash)
```
Finds every agent on the machine and installs the caveman skill into each.

<!-- @protected:gotchas -->
## Gotchas

- Caveman mode persists for the whole session unless you say `normal mode`
- /caveman-compress is destructive — back up CLAUDE.md before running
- ULTRA level loses some nuance — fine for code, risky for prose
<!-- /@protected:gotchas -->

const re = /<!-- @protected:([a-zA-Z]+) -->\n?([\s\S]*?)\n?<!-- \/@protected:\1 -->/g;

13SQL recipes

All three queries run against the Cloudflare Analytics Engine SQL API. Full recipe collection in docs/analytics-queries.md. Analytics Engine samples on write, so always use sum(_sample_interval) — not count() — to get the true total.

All-time install leaderboard (with the splitByChar wrapper)

SELECT
  splitByChar('@', blob2)[1] AS slug,
  sum(_sample_interval) AS installs
FROM skillmake_metrics
WHERE index1 = 'install_hit' AND blob2 != ''
GROUP BY slug
ORDER BY installs DESC
LIMIT 50

Active A/B candidates (versions_seen > 1)

SELECT
  splitByChar('@', blob2)[1] AS slug,
  count(DISTINCT splitByChar('@', blob2)[2]) AS versions_seen,
  sum(_sample_interval) AS total_installs
FROM skillmake_metrics
WHERE index1 = 'install_hit'
  AND blob2 LIKE '%@%'
  AND timestamp >= NOW() - INTERVAL '14' DAY
GROUP BY slug
HAVING versions_seen > 1
ORDER BY total_installs DESC

When this returns rows, a candidate is live in production. versions_seen = 2 is a clean current+candidate A/B; 3+ means a candidate was promoted and a new candidate is already being tested against it.

Per-version conversion ratio for one skill (caveman example)

WITH
  installs AS (
    SELECT
      splitByChar('@', blob2)[2] AS version_hash,
      sum(_sample_interval) AS installs
    FROM skillmake_metrics
    WHERE index1 = 'install_hit'
      AND splitByChar('@', blob2)[1] = 'caveman'
      AND timestamp >= NOW() - INTERVAL '7' DAY
    GROUP BY version_hash
  ),
  views AS (
    SELECT sum(_sample_interval) AS views
    FROM skillmake_metrics
    WHERE index1 = 'marketplace_view'
      AND blob2 = 'caveman'
      AND timestamp >= NOW() - INTERVAL '7' DAY
  )
SELECT
  installs.version_hash AS version_hash,
  installs.installs AS installs,
  views.views AS detail_page_views,
  round(installs.installs / views.views, 3) AS conversion_ratio
FROM installs CROSS JOIN views
ORDER BY conversion_ratio DESC

marketplace_view is route-agnostic (same slug for both arms), so the denominator is shared. This is the exact ratio the Phase 3 cron runs Welch's t-test against.

14Test summary

All 25 tests green by suite. Vitest, no integration mocks needed — the libs are pure.

15Commit ledger

Phase 1 shipped in four commits on main:

16Deploy state

17What's next

Phase 2 — A/B routing. The install route at /i/<name> needs to call getCandidateSkillByName in parallel with the current lookup, hash (cf-connecting-ip + name) to assign an arm, and flip cache headers to private, no-store for the duration of any in-flight A/B. The deterministic hash means visitors never bounce between arms, even across reloads or fresh tabs — the only thing that resets is when the candidate gets promoted or retired.

Phase 3 — dynamic validation gate. Hourly cron at /api/cron/validate-candidates reads the last 7 days of Analytics Engine, computes the conversion ratio per arm, runs Welch's t-test, and either calls promoteCandidate(name) or retireCandidate(name). The long-tail fallback to the skill-judge LLM-judge skill kicks in at 14 days without crossing the 200-installs-per-arm threshold. The cron is gated by SKILLOPT_ENABLED — one env flip kills the whole loop.

Phase 4 — optimizer loop. Weekly cron at /api/cron/propose-edits picks the bottom-quartile skills by conversion, pulls 14 days of telemetry, reads the rejected-edits store for that skill, and prompts a frontier model for a Zod-validated SkillEdit[] of length 4–8. Apply → static gate → save as candidate → enters A/B at 20% share. Budget at current corpus size is roughly $10/month.