AI career guidance, accountable to evidence

Two different artifacts are easy to collapse into one AI claim. I independently designed and shipped a shared Bedrock safety layer (opens in a new tab) into MyCareer.NJ.gov's production codebase (opens in a new tab). I also designed and built the largest resident-facing proof of concept on top of it: a grounded career coach. That coach remained unmerged, behind disabled feature flags and pilot approval. No resident used it, and I attribute no employment or training outcome to it.

What shipped

The production foundation includes a reused Bedrock client, circuit breakers, input-budget guards, fourteen named prompt-injection pattern categories (opens in a new tab), safe parsing for malformed model output, pinned model identifiers, per-feature flags, a master kill switch, and enforced module boundaries. The shared TypeScript layer carried 761 lines of tests across five files; a separate Python Bedrock wrapper with its own circuit breaker and input guard also reached the Career Navigator main branch. Shipping the control plane first meant experiments inherited a common stop mechanism instead of inventing safety at the feature edge.

What stayed behind the pilot gate

The unlaunched coach was a bilingual RAG system over New Jersey's approved training, occupation, wage, funding, pathway, and support-service data (opens in a new tab). An intent classifier routed training, funding, skill-gap, crisis-resource, and returning-citizen questions into different query paths. More than thirty tools supplied structured context. Simple lookups went to Claude Haiku while genuine coaching used Claude Sonnet, preserving an intentional roughly twelve-fold model-cost difference. Streaming, page context, session controls, token ceilings, and a monthly kill switch were designed in; both resident-facing activation flags remained off.

Grounding had to survive identifiers

A real defect made the grounding lesson concrete: an answer could name the right occupation and still construct the wrong SOC link (opens in a new tab). I changed the context contract so every reference arrived as an exact copyable Title (SOC) pair and added an output assertion that demoted links not grounded in retrieved context. Across four validation runs after the fix, all 26 occupation links used supplied identifiers and none hallucinated a SOC code. The durable rule was architectural: supply identifiers; never ask the model to infer them.

The evaluation system was larger than one score

The POC spanned roughly 191 files, 47,500 added lines, and 34 test files. Its merge-blocking safety gate covered about 79 bilingual adversarial cases across prompt override, role manipulation, jailbreaks, prompt leakage, code execution, and obfuscation, plus a 19-message legitimate allow-list to catch over-refusal. A 210-entry known-answer set, an enumerated 306-unit-test plan, and a 390-conversation QA harness covered prompt, matrix, grounding, crisis-routing, refusal, and bias behavior. Those are designed test assets. The executed Bedrock runs below are the runtime evidence; the distinction matters.

Executed QA changed the release decision

The May 15, 2026 baseline executed 50 conversations and 202 turns: 36 percent of conversations and 89 percent of turns passed, with empty responses, missing program links, and only 3 percent average context retention. By May 29, the executed run covered 200 conversations and 810 turns; 168 conversations and 780 turns passed, or 84 and 96 percent. That is a meaningful engineering improvement. It is not shorthand for ready to ship.

What still blocked a pilot

The latest run still scored 70 percent on life-situation prompts, 60 percent on comparisons, 40 percent on page context, and 45 percent average context retention; it logged 22 missing-program-link and eight low-relevance issues. Spanish scored 100 percent in the automated category, but required human escalation before any public claim. Eight pilot-readiness gates defined block conditions across response quality, grounding, high-stakes English and Spanish routing, privacy, cost, reliability, accessibility, and operations. Six pre-launch audits were complete. The unresolved gates—not the best aggregate number—controlled the release decision.

One foundation, nineteen experiments

The shared layer supported six primary and thirteen secondary proofs of concept. The primary set covered the coach, semantic search, résumé and skill matching, predictive career intelligence, smart notifications, and content and accessibility automation. The secondary set explored credential stackability, learning paths, data freshness, program health, labor-market narratives, regional equity, open data, enrollment trends, and outcome metrics. This is portfolio breadth, not nineteen shipped products: the features remained open or draft pending pilot approval.

A time-bounded comparison, not a novelty claim

A fifty-state scan did not surface an equivalent resident-facing conversational coach grounded in state program records at the time. That finding helped orient the work; it does not prove that no comparable system existed, remained unpublished, or appeared later.

This prototype only makes sense on top of the statewide data and delivery system described in the MyCareer.NJ.gov platform case study. The right combined claim is precise: the platform served residents and published program evidence; the shared AI guardrails shipped; the career coach did not.