AI evaluation and governance

Proving an AI feature is safe to ship.

Most teams shipping features built on language models can tell you which model they chose. Far fewer can tell you what would have stopped a bad answer from reaching someone. I build that gate: a merge-blocking adversarial evaluation suite, golden-answer harnesses, and grounding assertions that fail a release rather than let an unsupported answer through.

The discipline matters more than the tooling. A quality number that cannot fail a build is a description, not a control. Before trusting a new gate I break something on purpose and confirm the gate catches it, because a check that has never failed is not evidence of health.

Useful when a feature is in pilot and there is no defensible answer to how you know it is safe to ship.

Public data quality and integrity

A rating you can defend line by line.

Public data misleads people in predictable ways, and the most common is treating a value that is missing as a value that is zero. I build systems that measure what institutions actually disclose, publish the methodology next to the result, and state their own coverage gaps rather than hiding them.

disclosed grades US higher-education institutions on what they disclose rather than how they perform, across two federal datasets, with applicability rules that keep the denominators honest. Afterward joins 3,266 California training programs to their federally reported outcomes; roughly a third publish no outcome data at all, and the tool says so on the page.

Useful when you publish data other people make decisions with, and the party you rate will argue with the rating.

Standards and interoperability

Conformance checked against the spec, not against opinion.

I am a registered member of the Credential Engine CTDL Advisory Group, and I did the 2023 data modeling that began a state's migration of its training program registry to Credential Engine's CTDL standard; those programs are live in the Credential Engine Registry today.

I build deterministic validators that check conformance against cited specification requirements. fhir-scorecard does that for publicly observable health API discovery surfaces, with spec-cited findings and capability drift tracked between runs.

Useful when you are adopting an open standard and need to know whether your data actually conforms.

Engineering leadership, fractional or interim

Engineers, and the practice around them.

Manager of managers across application engineering, data, and platform operations. I have built functions that did not exist: test coverage from zero to the mid-nineties, a security backlog cut 94 percent, a CI platform replaced, and intake and release management turned from habits into processes.

I have also run a zero-downtime migration between cloud providers with more than 200 schema changes and no service disruption, because the organization could not afford an outage and I do not think it should have had to.

How it works: one or two days a week on an ongoing basis, or most of a week for a fixed interim term when you are between leaders or mid-rebuild. Three-month minimum either way. You get someone accountable for the technical function, not a developer seat: strategy and roadmap, the team and its standards, delivery cadence, vendor oversight, security posture, hiring, and where AI belongs in the product and in how the team builds. Engagements end with a named handoff, usually to a permanent leader I help you hire.

Useful when you have engineers and no engineering practice, or a gap at the top of a team for a defined period. Start with a thirty-minute conversation.

How I work

  • In your repository, under your conventions. I am accountable for reviewed, merged code, not for a document describing what someone else should build.
  • Measured before and after. Engagements begin by establishing what is true now, because a change without a baseline is an opinion, and end with a number you can check.
  • Handed off deliberately. Every phase closes with the work owned by someone on your team. If at the end only I can safely modify a subsystem, I have done the job badly.
  • Corrections in public. I have published corrections to my own live figures, in the changelog, next to the original claim.

For California public agencies

I am a California Certified Small Business (DGS certification 2054235, sole proprietorship, Davis, CA). State and local agencies can engage me through the SB/DVBE Option, which allows direct award of contracts from $5,000 to $249,999 on the basis of two small-business quotes (Government Code 14838.5), and my bids carry the 5 percent small-business preference in competitive solicitations.

NAICS 541511, 541512, 541519, and 518210. Registered on Cal eProcure with a full UNSPSC commodity profile.

Download my capability statement (PDF)

Fit

Public interest and civic technology, education and workforce, energy and climate, standards bodies, and nonprofits building data infrastructure.

I do not take defense, carceral, enforcement, or surveillance work, and I do not work for federal agencies as an end customer.

Get in touch