The score was the easy part

A transit feed can pass validation and still fail a rider.

Most riders will never open one. They meet it through a trip planner that offers a route, shows a departure, or says whether a stop or trip is marked for wheelchair access. The General Transit Feed Specification, usually called GTFS (opens in a new tab) is the shared format behind many of those answers. It packages stops, routes, schedules, fares, and accessibility fields for software to read. When that information is stale, incomplete, or ambiguous, the problem travels downstream to the rider.

I started GTFS Scorecard in June 2026 with two local systems, Unitrans and Yolobus, and what looked like a narrow engineering task. Run the standard's established validator. Turn its findings into a grade. Publish the result. Then I had to decide which findings mattered most, what a missing field meant, and what a transit manager who inherited a vendor feed should fix first.

Computing a grade turned out to be the easy part. The moment I assigned weights, translated machine notices into rider consequences, and ranked repairs, I was no longer just reporting data. I was making a public claim. That gap became the project.

The project grew beyond one grade page. It now produces scorecards, fix guidance, board briefs, program views, alerts, a pre-publish checker, a public API, data exports, a read-only data service for AI tools, and a reusable GitHub Action. Maintainers, program liaisons, boards, researchers, and developers can approach the same evidence from different seats. The harder product question is whether one finding can travel from public record, to respectful action, to a comparable recheck without changing the evidence underneath.

Project artifact

The public interface is a promise about the underlying evidence

Explore GTFS Scorecard (opens in a new tab)Inspect the source (opens in a new tab)

A validator can find an error. It cannot decide what matters

MobilityData already maintains the standard's canonical open-source validator (opens in a new tab). Reimplementing hundreds of structural rules would have created another source of disagreement while adding little value for the person maintaining a feed. I reused the shared rule engine and treated its output as source material, not as a finished public answer.

Scorecard preserves the validator version, notice code, severity, and number of examples. The project-authored work begins after that: explain the finding in plain language, connect it to a rider consequence, rank a small number of repairs, and keep enough evidence for someone else to challenge the interpretation. The grade describes published information. It does not grade the transit service, determine compliance, or certify physical accessibility.

The first product hypothesis was that three prioritized fixes would help a transit manager act. The scorecard, alerts, and public receipts can now support discovery and triage. The next hypothesis is harder: can an accepted action be traced to new bytes from the same intended feed and closed with comparable evidence? The last steps in the delivery line are the workflow the project is testing now, not a proven outcome.

A grade is an editorial decision with math attached

The current published rubric (opens in a new tab) gives correctness 35 percent, freshness 20 percent, rider experience 25 percent, and realtime 20 percent when all four can be measured. Those weights do not come from nature or from an official GTFS authority. I chose them. Accessibility fields receive substantial weight because missing information can remove a trip from someone’s set of usable choices. That is a product value expressed as arithmetic.

A missing realtime source is not a failing realtime source. Some require a key. Some operators do not publish realtime data at all. When that category cannot be measured, Scorecard recalculates the grade using only the categories it could measure instead of quietly inserting a zero. Not measured, not applicable, unknown, unreachable, and poor are different states.

That distinction travels well beyond transit. Missing wage data in a workforce directory is not zero wages. No recorded inspection result is not a passing inspection. An offline environmental sensor does not mean the air is clean. Civic software should carry uncertainty in its schema instead of erasing it in the presentation.

That is why every artifact stamps its scoring profile and validator version. The letter grade is a project-authored lens on published feed quality. It is not a legal finding, an official GTFS grade, a judgment about the agency’s service, or proof that a stop is physically accessible.

One evidence chain has to support many decisions

The project publishes one versioned source record, then lets the agency page, board brief, call-prep view, fix library, badge, public API, command line, and GitHub Action read it in different ways. A board member and a developer need different presentations, but the underlying claim should not change with the audience. That separation also let the project move future daily outputs out of Git and into object storage without teaching every public surface a new data model.

The architecture still had a memory. By July 8, daily snapshots had helped grow the repository’s Git data to 884 MB and its published artifact tree to 534 MB. I stopped committing future daily outputs, but I did not rewrite the cited history that tags, pull requests, commit identifiers, and this essay already depended on. The decision record (opens in a new tab) treats that history as evidence that must outlive an interface, not as disposable build debris.

Publishing a grade about a named public organization creates procedural duties too. The project’s listing policy (opens in a new tab) gives agencies a direct path to correct a name, source, or detail and to request removal. The project avoids public leaderboards and percentiles, keeps owner and ticket details private by default, and treats an unwelcome or incorrect listing as a product defect to fix, not an argument to win.

A repair rehearsal should be allowed to fail

Without agency access, I ran the narrowest honest experiment: copy the two public local feeds, apply only deterministic fixes, and rescore the copies. I did not fill accessibility fields, invent fares or contacts, or delete records. Those changes require an owner who knows what is true.

The first Yolobus correction failed its own verification. Scorecard recased route_long_name, but the validator notices were mostly attached to route_desc, so all 15 remained. I changed the recipe, regenerated the copy, and ran the validator again. The verified pass reduced the notices to two and raised correctness from 90.0 to 92.0. The failed first attempt belongs in the evidence because it changed the tool.

Scale exposed every hidden default

At small scale, a feed identifier can feel like a filename and a publication run can feel like one linear job. At larger scale, those shortcuts become public errors. One operator may publish bus and rail separately. A regional bundle may represent several operators. A source address can move while the public identity should persist. Several workers can each finish correctly and still combine their results into stale public data.

Neither failure was solved by retrying. The parallel workers had to publish only the records they had just produced. The duplicate directory needed one reviewed list to define what belonged in public. The resulting identity model (opens in a new tab) now separates the organization, the feed it publishes, the address used to retrieve it, and any old name or address. One URL is not one agency, and a storage directory is not a publication list.

Responsible coverage includes the decision not to grade

The first internationalization work was not translation. It was finding every place where I had mistaken the United States for a default. The shared GTFS core can operate across countries. National Transit Database and United States census measures cannot. Those became explicitly regional modules. Discovery became country-first, and weak regional coverage could no longer be allowed to look like poor feed performance.

The project’s coverage roadmap (opens in a new tab) also draws a line around central curation. A formal open feed with clear reuse terms can be reviewed from primary sources. Community-produced or informal transit data needs a named local steward, consent or reuse evidence, attribution, and a removal path before the project should ingest or grade it. Sometimes the responsible data decision is not to count.

For this essay, I reviewed the July 18 registry commit (opens in a new tab), which records 1,609 configured feed records tied to 1,590 organization identities across 33 countries. The preceding completed scoring run (opens in a new tab) had used the earlier 1,518-record snapshot and published 1,501 scorecards. The 91 newer records arrived after that run took its input. Those are dated operating counts. Configured is not published, a feed record is not always an agency, and a large registry is not a census of transit service.

The same July 18 work made the research burden concrete. A review of Japan's gtfs-data.jp listings (opens in a new tab) added 40 records. A later eighth European pass (opens in a new tab) added 51 more. What matters is what those totals hide: each candidate needed first-party reuse terms, a downloadable archive, current service, a size check, and a reviewed location. Coverage growth was not list ingestion. It was evidence work.

Language evidence needed the same discipline. A feed’s translations.txt can show that rider information is published in more than one language. It says nothing about whether people can use the Scorecard interface in that language. The reviewed lookup at /es/ (opens in a new tab) is a bounded Spanish-first surface, not permission to claim full interface localization. Measuring content and delivering language access are separate obligations.

The European beta gate makes the same choice in executable form. The configured set reached 251 reviewed feed records across 22 countries and crossed its numeric coverage thresholds. At 12:08 a.m. UTC on July 19, the preserved gate snapshot (opens in a new tab) still returned not_ready because the latest scored evidence and other release conditions had not caught up. A registry count can cross its threshold before the public evidence does. I would rather publish a gate that says “not ready” than infer readiness from configuration or make a launch claim I cannot defend.

The measuring stick is part of the evidence

A validator upgrade can change thousands of notices without any agency changing a byte. A rubric adjustment can move grades even when the source feed is identical. If the site draws a continuous trend line across those boundaries, it turns a methodology change into a false story about agency performance.

The comparison policy (opens in a new tab) now refuses to call two results a trend unless the feed identity and measuring method are comparable. The governed upgrade protocol (opens in a new tab) runs new rules beside the old ones on a small reviewed group and reports whose results would move before the new method takes effect. If I change the measuring stick, I need to show the consequences before I call it an upgrade.

Accessible data and an accessible product are separate obligations

A letter is easy to scan and easy to misuse. The scorecard is useful only if a maintainer can move from the letter to the category, from the category to a finding, and from the finding to a source-backed repair. The data behind the Unitrans receipt below is not a mockup. The custom presentation preserves the JSON, scoring profile, validator version, category math, and ranked fixes from a published run.

Published receipt

One scorecard, with the evidence still attached

Correctness

84.8

Freshness

100

Rider experience

60

Realtime

Not measured

Highest-impact fixes

1Stops do not state wheelchair boarding status296 records · 25 points

Populate wheelchair_boarding only from verified source data. An unknown value should remain unknown until the agency can support a claim.

2Trips do not state wheelchair accessibility3,071 records · 15 points

Populate wheelchair_accessible at the trip level from verified fleet and operations data. The scorecard does not infer it from route type.

3Published shapes are not used by any trip54 records · 8 points

Remove unused shapes or connect them to the intended trips. This finding comes from the canonical validator notice unused_shape.

Download the source JSONOpen the current scorecard (opens in a new tab)

This preserved snapshot is dated July 10, 2026. A score describes the published feed at a point in time. It does not certify service quality or physical accessibility.

The two largest opportunities concern wheelchair fields. That does not prove 296 stops or 3,071 trips are inaccessible. It proves the published feed does not state their status. A feed can state that a stop is accessible. It cannot certify that the stop is usable. The repair must start with verified operations data, not an inference designed to improve the grade.

The same distinction applies to the product itself. GTFS Scorecard's accessibility standard (opens in a new tab) targets WCAG 2.2 AAA and treats keyboard access, contrast, reflow, and automated checks as release requirements. Its public accessibility report (opens in a new tab) also records a real limitation: human testing with NVDA and VoiceOver has not yet been documented. Automation cannot certify lived usability, just as a field in a feed cannot certify the curb, platform, or vehicle that a rider encounters.

Reach is not impact

The project has extensive automated tests, but its user-research record (opens in a new tab) is explicit that the original personas and quotations are synthetic. One real participant from the MobilityData community later found useful features, weak discoverability, missing evidence about rider-facing translations, limited filtering, and thin European coverage. I treated that as one bounded signal. It justified clearer navigation, an ungraded translation measure, and a cautious coverage gate, not a claim of market validation or a global launch.

The next test is a 90-day concierge pilot (opens in a new tab). The project will follow a small number of findings from an alert, to an accepted owner and action, to a fresh copy of the exact intended feed. It will issue a closure receipt only when the feed identity and measurement method are comparable. That loop is under test, not a proven outcome. Until real maintainers complete it, the project can demonstrate technical reach and public inspectability, not operational impact.

What I would carry into the next civic project

• Start with the public decision, not the validator or dashboard. Ask who needs to act, what evidence they need, and how they can challenge the result.
• Reuse the canonical standard. Add project-specific value at the human handoff, where a technical finding becomes a consequence, priority, and possible repair.
• Treat a public metric as policy. Publish its weights, unknown states, exclusions, correction rights, and limits, not just the attractive number.
• Model identity and denominators before scale. Organizations, feeds, endpoints, aliases, configured records, and published results are different things.
• Version the evidence and methodology. Refuse to compare results when the source identity or measuring stick has changed.
• Design for accessibility in both the data and the delivery. Automated compliance is useful evidence, but it is not a substitute for human experience.
• Measure closed repairs, not feed count, traffic, or dashboard use. A finding becomes valuable when the right person can verify it, act on it, and prove that the source changed.

GTFS Scorecard began with a question about transit data and keeps returning me to a harder one: what does a civic technologist owe the public when software turns messy records into a clean answer?

Every number needs a denominator. Every status needs a meaning. Every methodology change needs a version. Every finding needs a route back to evidence and forward to repair. A rider should not have to trust the grade, and a maintainer should not have to guess what to do with it.

You can inspect the live project (opens in a new tab), read the source and decision records (opens in a new tab), and challenge the assumptions in the rubric. That is the point. Trustworthy public data is not data that asks to be trusted. It is data that leaves the work open.