How ScholarLens is built and measured.

The browser talks to one API, and the API runs one agent. The agent can read the curated knowledge base or search the web, and it chooses between them for each question.

Frontend

Next.js on Vercel. It sends the student's description to the API, and no API key ever reaches the browser.

API

FastAPI on Railway. Every request and response is validated with Pydantic, and each visitor has a rate limit.

Agent

Pydantic AI running Google Gemini (gemini-3.8-flash). It has two tools and decides which to call for each question.

Knowledge base

ChromaDB, holding the 10 program documents embedded with gemini-embedding-001. Documents are split into passages of up to 1,200 characters with no overlap, so a quoted clause stays whole, and a search returns the five closest passages.

Web search

Tavily, called only when the knowledge base doesn't cover the question.

Observability

OpenTelemetry traces, metrics, and logs sent to Grafana Cloud.

1. The API validates the request and checks the visitor's rate limit.
2. The agent searches the knowledge base, every time and before anything else.
3. If the passages it finds don't cover the question, it searches the web for the program's official pages.
4. The model returns a typed result: the verdict, an explanation, the quoted clause, its source, anything missing, and the steps it took.
5. A grounding check runs before the response is sent. An eligible or partial verdict needs a quoted clause and a matching source, and any alternative program whose link no tool returned is dropped.

The whole check runs inside a 45-second budget, and every call to the model, the knowledge base, and the web search has its own timeout inside it.

The agent is tested on 28 student profiles that a person labeled by hand, each answered three times for 84 answers in all. The latest run was on 27 October 2026 against the full knowledge base, with the same model the live site uses.

Eligibility accuracy

97.6%target above 80%

The verdict matches the one a person assigned to that profile.

Citation accuracy

93.9%target above 70%

The quoted rule and the program both match the knowledge base.

Hallucination rate

0.0%target below 10%

An eligible verdict with no citation, or a quoted rule that isn't in the knowledge base.

Consistency

92.9%no target

The same profile gets the same verdict on all three tries.

Answers took 8.9 seconds on average: 8.4 seconds when the knowledge base was enough, and 12.8 seconds when the agent also searched the web. Every citation miss named the right program but quoted a different clause of it than the label expected.

A visitor who runs too many checks gets a 429 response with a Retry-After header, and the check page tells them how long to wait. A timeout, a tool outage, or a limit at the model provider returns a 503 instead. When the provider asks for a short wait that still fits inside the time budget, the agent retries once before giving up.

In every case the student sees a plain message and their description stays in the form. The service never fills a gap with a guessed answer.

There are no accounts, no saved profiles, and no database of students. Logs and traces record that a check ran, how long it took, and which tools it used, but never what the student wrote or what the tools sent back.