Evaluate your server with models
Run an evaluation of your registered MCP server in the dashboard, and read it: how the cases are written, the distractor servers, what the metrics mean, how clients that search for tools find them, how the units are counted, and rewrite proposals for the tools it missed.
On this page
The linter scores how your tools are written. An evaluation checks what models do with them: for utterances written for each of your tools, it asks models which tool they would call, and checks the arguments they give. Evaluations come with the Pro and Team plans. Run them in the dashboard as below, or with the REST API.
Start an evaluation
Open Evaluations in the dashboard at app.forecall.dev and Start an evaluation. Choose one of your registered servers: its latest version is evaluated. Choose the models, any distractor servers and the utterances' language, then Estimate. Nothing is taken until you start; the button says how many units it takes.
The models you can choose are anthropic/claude-opus-5-5, anthropic/claude-sonnet-5-5,
openai/gpt-6-astra and openai/gpt-6.1-sol; google/gemini-3.8-flash is paused for now. The
middle ones, anthropic/claude-sonnet-5-5 and openai/gpt-6.1-sol, are chosen at first.
An evaluation takes a few minutes. Its page updates by itself while it runs (reload it when JavaScript is off), and shows each model's results as soon as its run is done.
The cases
A model (anthropic/claude-sonnet-5-5) writes the cases from your tools' names, descriptions and
input schemas. There are three kinds:
- For the tool: utterances that call for one tool, 8 for each tool on Pro and 20 on Team.
- Easily confused: for each pair of tools whose descriptions the linter finds nearly the same, two utterances for each of the two tools.
- Calls for no tool: utterances none of your tools should answer, a fifth as many as those for the tools.
The utterances carry what the tool needs to act on, such as the id of a page or the element in front of the user, with a concrete value for every argument the tool requires, so that the right answer is the tool itself and not a search for the id first. A tool whose description says it is deprecated gets no cases, and the estimate says so: models rightly avoid such a tool when asked for what it does.
What an agent picks from at run time, such as the results a search tool returns or a list of endpoints, can be evaluated the same way: register the list of entries as a server, and each entry is treated as a tool. The evaluation then measures which entry a model calls for each request. What it does not measure is the step before, the search itself, and how its results are laid out.
The utterances are in English, or in Japanese on Team. The cases are kept with the evaluation: a later evaluation of the same version with the same settings uses the same cases, so the two compare fairly, and writing them is not counted again.
Distractor servers
Your tools are seldom the only ones a model sees. Distractor servers put the tools of public
servers beside yours: bhived, context7, debugbase, filesystem, firecrawl, memory,
notion and playwright. None are chosen at first. A call to a distractor's tool counts as another
tool.
The estimate warns you when a distractor has a tool named like one of yours, as a call to either cannot be told apart, and when the models see more than 60 tools in all, which they handle less well.
Read the results
For each model, the evaluation's page shows four metrics, each with the counts behind it:
| Metric | What it is |
|---|---|
Tool choice (selectionAcc) |
Of the cases written for a tool, the share that called that tool. |
Arguments (argValidity) |
Of those calls, the share whose arguments fit the tool's input schema. A schema that cannot be read is left out. |
False calls (falseCallRate) |
Of the cases that call for no tool, the share that got a call. Lower is better. |
Lookups first (lookupRate) |
Of the cases written for a tool, the share that called a tool that reads instead, such as a list, get or search tool: the model went to get an id or a snapshot before acting. These are among the misses of Tool choice; they usually say the tool's description does not tell where its ids come from, not that it was mistaken for another. Lower is better. |
An answer cut off at the length limit is left out of all four. A metric with nothing to count
shows a dash. A tool counts as one that reads when its readOnlyHint annotation is true or its
name has a word such as list, get, search, read, open, retrieve, find, snapshot or query.
The page compares each model with the server's previous evaluation that has results, and shows Which tool was called: for each model, a row for the tool each case was written for and a column for the tool it called, the diagonal being right; a call to a tool that reads is marked apart, as a lookup first. Past 12 tools, it shows only the rows with a mistake and the tools called by mistake, and Show all brings back the rest. See each case lists every utterance with the tool each model called and a verdict, narrowed if you like to the cases any model missed, those one model missed, or one kind of mistake. A call with wrong arguments says which argument and how: missing, of the wrong type, not an allowed value. The utterances come from your tools, so only your organization's members see them.
Clients that search for tools
Some clients don't show the model all of a server's tools. Claude Code, by default, and the Anthropic API's tool search keep the tools' definitions out of the model's context (Claude Code shows their names): the model writes a search, and only the few tools that come back reach it in full. A description then has a second job, to be found.
Evaluations show the model every tool, so they don't measure this. We measured it separately: the same 404 requests for 88 tools of four servers, with 25–55 tools of three other servers beside them, asked of Claude Sonnet 5.5 with every tool deferred behind the API's tool search.
- With the word search (BM25), the right tool came back for 97.8% of the requests and was called for 94.6%, against 96.5% with every tool shown.
- With the pattern search (regex), it came back for 92.1% and was called for 88.1%.
- The score didn't predict which tools were found: tools under 40 came back as often as the others (BM25 97.7%, regex 95.5%), and the correlation was about zero with either search.
What decided the misses was how the model searched. Claude rewrites the request into its own words, so a tool is found by the words the model thinks of, not the user's. A pattern returns the tools that match in the order the server lists them, so a broad pattern on a long list can stop before a tool near its end. To help a client that searches first, say in the server's instructions what tasks the server handles (Claude Code loads them before any search), give related tools a common prefix, and put the words of the task near the start of each description.
Units
Each model's run takes units by the model's price and the size of what it is asked: the tools it is shown, yours and the distractors', and each utterance. Writing the cases takes units too, counted with the first model. A result your organization already has, for the same model, tools and utterance, is reused and not counted again. The estimate shows the total, each part, and this month's units left with your credits.
An evaluation takes its units from the month's allowance first, then from your credits, the ones that expire first. When both fall short, it does not start, and neither does it while the organization's last payment has failed, until it is paid. If it fails, the units of the cases it did not finish come back as credits, valid for six months.
Rewrite proposals
When an evaluation is done, its page lists under Rewrite proposals the tools the models missed: a case written for the tool that called another tool, called nothing, or called it with arguments that do not fit; or a case for another tool, or for no tool, that called it. Rewrite proposals come with the Pro and Team plans. For a tool, Make a rewrite proposal shows the cases it will measure, the models, and the units it takes; nothing is taken until you start.
A proposal changes the tool's description only: its name, input schema and annotations stay as
they are. A model (anthropic/claude-sonnet-5-5) writes 12 candidate descriptions from the current
one, the linter's findings, the tools whose descriptions are nearly the same, and the requests that
were missed. jev, a judging model, narrows them to three. The evaluation's models then answer the
same cases again with each of the three in place of the description: the cases written for the
tool, those of the tool it is confused with, those that called it by mistake, and four that call
for no tool. A candidate is proposed only when the models together answered more of these cases
right than in the evaluation, and no model answered fewer. Each case is asked once, so a small
difference can be chance: the page shows the counts, not a promise.
The proposal's page shows the description, with a button to copy it, what changes from the current one, each model's right answers before and after, where the evaluation's answers went wrong, and each case. When no candidate did better, it says so and shows the ones it measured. You can mark a proposal you will not use, and undo that.
Put the description in your server, register the new version, and evaluate it: the next evaluation checks the change on all your cases. The proposal's page says when a later version uses its description. Proposals are made in the dashboard; the REST API does not make them yet.
A proposal takes units like an evaluation: writing the candidates once, and three candidates asked each case by each model. Nothing is reused, as a new description makes a new list of tools. The units are taken before it starts, from the month's allowance and then your credits, and used even when no candidate does better. When fewer than three candidates could be written, or it fails, the units of what was not measured come back as credits valid for six months.