Keep your server's scores

The linter keeps a result for 30 days. To follow your server over time, register it in the dashboard at app.forecall.dev: every time its tools change, a new version is kept and scored, and you can see what each change did to the scores.

Sign in

Sign in with GitHub or with a link sent to your email address. The first sign-in creates your account and an organization for it. Your servers belong to the organization, and only its members can see them.

Register a server

Open Servers and Register a server. Give its name, and if you like its homepage, its transport (stdio, HTTP or both) and how to install it. Then paste its tools/list JSON or choose a file written by forecall dump, in any of the accepted formats. The list becomes the server's first version and is scored at once.

The server gets a slug from its name. You can change its details later with Edit the details; the slug stays the same.

Versions

Import the tools again from the server's page whenever they change, with an optional label such as 1.4.0. A version is kept only when the tools differ from the latest version: their names, titles, descriptions, input and output schemas, annotations, or their order. Importing the same list again adds no version.

Each version is scored with the scoring rules of the day. When the rules change, the latest version is scored again the next time you import it.

Compare versions

A version's page shows its score and what changed from the version before:

• the average score before and after
• tools added and removed
• issues across the server resolved and new
• for each tool that changed, its score before and after, the issues it resolved and the new ones, and a word-by-word diff of its description and of each argument's description

When the two versions were scored by different versions of the scoring rules, the scores are not compared; the changes to the tools still are.

Import a result from the linter

A result from the linter can become a server, or a new version of one, so that it is kept past its 30 days. On the result's page, the browser that scored it sees Keep it in your dashboard: press Import into the dashboard, sign in if needed, and choose a new server or one of yours. The link works once, for 10 minutes.

The tools are scored again with the current rules as they come in. After the import, the result's share URL stops working.

Delete a server

Owners and admins of the organization can delete a server from its page. Its versions and their scores are deleted with it, and this cannot be undone.

Failures reported by agents

When agents report a failure of one of your server's tools to the Failure KB, the server's page shows how many were reported in the last 30 days, and See the failures opens them for the last 7, 30 or 90 days: the totals, how many were reproduced and looked up, the error classes, the tools, the models and the clients that hit them, and the newest records. Records that are public link to their page in the KB; the others show only their tool, class and state. This page comes with the Pro and Team plans (plans), for every server of the organization.

From CI

To keep a version from your CI each time you build, send the tools/list with an API key: see API keys and the REST API.