Forecall
Failure recordsDocs
All scored servers

IA-QA Eval & RAG — 38 tools: tool descriptions scored io.github.JcJamet/ia-qa-eval

tools/list taken
(UTC)
From
Official MCP Registry
Connected to
https://www.ia-qa.com/mcp/eval
Repository
github.com/ia-qa-com/mcp
Scoring rules
v2
Removal
support@forecall.dev
Score this URL as it is today

Scores, issues and names only

This page shows how IA-QA Eval & RAG — 38 tools (io.github.JcJamet/ia-qa-eval)'s tool descriptions scored, the issues found and the names of its tools and parameters. The descriptions are the server's own: read them where the copy came from.

Forecall asked the URL this server lists in the official MCP Registry for its tools/list once, on the date above, without calling any tool. The server may have changed since.

servers/io.github.JcJamet/ia-qa-eval
Average score
62.9 / 100Cloudy
Tools
38
Near-identical descriptions
0

Across the server

  • Minor
    38 tools. Above 25, models start to tell tools apart less well.many_tools
  • Minor
    The server's instructions are 2125 characters. Claude Code reads only the first 2,048 by default (Codex 0.154.0 reads them whole). Put what the server is for first.instructions_long

Tools

conversation_analyze

39 / 100Rain

Purpose
14 / 20
When to use
0 / 20
Arguments
10 / 25
Return value
8 / 15
Constraints and side effects
7 / 10
Examples
0 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context
  • Major
    Arguments without a description: messages[].role, messages[].content.param_no_description
  • Minor
    String arguments with no constraint or example: messages[].content.loose_string_param

context_window_check

49 / 100Cloudy

Purpose
17 / 20
When to use
0 / 20
Arguments
12 / 25
Return value
8 / 15
Constraints and side effects
7 / 10
Examples
5 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context
  • Minor
    Arguments without a description: messages[].role, messages[].content.param_no_description
  • Minor
    String arguments with no constraint or example: messages[].role, messages[].content.loose_string_param

build_rag_prompt

50 / 100Cloudy

Purpose
17 / 20
When to use
0 / 20
Arguments
13 / 25
Return value
8 / 15
Constraints and side effects
7 / 10
Examples
5 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context
  • Minor
    Arguments without a description: chunks[].text, chunks[].source, chunks[].score.param_no_description
  • Minor
    String arguments with no constraint or example: chunks[].text, chunks[].source.loose_string_param

similarity_score

51 / 100Cloudy

Purpose
20 / 20
When to use
0 / 20
Arguments
11 / 25
Return value
8 / 15
Constraints and side effects
7 / 10
Examples
5 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context
  • Minor
    Arguments without a description: batch[].reference, batch[].hypothesis.param_no_description
  • Minor
    String arguments with no constraint or example: batch[].reference, batch[].hypothesis.loose_string_param
  • Minor
    No `required` list, so every argument is optional.no_required

run_eval_contract

52 / 100Cloudy

Purpose
20 / 20
When to use
0 / 20
Arguments
5 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
5 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context
  • Major
    Arguments without a description: api_keys.groq, api_keys.openai, api_keys.anthropic, api_keys.google, api_keys.hf, api_keys.deepseek, api_keys.xai, overrides.model, overrides.provider, overrides.temperature, overrides.system_prompt.param_no_description
  • Minor
    String arguments with no constraint or example: api_keys.groq, api_keys.openai, api_keys.anthropic, api_keys.google, api_keys.hf, api_keys.deepseek, api_keys.xai, overrides.model, overrides.provider, overrides.system_prompt.loose_string_param
  • Minor
    No `required` list, so every argument is optional.no_required

run_vlm_test_suite

52 / 100Cloudy

Purpose
20 / 20
When to use
0 / 20
Arguments
10 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
0 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context
  • Major
    Arguments without a description: image_base64, system_prompt, model, test_cases, test_cases[].id, test_cases[].question, test_cases[].assertion_type.param_no_description
  • Minor
    String arguments with no constraint or example: image_base64, system_prompt, test_cases[].id, test_cases[].question.loose_string_param

rag_relevance_rank

53 / 100Cloudy

Purpose
20 / 20
When to use
0 / 20
Arguments
18 / 25
Return value
8 / 15
Constraints and side effects
7 / 10
Examples
0 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context
  • Minor
    String arguments with no constraint or example: query.loose_string_param

cot_analyzer

55 / 100Cloudy

Purpose
17 / 20
When to use
0 / 20
Arguments
18 / 25
Return value
8 / 15
Constraints and side effects
7 / 10
Examples
5 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context

embedding_similarity

55 / 100Cloudy

Purpose
20 / 20
When to use
0 / 20
Arguments
10 / 25
Return value
8 / 15
Constraints and side effects
7 / 10
Examples
10 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context
  • Minor
    Arguments without a description: batch[].text_a, batch[].text_b.param_no_description
  • Minor
    String arguments with no constraint or example: batch[].text_a, batch[].text_b.loose_string_param
  • Minor
    No `required` list, so every argument is optional.no_required

rerank_evaluate

55 / 100Cloudy

Purpose
17 / 20
When to use
5 / 20
Arguments
13 / 25
Return value
8 / 15
Constraints and side effects
7 / 10
Examples
5 / 10
  • Minor
    Arguments without a description: query, passages[].text, passages[].id.param_no_description
  • Minor
    String arguments with no constraint or example: query, passages[].text, passages[].id.loose_string_param
  • Minor
    The meaning of the boolean argument passages[].relevant is unclear.vague_boolean

model_info

58 / 100Cloudy

Purpose
20 / 20
When to use
0 / 20
Arguments
18 / 25
Return value
8 / 15
Constraints and side effects
7 / 10
Examples
5 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context

prompt_test_suite

58 / 100Cloudy

Purpose
17 / 20
When to use
0 / 20
Arguments
19 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
0 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context
  • Minor
    String arguments with no constraint or example: must_include, must_not_include.loose_string_param
  • Minor
    The description mentions none of its 3 or more arguments.params_not_in_description

run_semantic_tests

58 / 100Cloudy

Purpose
17 / 20
When to use
0 / 20
Arguments
19 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
0 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context
  • Minor
    The description mentions none of its 3 or more arguments.params_not_in_description

bm25_score

60 / 100Cloudy

Purpose
20 / 20
When to use
0 / 20
Arguments
18 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
0 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context
  • Minor
    String arguments with no constraint or example: query.loose_string_param

llm_json_schema_check

60 / 100Cloudy

Purpose
20 / 20
When to use
0 / 20
Arguments
18 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
0 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context

estimate_llm_cost

61 / 100Cloudy

Purpose
20 / 20
When to use
0 / 20
Arguments
18 / 25
Return value
8 / 15
Constraints and side effects
10 / 10
Examples
5 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context

token_budget_calculator

62 / 100Cloudy

Purpose
17 / 20
When to use
0 / 20
Arguments
18 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
5 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context

consistency_check

63 / 100Cloudy

Purpose
20 / 20
When to use
0 / 20
Arguments
23 / 25
Return value
8 / 15
Constraints and side effects
7 / 10
Examples
5 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context

count_tokens

63 / 100Cloudy

Purpose
20 / 20
When to use
0 / 20
Arguments
18 / 25
Return value
15 / 15
Constraints and side effects
10 / 10
Examples
0 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context

hallucination_check

63 / 100Cloudy

Purpose
17 / 20
When to use
10 / 20
Arguments
9 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
5 / 10
  • Minor
    Arguments without a description: cases[].id, cases[].answer, cases[].context.param_no_description
  • Minor
    String arguments with no constraint or example: cases[].id, cases[].answer, cases[].context.loose_string_param
  • Minor
    No `required` list, so every argument is optional.no_required

metamorphic_check

63 / 100Cloudy

Purpose
17 / 20
When to use
10 / 20
Arguments
14 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
0 / 10
  • Minor
    Arguments without a description: variants[].id, thresholds.case, thresholds.typo, thresholds.paraphrase, thresholds.reorder, thresholds.translation, thresholds.specialization.param_no_description
  • Minor
    String arguments with no constraint or example: base.question, variants[].id, variants[].question.loose_string_param

response_quality_score

65 / 100Cloudy

Purpose
20 / 20
When to use
5 / 20
Arguments
18 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
0 / 10

No issues.

test_skill

65 / 100Cloudy

Purpose
20 / 20
When to use
5 / 20
Arguments
13 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
5 / 10
  • Minor
    Arguments without a description: scenarios[].prompt, scenarios[].should_trigger, other_skills[].name, other_skills[].description.param_no_description
  • Minor
    String arguments with no constraint or example: scenarios[].prompt, other_skills[].name, other_skills[].description.loose_string_param
  • Minor
    The meaning of the boolean argument scenarios[].should_trigger is unclear.vague_boolean

llm_generate

66 / 100Cloudy

Purpose
20 / 20
When to use
0 / 20
Arguments
18 / 25
Return value
8 / 15
Constraints and side effects
10 / 10
Examples
10 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context

generate_ci_workflow

67 / 100Cloudy

Purpose
20 / 20
When to use
0 / 20
Arguments
15 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
10 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context
  • Minor
    Arguments without a description: workflow_name.param_no_description
  • Minor
    String arguments with no constraint or example: cron, workflow_name, node_version.loose_string_param
  • Minor
    No `required` list, so every argument is optional.no_required

generate_eval_yaml

67 / 100Cloudy

Purpose
20 / 20
When to use
5 / 20
Arguments
20 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
0 / 10

No issues.

get_testing_guidelines

67 / 100Cloudy

Purpose
20 / 20
When to use
0 / 20
Arguments
20 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
5 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context
  • Minor
    No `required` list, so every argument is optional.no_required

llm_format_check

67 / 100Cloudy

Purpose
20 / 20
When to use
0 / 20
Arguments
20 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
5 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context

compare_models

70 / 100Clear

Purpose
20 / 20
When to use
0 / 20
Arguments
23 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
5 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context

compare_responses

70 / 100Clear

Purpose
20 / 20
When to use
0 / 20
Arguments
18 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
10 / 10
  • Major
    Says nothing about when to use it, or when not to.no_usage_context

llm_output_validator

71 / 100Clear

Purpose
20 / 20
When to use
5 / 20
Arguments
19 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
5 / 10
  • Minor
    The description mentions none of its 3 or more arguments.params_not_in_description

ab_test_report

75 / 100Clear

Purpose
20 / 20
When to use
10 / 20
Arguments
18 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
5 / 10
  • Minor
    The description mentions none of its 3 or more arguments.params_not_in_description

extract_json_from_text

75 / 100Clear

Purpose
20 / 20
When to use
10 / 20
Arguments
18 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
5 / 10

No issues.

find_tool

75 / 100Clear

Purpose
20 / 20
When to use
10 / 20
Arguments
18 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
5 / 10

No issues.

analyze_responses

77 / 100Clear

Purpose
17 / 20
When to use
10 / 20
Arguments
18 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
10 / 10
  • Minor
    String arguments with no constraint or example: responses[].text.loose_string_param

list_llm_models

77 / 100Clear

Purpose
20 / 20
When to use
10 / 20
Arguments
15 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
10 / 10
  • Minor
    No `required` list, so every argument is optional.no_required

needle_haystack_generate

77 / 100Clear

Purpose
20 / 20
When to use
5 / 20
Arguments
20 / 25
Return value
15 / 15
Constraints and side effects
7 / 10
Examples
10 / 10

No issues.

rate_tool

78 / 100Clear

Purpose
20 / 20
When to use
15 / 20
Arguments
18 / 25
Return value
8 / 15
Constraints and side effects
7 / 10
Examples
10 / 10

No issues.

What static scoring cannot tell

This score reads only the text of each description and schema: it says what is written, not what a model will do. Short tools whose names already say what they do, such as browser_close, score low here even when models use them well, and in our measurements current models chose the right tool in one step even from one-line descriptions, as long as the names and schemas differed. What the text cannot show, such as what happens over several steps and whether an added sentence helps or misleads, is what an evaluation measures, on the Pro and Team plans.

Is your server listed here and you would rather it were not? Email support@forecall.dev, and we will take it off within 7 days.

All scored servers