Skill Evaluation Cases

These models define the structure of the eval_cases.yml file packaged with an agent skill.

Models for validating packaged agent-skill evaluation cases.

class EvalCase(*, id: Annotated[str, _PydanticGeneralMetadata(pattern='\\S')], category: Annotated[str, _PydanticGeneralMetadata(pattern='\\S')], prompt: Annotated[str, _PydanticGeneralMetadata(pattern='\\S')], expected: ExpectedResponse)

One prompt and its expected response constraints.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class ExpectedResponse(*, must_include: Annotated[list[Annotated[str, FieldInfo(annotation=NoneType, required=True, metadata=[_PydanticGeneralMetadata(pattern='\\S')])]], MinLen(min_length=1)], must_not_include: Annotated[list[Annotated[str, FieldInfo(annotation=NoneType, required=True, metadata=[_PydanticGeneralMetadata(pattern='\\S')])]], MinLen(min_length=1)])

Required and forbidden content for one evaluation response.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class PackagedSkillEvalCases(*, version: Literal[1], skill: Annotated[str, _PydanticGeneralMetadata(pattern='\\S')], cases: Annotated[list[EvalCase], MinLen(min_length=1)])

Schema for a packaged skill’s eval_cases.yml file.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].