test custom widget assistant authoring
This commit is contained in:
ajnart committed
1 parent
ceed0979a1
commit
dfafad15bf
20 files changed
+1675
-36
No files matched your search
@@ -7,18 +7,26 @@ description: Author safe Homarr Custom JSX v2 widgets from API documentation.
|
||||
|
||||
Author one widget at a time. When connected to Homarr, retrieve `homarr://custom-widgets/schema` and only the component resources needed for the design. The live resources are authoritative for the installed release.
|
||||
|
||||
Keep documentation retrieval targeted: fetch at most eight named component documents before the first validation, or at most four after loading a complete example. Fetch another component only when a validation issue or a concrete missing interaction requires it; never load the catalog component-by-component.
|
||||
|
||||
1. Read the API documentation.
|
||||
2. Create one credential-free widget with keyed `sources`, keyed `requests`, optional keyed `options`, and safe JSX `template`.
|
||||
3. Validate, preview, test queries and simulated actions, visually inspect, and iterate.
|
||||
4. Configure deployment-specific source URLs and request credentials through Homarr; never repeat plaintext.
|
||||
5. Save only after narrow/wide, light/dark, loading, empty, error, and success states work. When connected through MCP, persist the exact final tested preview with `customWidget_createFromPreview`; do not resend a large template through `customWidget_create` unless no preview session is available.
|
||||
|
||||
Treat a supplied sample or successful preview response as the binding contract. Render every core requested field, guard optional arrays and nested values before indexing, and do not silently drop returned items. Show concise freshness context when the API supplies a timestamp. Give recoverable load errors and empty states a clear refresh or retry path.
|
||||
|
||||
Use `{option:name}` or `$option` for saved options. Use `{param:name}` or `$param` for values supplied by `SubFetch`, `ActionButton`, or `ToggleSwitch`. Load queries cannot use invocation parameters. Templates read `data`, `status`, `options`, and temporary `inputs`.
|
||||
|
||||
`SubFetch` with `trigger="manual"` renders its own load button. Pass a card or image through `triggerContent` with `triggerAriaLabel` when that content should launch the request. Its callback is `(result, meta)`; never author `onClick` or a fetch callback.
|
||||
|
||||
Use standard Mantine compound names and deliberate visual hierarchy, spacing, restrained semantic color, and responsive layouts. Avoid excessive nested cards.
|
||||
|
||||
Aim for distinctive polish without decorative gradients or fragile custom styling: use one semantic accent, clear surface contrast, purposeful typography, and compact rows that remain scannable in narrow tiles. Give standalone icons an accessible label or pair them with visible text.
|
||||
|
||||
Make initial states actionable with an example, useful hint, or clear next step. Use wrapping groups for variable-length labels and values on narrow tiles. Do not use an unlabeled decorative icon as an empty state.
|
||||
|
||||
Do not use imports, hooks, refs, raw HTML, raw event callbacks, browser requests, arbitrary functions, eval, bigint, npm packages, authored statement blocks, IIFEs, or recursion. Do not pretend MCP tools exist in an offline chat session.
|
||||
|
||||
To install an existing widget, use `customWidget_workshopSearch`, inspect it with `customWidget_workshopGet`, then use `customWidget_workshopInstall`. Configure each self-hosted source afterward with `customWidget_sourceConfigure`, or call `customWidget_configurationRequestUser` so the user can enter the server URL and credentials directly in Homarr.
|
||||
|
||||
@@ -77,11 +77,8 @@ provider or base URL clears existing credentials unless replacements are submitt
|
||||
prevents credentials for one service from being sent to another endpoint.
|
||||
|
||||
When a key is already stored, the page shows its protected state instead of an ambiguous empty password
|
||||
field. **Replace key** starts a two-step flow: enter the new key, then review the provider, destination,
|
||||
and impact before confirming. The existing key remains active until confirmation, and its value cannot be
|
||||
restored after replacement. Removing only the API key also requires explicit confirmation. **Remove all
|
||||
credentials** uses a separate destructive confirmation because it also clears the selected model and
|
||||
disables the assistant.
|
||||
field. Removing the API key requires explicit confirmation. Save that removal before entering and saving a
|
||||
different key.
|
||||
|
||||
:::info Container networking
|
||||
|
||||
@@ -250,6 +247,11 @@ create result and placement continues without asking the user to select it again
|
||||
include safe Homarr paths that open in the assistant without a page reload. When placement is still a choice, the
|
||||
assistant uses the structured question UI instead of ending with a prose question.
|
||||
|
||||
The authoring phase limits broad component-document retrieval and then moves into validation with the smallest useful
|
||||
reference set. Supplied samples and successful preview responses are treated as binding contracts: the assistant is
|
||||
instructed to render the requested fields, guard optional nested data, preserve every returned item, expose freshness
|
||||
when timestamps are available, and provide a refresh path for recoverable empty or error states.
|
||||
|
||||
The shield control in the conversation header can enable automatic approvals for the current conversation.
|
||||
While enabled, Homarr accepts every affirmative approval step as it arrives: protected mutations are approved,
|
||||
presented app details and board changes are used, and structured questions with an explicitly categorized
|
||||
|
||||
@@ -113,7 +113,29 @@ Homarr maintainers can run the same self-contained prompt that the workbench cop
|
||||
OPENROUTER_API_KEY=... pnpm --filter @homarr/custom-widgets test:ai
|
||||
```
|
||||
|
||||
The suite uses DeepSeek V4 Pro to author Pokédex, Portainer, football, Jellyfin, and Home Assistant widgets. Before DeepSeek V4 Flash judges a result, it must pass Homarr's real clipboard parser, canonical schema, and deterministic scenario checks for the documented API routes, authentication, bindings, actions, and essential runtime components. It saves every response, validated widget manifest, and detailed category scores under `packages/custom-widgets/.ai-evaluations/`.
|
||||
The manifest suite covers nine complex and public-API prompts. Its deterministic fixture-backed cases include a fake
|
||||
homelab health API, CoinMarketCap's keyless public API, Bored, Agify, and a multi-request Pokédex. Before the configured
|
||||
model judges a result, the candidate must pass Homarr's real clipboard parser, canonical schema, and deterministic
|
||||
checks for documented endpoints, authentication, request triggers, query bindings, response fields, and essential
|
||||
runtime components.
|
||||
|
||||
To exercise the real Homarr Assistant lifecycle instead of accepting a returned JSON block, run:
|
||||
|
||||
```sh
|
||||
OPENROUTER_API_KEY=... pnpm --filter @homarr/custom-widgets test:assistant-ai
|
||||
```
|
||||
|
||||
The assistant test exposes the same lazy skill, live schema, component catalog, targeted component references,
|
||||
validation, preview-query, and create-from-preview tools as the product. It refuses to save a definition that was not
|
||||
validated exactly or whose preview queries were not all tested. Multi-request fixtures return a different payload for
|
||||
each documented path and require manual-query parameters. Use `--case=pokedex`, `--case=coinmarketcap-keyless`,
|
||||
`--case=bored-activity`, or `--case=agify-name` for a focused run.
|
||||
|
||||
Both suites default generation and harsh review to OpenRouter's `~deepseek/deepseek-v4-flash-latest` route. They save
|
||||
tool traces, validated manifests, raw reviews, and detailed category scores under
|
||||
`packages/custom-widgets/.ai-evaluations/`. These live tests are maintainer-invoked rather than CI-required because
|
||||
they consume a configured provider account and exercise a nondeterministic model. The repository's ordinary tests
|
||||
cover the lifecycle enforcement, fixture routing, score calculation, and failure behavior without network access.
|
||||
|
||||
Each scenario stops as soon as it passes and has a hard limit of ten generation attempts. Override the models or
|
||||
per-scenario limit with `OPENROUTER_GENERATOR_MODEL`, `OPENROUTER_JUDGE_MODEL`, and
|
||||
@@ -121,3 +143,8 @@ per-scenario limit with `OPENROUTER_GENERATOR_MODEL`, `OPENROUTER_JUDGE_MODEL`,
|
||||
into the bundled-widget source deliberately.
|
||||
|
||||
The reference API check uses a real PokeAPI list/detail request and a read-only Portainer container query when `PORTAINER_BASE_URL` and `PORTAINER_API_KEY` are set. Use `pnpm --filter @homarr/custom-widgets test:reference-apis -- --pokeapi-only` for the public PokeAPI check alone. To exercise start, restart, and stop against a disposable container, also set `PORTAINER_TEST_CONTAINER_ID` and the explicit safeguard `PORTAINER_ALLOW_MUTATIONS=1`; the script restores the container's original running state.
|
||||
|
||||
Use `pnpm --filter @homarr/custom-widgets test:reference-apis -- --public-ai-apis-only` to verify the live Bored and
|
||||
CoinMarketCap response shapes used by the assistant fixtures. The same command also verifies Agify when
|
||||
`AGIFY_API_KEY` is configured; otherwise it reports the authenticated check as skipped without weakening the
|
||||
deterministic Agify fixture tests.
|
||||
@@ -42,6 +42,13 @@ describe("customWidgetAssistantInstructions", () => {
|
||||
expect(customWidgetAssistantInstructions).toContain("customWidget_getComponentCatalog");
|
||||
expect(customWidgetAssistantInstructions).toContain("customWidget_getComponent");
|
||||
expect(customWidgetAssistantInstructions).toContain("customWidget_getSharedProps once");
|
||||
expect(customWidgetAssistantInstructions).toContain("at most eight named component documents");
|
||||
expect(customWidgetAssistantInstructions).toContain("at most four after loading a complete example");
|
||||
expect(customWidgetAssistantInstructions).toContain("successful preview responses as binding contracts");
|
||||
expect(customWidgetAssistantInstructions).toContain("Make initial states actionable");
|
||||
expect(customWidgetAssistantInstructions).toContain("never load the catalog component-by-component");
|
||||
expect(customWidgetAssistantInstructions).toContain("compact rows that remain scannable in narrow tiles");
|
||||
expect(customWidgetAssistantInstructions).toContain("Label standalone icons");
|
||||
expect(customWidgetAssistantInstructions).toContain("customWidget_getExample");
|
||||
expect(customWidgetAssistantInstructions).toContain("templateLines");
|
||||
expect(customWidgetAssistantInstructions).toContain("installed `pokedex` example");
|
||||
|
||||
@@ -9,6 +9,10 @@ export const customWidgetAssistantInstructions = `
|
||||
Custom Widget authoring rules:
|
||||
- Custom Widget tools are mandatory for Custom JSX work. Never answer a request to create, repair, or materially change a Custom Widget with only a manifest, JSX, instructions, or an unsupported claim that it should work.
|
||||
- For every request to create, repair, validate, preview, or explain a Custom JSX widget, use the complete installed skill and JSON Schema from the trusted Custom Widget authoring context when they are preloaded below; do not call customWidget_getSkill or customWidget_schema in that case because Homarr has already loaded their complete contents into this system prompt. When they are not preloaded, first call customWidget_getSkill, whose response contains the complete installed skill and all references, then call customWidget_schema. Do not merely say that you loaded them: use their rules for the subsequent tool inputs. Follow those resources as authoritative. Next call the compact customWidget_getComponentCatalog for the installed release. Fetch only the relevant named component documentation with customWidget_getComponent and, when its interaction pattern applies, one named example with customWidget_getExample before authoring. Component results contain only component-specific props; if the design needs shared props such as spacing or color, collect their names from the catalog and call customWidget_getSharedProps once with only those names.
|
||||
- Keep documentation retrieval bounded so the authoring tool call has enough output budget. Fetch at most eight named component documents before the first validation, or at most four after loading a complete example. Fetch another only to resolve a concrete validation issue or missing interaction; never load the catalog component-by-component.
|
||||
- For polished widgets, use one restrained semantic accent, clear surface contrast, purposeful typography, and compact rows that remain scannable in narrow tiles. Avoid decorative gradients and repetitive nested cards. Label standalone icons for assistive technology or pair them with visible text.
|
||||
- Make initial states actionable with an example, useful hint, or clear next step. Use wrapping groups for variable-length labels and values on narrow tiles. Do not use an unlabeled decorative icon as an empty state.
|
||||
- Treat supplied sample data and successful preview responses as binding contracts: render every core requested field, guard optional arrays and nested values before indexing, and do not silently drop returned items. Show freshness context when a timestamp is available, and give recoverable empty or load-error states a clear refresh or retry path.
|
||||
- Use templateLines for multiline JSX tool input. Do not send a heavily escaped multiline template string when templateLines is accepted.
|
||||
- Keep each tool call within a reliable streamed size. For Pokédex or PokéAPI requests, start from the installed \`pokedex\` example returned by customWidget_getExample and improve it through validated iterations instead of generating one enormous untested JSX argument from scratch. Completeness comes from working interactions and states, not gratuitous line count.
|
||||
- Honor an explicit iteration count. Each requested iteration must make a meaningful improvement and complete its own validation, preview, and required preview-query checks. Persist only the last tested preview session.
|
||||
|
||||
@@ -6,9 +6,12 @@ import { getCustomWidgetSkill, getCustomWidgetSkillContent } from "@homarr/custo
|
||||
import { getCustomWidgetJsonSchema } from "@homarr/custom-widgets/core";
|
||||
|
||||
import {
|
||||
createCustomWidgetComponentDocumentBudget,
|
||||
createCustomWidgetDynamicContextController,
|
||||
getCustomWidgetAuthoringContext,
|
||||
isCustomWidgetAuthoringToolName,
|
||||
MAX_CUSTOM_WIDGET_COMPONENT_DOCUMENTS,
|
||||
MAX_CUSTOM_WIDGET_COMPONENT_DOCUMENTS_WITH_EXAMPLE,
|
||||
needsCustomWidgetAuthoringContext,
|
||||
preloadedCustomWidgetToolNames,
|
||||
prunePreloadedCustomWidgetModelMessages,
|
||||
@@ -190,6 +193,59 @@ describe("createCustomWidgetDynamicContextController", () => {
|
||||
}),
|
||||
).toBeUndefined();
|
||||
});
|
||||
|
||||
test("removes component documentation after the targeted retrieval budget is exhausted", () => {
|
||||
const prepareContext = createCustomWidgetDynamicContextController({
|
||||
isAdmin: true,
|
||||
baseInstructions: "Base instructions",
|
||||
availableToolNames: ["customWidget_getComponent", "customWidget_validate", "customWidget_previewCreate"],
|
||||
});
|
||||
const toolCalls = Array.from({ length: MAX_CUSTOM_WIDGET_COMPONENT_DOCUMENTS }, () => ({
|
||||
toolName: "customWidget_getComponent",
|
||||
}));
|
||||
|
||||
expect(prepareContext({ instructions: "Base instructions", messages: [], steps: [{ toolCalls }] })).toMatchObject({
|
||||
activeTools: ["customWidget_validate", "customWidget_previewCreate"],
|
||||
});
|
||||
});
|
||||
|
||||
test("uses a smaller component budget after a complete example is loaded", () => {
|
||||
const prepareContext = createCustomWidgetDynamicContextController({
|
||||
isAdmin: true,
|
||||
baseInstructions: "Base instructions",
|
||||
availableToolNames: ["customWidget_getComponent", "customWidget_getExample", "customWidget_validate"],
|
||||
});
|
||||
const toolCalls = [
|
||||
{ toolName: "customWidget_getExample" },
|
||||
...Array.from({ length: MAX_CUSTOM_WIDGET_COMPONENT_DOCUMENTS_WITH_EXAMPLE }, () => ({
|
||||
toolName: "customWidget_getComponent",
|
||||
})),
|
||||
];
|
||||
|
||||
expect(prepareContext({ instructions: "Base instructions", messages: [], steps: [{ toolCalls }] })).toMatchObject({
|
||||
activeTools: ["customWidget_getExample", "customWidget_validate"],
|
||||
});
|
||||
});
|
||||
|
||||
test("rejects excess parallel component document executions without blocking other tools", () => {
|
||||
const budget = createCustomWidgetComponentDocumentBudget(2);
|
||||
|
||||
expect(budget.claim("customWidget_getComponent")).toBe(true);
|
||||
expect(budget.claim("customWidget_getComponent")).toBe(true);
|
||||
expect(budget.claim("customWidget_getComponent")).toBe(false);
|
||||
expect(budget.claim("customWidget_validate")).toBe(true);
|
||||
});
|
||||
|
||||
test("reduces the hard execution budget after loading an example", () => {
|
||||
const budget = createCustomWidgetComponentDocumentBudget();
|
||||
|
||||
expect(budget.claim("customWidget_getExample")).toBe(true);
|
||||
for (let index = 0; index < MAX_CUSTOM_WIDGET_COMPONENT_DOCUMENTS_WITH_EXAMPLE; index += 1) {
|
||||
expect(budget.claim("customWidget_getComponent")).toBe(true);
|
||||
}
|
||||
expect(budget.claim("customWidget_getComponent")).toBe(false);
|
||||
expect(budget.claim("customWidget_validate")).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("prunePreloadedCustomWidgetModelMessages", () => {
|
||||
|
||||
@@ -19,6 +19,7 @@ const customWidgetAuthoringToolNameSet = new Set([
|
||||
"customWidget_previewAction",
|
||||
"customWidget_previewJournal",
|
||||
"customWidget_create",
|
||||
"customWidget_createFromPreview",
|
||||
"customWidget_update",
|
||||
"customWidget_templatePatch",
|
||||
"customWidget_legacyMigrationPrompt",
|
||||
@@ -139,6 +140,29 @@ type CustomWidgetToolStep = {
|
||||
toolCalls: readonly { toolName: string }[];
|
||||
};
|
||||
|
||||
export const MAX_CUSTOM_WIDGET_COMPONENT_DOCUMENTS = 8;
|
||||
export const MAX_CUSTOM_WIDGET_COMPONENT_DOCUMENTS_WITH_EXAMPLE = 4;
|
||||
|
||||
export const createCustomWidgetComponentDocumentBudget = (limit = MAX_CUSTOM_WIDGET_COMPONENT_DOCUMENTS) => {
|
||||
let served = 0;
|
||||
let exampleWasLoaded = false;
|
||||
return {
|
||||
claim(toolName: string) {
|
||||
if (toolName === "customWidget_getExample") {
|
||||
exampleWasLoaded = true;
|
||||
return true;
|
||||
}
|
||||
if (toolName !== "customWidget_getComponent") return true;
|
||||
const activeLimit = exampleWasLoaded
|
||||
? Math.min(limit, MAX_CUSTOM_WIDGET_COMPONENT_DOCUMENTS_WITH_EXAMPLE)
|
||||
: limit;
|
||||
if (served >= activeLimit) return false;
|
||||
served += 1;
|
||||
return true;
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
export const createCustomWidgetDynamicContextController = <TToolName extends string>(options: {
|
||||
isAdmin: boolean;
|
||||
baseInstructions: string;
|
||||
@@ -163,6 +187,22 @@ export const createCustomWidgetDynamicContextController = <TToolName extends str
|
||||
const shouldInjectContext = !contextWasInjected;
|
||||
if (shouldInjectContext) contextWasInjected = true;
|
||||
|
||||
const componentDocumentCount = step.steps.reduce(
|
||||
(count, result) =>
|
||||
count + result.toolCalls.filter((toolCall) => toolCall.toolName === "customWidget_getComponent").length,
|
||||
0,
|
||||
);
|
||||
const exampleWasLoaded = step.steps.some((result) =>
|
||||
result.toolCalls.some((toolCall) => toolCall.toolName === "customWidget_getExample"),
|
||||
);
|
||||
const componentDocumentLimit = exampleWasLoaded
|
||||
? MAX_CUSTOM_WIDGET_COMPONENT_DOCUMENTS_WITH_EXAMPLE
|
||||
: MAX_CUSTOM_WIDGET_COMPONENT_DOCUMENTS;
|
||||
const boundedActiveToolNames =
|
||||
componentDocumentCount >= componentDocumentLimit
|
||||
? activeToolNames.filter((toolName) => toolName !== "customWidget_getComponent")
|
||||
: activeToolNames;
|
||||
|
||||
return {
|
||||
...(shouldInjectContext
|
||||
? {
|
||||
@@ -170,7 +210,7 @@ export const createCustomWidgetDynamicContextController = <TToolName extends str
|
||||
}
|
||||
: {}),
|
||||
messages: prunePreloadedCustomWidgetModelMessages(step.messages),
|
||||
activeTools: activeToolNames,
|
||||
activeTools: boundedActiveToolNames,
|
||||
};
|
||||
};
|
||||
};
|
||||
@@ -52,6 +52,7 @@ import { getSafeAssistantToolError } from "./assistant-tool-error";
|
||||
import { repairAssistantToolInput } from "./assistant-tool-input-repair";
|
||||
import { toAssistantToolOutput } from "./assistant-tool-output";
|
||||
import {
|
||||
createCustomWidgetComponentDocumentBudget,
|
||||
createCustomWidgetDynamicContextController,
|
||||
getCustomWidgetAuthoringContext,
|
||||
prunePreloadedCustomWidgetModelMessages,
|
||||
@@ -476,6 +477,7 @@ export async function POST(request: Request) {
|
||||
const procedureTypes = getProcedureTypeMap();
|
||||
const omittedCustomWidgetTools = new Set<string>(customWidgetAuthoringContext.omittedToolNames);
|
||||
const mcpTools = extractMcpTools().filter((mcpTool) => !omittedCustomWidgetTools.has(mcpTool.name));
|
||||
const customWidgetComponentDocumentBudget = createCustomWidgetComponentDocumentBudget();
|
||||
|
||||
const homarrTools = Object.fromEntries(
|
||||
mcpTools.map((mcpTool) => {
|
||||
@@ -488,6 +490,12 @@ export async function POST(request: Request) {
|
||||
(mcpTool.inputSchema ?? { type: "object", properties: {} }) as Parameters<typeof jsonSchema>[0],
|
||||
),
|
||||
execute: async (input) => {
|
||||
if (!customWidgetComponentDocumentBudget.claim(mcpTool.name)) {
|
||||
return {
|
||||
error:
|
||||
"The targeted component-document budget is exhausted. Continue with validation and fetch another component only in a new repair turn if a concrete issue requires it.",
|
||||
};
|
||||
}
|
||||
try {
|
||||
const procedure = mcpTool.pathInRouter.reduce<unknown>(
|
||||
(current, segment) =>
|
||||
|
||||
@@ -28,8 +28,9 @@
|
||||
"generate:catalog": "tsx ./scripts/generate-component-metadata.ts",
|
||||
"generate:components": "pnpm generate:catalog",
|
||||
"generate": "pnpm generate:catalog",
|
||||
"evaluate:ai": "bun ./scripts/evaluate-ai-authoring.ts",
|
||||
"test:ai": "bun ./scripts/evaluate-ai-authoring.ts --ci",
|
||||
"evaluate:ai": "tsx ./scripts/evaluate-ai-authoring.ts",
|
||||
"test:ai": "tsx ./scripts/evaluate-ai-authoring.ts --ci",
|
||||
"test:assistant-ai": "tsx ./scripts/evaluate-ai-authoring.ts --assistant --ci",
|
||||
"test:reference-apis": "tsx ./scripts/verify-reference-apis.ts",
|
||||
"format": "oxfmt .",
|
||||
"lint": "oxlint",
|
||||
|
||||
@@ -0,0 +1,702 @@
|
||||
import { mkdir, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
|
||||
import { z } from "zod/v4";
|
||||
|
||||
import {
|
||||
getCustomWidgetComponent,
|
||||
getCustomWidgetComponentCatalog,
|
||||
getCustomWidgetExample,
|
||||
getCustomWidgetSharedProps,
|
||||
getCustomWidgetSkill,
|
||||
} from "../src/core/authoring-resources";
|
||||
import {
|
||||
customWidgetAuthoringDefinitionSchema,
|
||||
customWidgetDefinitionSchema,
|
||||
normalizeCustomWidgetAuthoringDefinition,
|
||||
} from "../src/core/custom-jsx-schema";
|
||||
import type { HomarrCustomWidgetV2 } from "../src/core/custom-jsx-schema";
|
||||
import { CUSTOM_WIDGET_MCP_AUTHORING_PROMPT } from "../src/core/ai-prompt";
|
||||
import { getCustomWidgetJsonSchema } from "../src/core/schema";
|
||||
import type { CustomWidgetAiEvaluationCase } from "./ai-evaluation-cases";
|
||||
import {
|
||||
DEFAULT_GENERATOR_MODEL,
|
||||
getDeterministicEvaluationIssues,
|
||||
judgePasses,
|
||||
parseJudgeResult,
|
||||
requestCustomWidgetJudge,
|
||||
} from "./ai-evaluation";
|
||||
import type { CustomWidgetJudgeResult } from "./ai-evaluation";
|
||||
|
||||
const MAX_ASSISTANT_STEPS = 24;
|
||||
|
||||
interface OpenRouterToolCall {
|
||||
id: string;
|
||||
type: "function";
|
||||
function: { name: string; arguments: string };
|
||||
}
|
||||
|
||||
interface OpenRouterAssistantMessage {
|
||||
role: "assistant";
|
||||
content: string | null;
|
||||
tool_calls?: OpenRouterToolCall[];
|
||||
}
|
||||
|
||||
type OpenRouterMessage =
|
||||
| { role: "system" | "user"; content: string }
|
||||
| OpenRouterAssistantMessage
|
||||
| { role: "tool"; tool_call_id: string; content: string };
|
||||
|
||||
interface OpenRouterResponse {
|
||||
choices?: Array<{ message?: OpenRouterAssistantMessage }>;
|
||||
error?: { message?: string };
|
||||
}
|
||||
|
||||
interface ToolDefinition {
|
||||
type: "function";
|
||||
function: {
|
||||
name: string;
|
||||
description: string;
|
||||
parameters: Record<string, unknown>;
|
||||
};
|
||||
}
|
||||
|
||||
interface PreviewState {
|
||||
widget: HomarrCustomWidgetV2;
|
||||
testedQueries: Set<string>;
|
||||
}
|
||||
|
||||
export const getRequiredAssistantEvaluationRequestParams = (request: HomarrCustomWidgetV2["requests"][string]) => {
|
||||
const serialized = JSON.stringify(request);
|
||||
return [
|
||||
...new Set([
|
||||
...Array.from(serialized.matchAll(/\{param:([^}]+)\}/gu), (match) => match[1]).filter(
|
||||
(name): name is string => name !== undefined,
|
||||
),
|
||||
...Array.from(serialized.matchAll(/"\$param":"([^"]+)"/gu), (match) => match[1]).filter(
|
||||
(name): name is string => name !== undefined,
|
||||
),
|
||||
]),
|
||||
];
|
||||
};
|
||||
|
||||
export const getAssistantEvaluationPreviewResponse = (
|
||||
testCase: CustomWidgetAiEvaluationCase,
|
||||
request: HomarrCustomWidgetV2["requests"][string],
|
||||
) =>
|
||||
testCase.previewResponses?.find(({ pathIncludes }) => request.path.includes(pathIncludes))?.response ??
|
||||
testCase.sampleResponse;
|
||||
|
||||
export interface AssistantAttemptState {
|
||||
calledTools: string[];
|
||||
validatedDefinitions: Set<string>;
|
||||
previews: Map<string, PreviewState>;
|
||||
createdWidget: HomarrCustomWidgetV2 | null;
|
||||
finalText: string;
|
||||
}
|
||||
|
||||
export interface CustomWidgetAssistantEvaluationResult {
|
||||
caseId: string;
|
||||
attempts: number;
|
||||
widget: HomarrCustomWidgetV2 | null;
|
||||
judge: CustomWidgetJudgeResult | null;
|
||||
outputDirectory: string;
|
||||
errors: string[];
|
||||
calledTools: string[];
|
||||
}
|
||||
|
||||
const objectSchema = (properties: Record<string, unknown>, required: string[] = []) => ({
|
||||
type: "object",
|
||||
properties,
|
||||
required,
|
||||
additionalProperties: false,
|
||||
});
|
||||
|
||||
export const customWidgetAssistantEvaluationToolDefinitions: ToolDefinition[] = [
|
||||
{
|
||||
type: "function",
|
||||
function: {
|
||||
name: "customWidget_getSkill",
|
||||
description:
|
||||
"Load the complete installed Custom Widget skill, including SKILL.md and every bundled reference. Call this before authoring.",
|
||||
parameters: objectSchema({}),
|
||||
},
|
||||
},
|
||||
{
|
||||
type: "function",
|
||||
function: {
|
||||
name: "customWidget_schema",
|
||||
description: "Get the authoritative installed Homarr Custom Widget v2 JSON Schema.",
|
||||
parameters: objectSchema({}),
|
||||
},
|
||||
},
|
||||
{
|
||||
type: "function",
|
||||
function: {
|
||||
name: "customWidget_getComponentCatalog",
|
||||
description:
|
||||
"Get the compact catalog of supported JSX components, shared props, and example IDs before selecting component documentation.",
|
||||
parameters: objectSchema({}),
|
||||
},
|
||||
},
|
||||
{
|
||||
type: "function",
|
||||
function: {
|
||||
name: "customWidget_getComponent",
|
||||
description: "Get installed documentation and safety rules for one named Custom JSX component.",
|
||||
parameters: objectSchema({ name: { type: "string" } }, ["name"]),
|
||||
},
|
||||
},
|
||||
{
|
||||
type: "function",
|
||||
function: {
|
||||
name: "customWidget_getSharedProps",
|
||||
description: "Get installed documentation for only the named shared Custom JSX props.",
|
||||
parameters: objectSchema({ names: { type: "array", items: { type: "string" }, minItems: 1, maxItems: 64 } }, [
|
||||
"names",
|
||||
]),
|
||||
},
|
||||
},
|
||||
{
|
||||
type: "function",
|
||||
function: {
|
||||
name: "customWidget_getExample",
|
||||
description: "Get one installed Custom JSX example by catalog ID.",
|
||||
parameters: objectSchema({ name: { type: "string" } }, ["name"]),
|
||||
},
|
||||
},
|
||||
{
|
||||
type: "function",
|
||||
function: {
|
||||
name: "customWidget_validate",
|
||||
description:
|
||||
"Validate one complete widget without saving it. Supply the complete definition in widget and prefer templateLines for multiline JSX.",
|
||||
parameters: objectSchema({ widget: { type: "object", additionalProperties: true } }, ["widget"]),
|
||||
},
|
||||
},
|
||||
{
|
||||
type: "function",
|
||||
function: {
|
||||
name: "customWidget_previewCreate",
|
||||
description:
|
||||
"Create a preview for the exact complete definition after validation. Returns every query that must be tested.",
|
||||
parameters: objectSchema(
|
||||
{
|
||||
definition: { type: "object", additionalProperties: true },
|
||||
secrets: { type: "array", items: { type: "object" }, maxItems: 0 },
|
||||
},
|
||||
["definition"],
|
||||
),
|
||||
},
|
||||
},
|
||||
{
|
||||
type: "function",
|
||||
function: {
|
||||
name: "customWidget_previewQuery",
|
||||
description:
|
||||
"Execute one query from a preview against the evaluation fixture. Call once for every query returned by previewCreate.",
|
||||
parameters: objectSchema(
|
||||
{
|
||||
sessionId: { type: "string" },
|
||||
requestId: { type: "string" },
|
||||
params: { type: "object", additionalProperties: true },
|
||||
},
|
||||
["sessionId", "requestId"],
|
||||
),
|
||||
},
|
||||
},
|
||||
{
|
||||
type: "function",
|
||||
function: {
|
||||
name: "customWidget_createFromPreview",
|
||||
description: "Persist the exact final tested preview. Every query in that preview must have succeeded first.",
|
||||
parameters: objectSchema({ previewSessionId: { type: "string" } }, ["previewSessionId"]),
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
const getDefinitionSignature = (widget: HomarrCustomWidgetV2) => JSON.stringify(widget);
|
||||
|
||||
function parseDefinition(value: unknown) {
|
||||
const authoring = customWidgetAuthoringDefinitionSchema.safeParse(value);
|
||||
if (!authoring.success) {
|
||||
return {
|
||||
success: false as const,
|
||||
issues: authoring.error.issues.map((issue) => ({
|
||||
path: issue.path.map(String).join("."),
|
||||
code: issue.code,
|
||||
message: issue.message,
|
||||
})),
|
||||
};
|
||||
}
|
||||
try {
|
||||
return { success: true as const, widget: normalizeCustomWidgetAuthoringDefinition(authoring.data) };
|
||||
} catch (error) {
|
||||
if (!(error instanceof z.ZodError)) throw error;
|
||||
return {
|
||||
success: false as const,
|
||||
issues: error.issues.map((issue) => ({
|
||||
path: issue.path.map(String).join("."),
|
||||
code: issue.code,
|
||||
message: issue.message,
|
||||
})),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
export function executeAssistantEvaluationTool(
|
||||
testCase: CustomWidgetAiEvaluationCase,
|
||||
state: AssistantAttemptState,
|
||||
name: string,
|
||||
input: Record<string, unknown>,
|
||||
): unknown {
|
||||
const componentDocumentLimit = state.calledTools.includes("customWidget_getExample") ? 4 : 8;
|
||||
if (
|
||||
name === "customWidget_getComponent" &&
|
||||
state.calledTools.filter((toolName) => toolName === "customWidget_getComponent").length >= componentDocumentLimit
|
||||
) {
|
||||
return {
|
||||
error:
|
||||
"The targeted component-document budget is exhausted. Continue with validation using the documentation already loaded.",
|
||||
};
|
||||
}
|
||||
state.calledTools.push(name);
|
||||
if (name === "customWidget_getSkill") return getCustomWidgetSkill();
|
||||
if (name === "customWidget_schema") return getCustomWidgetJsonSchema();
|
||||
if (name === "customWidget_getComponentCatalog") return getCustomWidgetComponentCatalog();
|
||||
if (name === "customWidget_getComponent") {
|
||||
const component = typeof input.name === "string" ? getCustomWidgetComponent(input.name) : null;
|
||||
return component ?? { error: "Custom JSX component not found" };
|
||||
}
|
||||
if (name === "customWidget_getSharedProps") {
|
||||
const names = Array.isArray(input.names)
|
||||
? input.names.filter((entry): entry is string => typeof entry === "string")
|
||||
: [];
|
||||
return names.length > 0
|
||||
? getCustomWidgetSharedProps(names)
|
||||
: { error: "At least one shared prop name is required" };
|
||||
}
|
||||
if (name === "customWidget_getExample") {
|
||||
const example = typeof input.name === "string" ? getCustomWidgetExample(input.name) : null;
|
||||
if (!example) return { error: "Custom JSX example not found" };
|
||||
const { template, ...widget } = example.widget;
|
||||
return { ...example, widget: { ...widget, templateLines: template.split("\n") } };
|
||||
}
|
||||
if (name === "customWidget_validate") {
|
||||
const parsed = parseDefinition(input.widget);
|
||||
if (!parsed.success) return { valid: false, issues: parsed.issues };
|
||||
state.validatedDefinitions.add(getDefinitionSignature(parsed.widget));
|
||||
return {
|
||||
valid: true,
|
||||
issues: [],
|
||||
summary: {
|
||||
name: parsed.widget.name,
|
||||
sourceIds: Object.keys(parsed.widget.sources),
|
||||
requestIds: Object.keys(parsed.widget.requests),
|
||||
optionIds: Object.keys(parsed.widget.options),
|
||||
templateLineCount: parsed.widget.template.split("\n").length,
|
||||
},
|
||||
nextStep: "Reuse this exact definition with customWidget_previewCreate and test every returned query.",
|
||||
};
|
||||
}
|
||||
if (name === "customWidget_previewCreate") {
|
||||
const parsed = parseDefinition(input.definition);
|
||||
if (!parsed.success) return { error: "Definition is invalid", issues: parsed.issues };
|
||||
if (!state.validatedDefinitions.has(getDefinitionSignature(parsed.widget))) {
|
||||
return { error: "Validate this exact complete definition before creating its preview." };
|
||||
}
|
||||
const id = `preview-${state.previews.size + 1}`;
|
||||
state.previews.set(id, { widget: parsed.widget, testedQueries: new Set() });
|
||||
return {
|
||||
success: true,
|
||||
previewSession: { id },
|
||||
previewPath: `/manage/custom-widgets/preview/${id}`,
|
||||
queries: Object.entries(parsed.widget.requests).flatMap(([requestId, request]) =>
|
||||
request.kind === "query"
|
||||
? [
|
||||
{
|
||||
requestId,
|
||||
trigger: request.trigger,
|
||||
requiredParams: getRequiredAssistantEvaluationRequestParams(request),
|
||||
nextStep: `Call customWidget_previewQuery with sessionId '${id}' and requestId '${requestId}'.`,
|
||||
},
|
||||
]
|
||||
: [],
|
||||
),
|
||||
};
|
||||
}
|
||||
if (name === "customWidget_previewQuery") {
|
||||
const sessionId = typeof input.sessionId === "string" ? input.sessionId : "";
|
||||
const requestId = typeof input.requestId === "string" ? input.requestId : "";
|
||||
const preview = state.previews.get(sessionId);
|
||||
const request = preview?.widget.requests[requestId];
|
||||
if (!preview || request?.kind !== "query") return { error: "Preview query was not found" };
|
||||
const params =
|
||||
typeof input.params === "object" && input.params !== null && !Array.isArray(input.params)
|
||||
? (input.params as Record<string, unknown>)
|
||||
: {};
|
||||
const missingParams = getRequiredAssistantEvaluationRequestParams(request).filter(
|
||||
(paramName) => !(paramName in params) || params[paramName] === null || params[paramName] === "",
|
||||
);
|
||||
if (missingParams.length > 0) {
|
||||
return { error: `Supply the required manual preview parameters: ${missingParams.join(", ")}` };
|
||||
}
|
||||
const response = getAssistantEvaluationPreviewResponse(testCase, request);
|
||||
if (response === undefined) return { error: `No deterministic preview response is configured for ${request.path}` };
|
||||
preview.testedQueries.add(requestId);
|
||||
return {
|
||||
ok: true,
|
||||
status: 200,
|
||||
data: response,
|
||||
request: { method: request.method, path: request.path },
|
||||
};
|
||||
}
|
||||
if (name === "customWidget_createFromPreview") {
|
||||
const sessionId = typeof input.previewSessionId === "string" ? input.previewSessionId : "";
|
||||
const preview = state.previews.get(sessionId);
|
||||
if (!preview) return { error: "Preview session not found" };
|
||||
const untested = Object.entries(preview.widget.requests).flatMap(([requestId, request]) =>
|
||||
request.kind === "query" && !preview.testedQueries.has(requestId) ? [requestId] : [],
|
||||
);
|
||||
if (untested.length > 0) return { error: `Test every preview query before creation: ${untested.join(", ")}` };
|
||||
const completedPreviewCycles = [...state.previews.values()].filter((candidate) =>
|
||||
Object.entries(candidate.widget.requests).every(
|
||||
([requestId, request]) => request.kind !== "query" || candidate.testedQueries.has(requestId),
|
||||
),
|
||||
).length;
|
||||
const minimumPreviewCycles = testCase.minimumPreviewCycles ?? 1;
|
||||
if (completedPreviewCycles < minimumPreviewCycles) {
|
||||
return {
|
||||
error: `Complete at least ${minimumPreviewCycles} validated preview-and-query cycles before creation; ${completedPreviewCycles} completed. Revise the candidate meaningfully, validate it, create a fresh preview, and test every query again.`,
|
||||
};
|
||||
}
|
||||
state.createdWidget = customWidgetDefinitionSchema.parse(preview.widget);
|
||||
return {
|
||||
id: `created-${testCase.id}`,
|
||||
managementPath: `/manage/custom-widgets/edit/created-${testCase.id}`,
|
||||
nextAction: { type: "place-custom-widget", widgetKind: "customApi" },
|
||||
};
|
||||
}
|
||||
return { error: `Unknown evaluation tool '${name}'` };
|
||||
}
|
||||
|
||||
export function getAssistantEvaluationLifecycleIssues(state: AssistantAttemptState) {
|
||||
const required = [
|
||||
"customWidget_getSkill",
|
||||
"customWidget_schema",
|
||||
"customWidget_getComponentCatalog",
|
||||
"customWidget_validate",
|
||||
"customWidget_previewCreate",
|
||||
"customWidget_previewQuery",
|
||||
"customWidget_createFromPreview",
|
||||
];
|
||||
const issues = required.flatMap((name) =>
|
||||
state.calledTools.includes(name) ? [] : [`The assistant never called ${name}.`],
|
||||
);
|
||||
const componentDocumentCount = state.calledTools.filter((name) => name === "customWidget_getComponent").length;
|
||||
if (componentDocumentCount > 8) {
|
||||
issues.push(
|
||||
`The assistant fetched ${componentDocumentCount} component documents before creation; use at most eight targeted component documents.`,
|
||||
);
|
||||
}
|
||||
return issues;
|
||||
}
|
||||
|
||||
export function createAssistantEvaluationState(): AssistantAttemptState {
|
||||
return {
|
||||
calledTools: [],
|
||||
validatedDefinitions: new Set(),
|
||||
previews: new Map(),
|
||||
createdWidget: null,
|
||||
finalText: "",
|
||||
};
|
||||
}
|
||||
|
||||
function buildAssistantPrompt(testCase: CustomWidgetAiEvaluationCase, feedback: readonly string[]) {
|
||||
const sections = [
|
||||
testCase.request,
|
||||
`Verified API documentation: ${testCase.documentationUrl}`,
|
||||
`Verified API notes:\n${testCase.apiNotes}`,
|
||||
];
|
||||
if (testCase.sampleResponse !== undefined) {
|
||||
sections.push(`Representative preview response:\n${JSON.stringify(testCase.sampleResponse, null, 2)}`);
|
||||
}
|
||||
if (testCase.previewResponses?.length) {
|
||||
sections.push(
|
||||
`Representative preview responses:\n${testCase.previewResponses
|
||||
.map(({ pathIncludes, response }) => `${pathIncludes}:\n${JSON.stringify(response, null, 2)}`)
|
||||
.join("\n\n")}`,
|
||||
);
|
||||
}
|
||||
if (feedback.length > 0) {
|
||||
sections.push(
|
||||
`A previous independent review found these problems. Correct them in a fresh complete lifecycle:\n${feedback.join("\n")}`,
|
||||
);
|
||||
}
|
||||
sections.push(
|
||||
"Use the available Custom Widget tools and continue automatically until the exact tested preview is created. Do not merely return JSON or instructions.",
|
||||
);
|
||||
return sections.join("\n\n");
|
||||
}
|
||||
|
||||
async function callAssistantStep(args: {
|
||||
apiKey: string;
|
||||
model: string;
|
||||
messages: OpenRouterMessage[];
|
||||
tools: ToolDefinition[];
|
||||
toolChoice?: string;
|
||||
}): Promise<OpenRouterAssistantMessage> {
|
||||
const response = await fetch("https://openrouter.ai/api/v1/chat/completions", {
|
||||
method: "POST",
|
||||
headers: {
|
||||
Authorization: `Bearer ${args.apiKey}`,
|
||||
"Content-Type": "application/json",
|
||||
"HTTP-Referer": "https://homarr.dev",
|
||||
"X-Title": "Homarr Custom Widget Assistant Evaluation",
|
||||
},
|
||||
body: JSON.stringify({
|
||||
model: args.model,
|
||||
messages: args.messages,
|
||||
tools: args.tools,
|
||||
tool_choice: args.toolChoice ? { type: "function", function: { name: args.toolChoice } } : "auto",
|
||||
temperature: 0.1,
|
||||
max_tokens: 12_000,
|
||||
reasoning: { effort: "low", exclude: true },
|
||||
}),
|
||||
signal: AbortSignal.timeout(180_000),
|
||||
});
|
||||
const payload = (await response.json()) as OpenRouterResponse;
|
||||
if (!response.ok) {
|
||||
throw new Error(`OpenRouter request failed (${response.status}): ${payload.error?.message ?? "Unknown error"}`);
|
||||
}
|
||||
const message = payload.choices?.[0]?.message;
|
||||
if (!message) throw new Error("OpenRouter returned no assistant message");
|
||||
return message;
|
||||
}
|
||||
|
||||
export const getAssistantEvaluationRecoveryToolName = (state: AssistantAttemptState, minimumPreviewCycles = 1) => {
|
||||
for (const requiredTool of ["customWidget_getSkill", "customWidget_schema", "customWidget_getComponentCatalog"]) {
|
||||
if (!state.calledTools.includes(requiredTool)) return requiredTool;
|
||||
}
|
||||
if (state.validatedDefinitions.size === 0) return "customWidget_validate";
|
||||
const latestPreview = [...state.previews.entries()].at(-1);
|
||||
if (!latestPreview) return "customWidget_previewCreate";
|
||||
const [previewId, preview] = latestPreview;
|
||||
const untestedQuery = Object.entries(preview.widget.requests).find(
|
||||
([requestId, request]) => request.kind === "query" && !preview.testedQueries.has(requestId),
|
||||
);
|
||||
if (untestedQuery) return "customWidget_previewQuery";
|
||||
const completedPreviewCycles = [...state.previews.values()].filter((candidate) =>
|
||||
Object.entries(candidate.widget.requests).every(
|
||||
([requestId, request]) => request.kind !== "query" || candidate.testedQueries.has(requestId),
|
||||
),
|
||||
).length;
|
||||
if (completedPreviewCycles < minimumPreviewCycles) return "customWidget_validate";
|
||||
if (previewId && !state.createdWidget) return "customWidget_createFromPreview";
|
||||
return undefined;
|
||||
};
|
||||
|
||||
async function callAssistantStepWithRecovery(args: {
|
||||
apiKey: string;
|
||||
model: string;
|
||||
messages: OpenRouterMessage[];
|
||||
tools: ToolDefinition[];
|
||||
state: AssistantAttemptState;
|
||||
testCase: CustomWidgetAiEvaluationCase;
|
||||
}) {
|
||||
try {
|
||||
const message = await callAssistantStep(args);
|
||||
if ((message.tool_calls?.length ?? 0) > 0 || (args.state.createdWidget && (message.content ?? "").trim())) {
|
||||
return message;
|
||||
}
|
||||
} catch (error) {
|
||||
if (!(error instanceof Error) || (error.name !== "TimeoutError" && error.name !== "AbortError")) throw error;
|
||||
}
|
||||
|
||||
const toolChoice = getAssistantEvaluationRecoveryToolName(args.state, args.testCase.minimumPreviewCycles ?? 1);
|
||||
if (!toolChoice || !args.tools.some(({ function: definition }) => definition.name === toolChoice)) {
|
||||
throw new Error(
|
||||
"The model returned a prose-only, empty, or timed-out step before completing the Custom Widget lifecycle.",
|
||||
);
|
||||
}
|
||||
process.stdout.write(` recovering with required tool: ${toolChoice}\n`);
|
||||
const recovered = await callAssistantStep({ ...args, toolChoice });
|
||||
if ((!recovered.tool_calls || recovered.tool_calls.length === 0) && !(recovered.content ?? "").trim()) {
|
||||
throw new Error("The model returned an empty assistant step before completing the Custom Widget lifecycle.");
|
||||
}
|
||||
return recovered;
|
||||
}
|
||||
|
||||
async function runAssistantAttempt(args: {
|
||||
testCase: CustomWidgetAiEvaluationCase;
|
||||
apiKey: string;
|
||||
model: string;
|
||||
feedback: readonly string[];
|
||||
}) {
|
||||
const state = createAssistantEvaluationState();
|
||||
const messages: OpenRouterMessage[] = [
|
||||
{
|
||||
role: "system",
|
||||
content: `${CUSTOM_WIDGET_MCP_AUTHORING_PROMPT}\n\nThis is a tool-driven evaluation. You must begin by loading the complete installed skill, live schema, and component catalog through their tools. Do not stop until customWidget_createFromPreview succeeds.`,
|
||||
},
|
||||
{ role: "user", content: buildAssistantPrompt(args.testCase, args.feedback) },
|
||||
];
|
||||
|
||||
for (let step = 0; step < MAX_ASSISTANT_STEPS; step += 1) {
|
||||
const componentDocumentCount = state.calledTools.filter((name) => name === "customWidget_getComponent").length;
|
||||
const componentDocumentLimit = state.calledTools.includes("customWidget_getExample") ? 4 : 8;
|
||||
const tools =
|
||||
componentDocumentCount >= componentDocumentLimit
|
||||
? customWidgetAssistantEvaluationToolDefinitions.filter(
|
||||
({ function: definition }) => definition.name !== "customWidget_getComponent",
|
||||
)
|
||||
: customWidgetAssistantEvaluationToolDefinitions;
|
||||
const message = await callAssistantStepWithRecovery({
|
||||
apiKey: args.apiKey,
|
||||
model: args.model,
|
||||
messages,
|
||||
tools,
|
||||
state,
|
||||
testCase: args.testCase,
|
||||
});
|
||||
if ((!message.tool_calls || message.tool_calls.length === 0) && !(message.content ?? "").trim()) {
|
||||
throw new Error("The model returned an empty assistant step before completing the Custom Widget lifecycle.");
|
||||
}
|
||||
messages.push(message);
|
||||
if (!message.tool_calls || message.tool_calls.length === 0) {
|
||||
state.finalText = message.content ?? "";
|
||||
break;
|
||||
}
|
||||
for (const toolCall of message.tool_calls) {
|
||||
process.stdout.write(` tool: ${toolCall.function.name}\n`);
|
||||
let output: unknown;
|
||||
try {
|
||||
const parsed = JSON.parse(toolCall.function.arguments) as unknown;
|
||||
const input =
|
||||
parsed !== null && typeof parsed === "object" && !Array.isArray(parsed)
|
||||
? (parsed as Record<string, unknown>)
|
||||
: {};
|
||||
output = executeAssistantEvaluationTool(args.testCase, state, toolCall.function.name, input);
|
||||
} catch (error) {
|
||||
output = { error: error instanceof Error ? error.message : "Tool input was not valid JSON" };
|
||||
}
|
||||
messages.push({ role: "tool", tool_call_id: toolCall.id, content: JSON.stringify(output) });
|
||||
}
|
||||
if (state.createdWidget) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
return { state, messages };
|
||||
}
|
||||
|
||||
export async function evaluateCustomWidgetAssistantCase(args: {
|
||||
testCase: CustomWidgetAiEvaluationCase;
|
||||
apiKey: string;
|
||||
outputRoot: string;
|
||||
maxLoops: number;
|
||||
generatorModel?: string;
|
||||
judgeModel?: string;
|
||||
}): Promise<CustomWidgetAssistantEvaluationResult> {
|
||||
const caseDirectory = path.join(args.outputRoot, `assistant-${args.testCase.id}`);
|
||||
await mkdir(caseDirectory, { recursive: true });
|
||||
const errors: string[] = [];
|
||||
const feedback: string[] = [];
|
||||
let bestWidget: HomarrCustomWidgetV2 | null = null;
|
||||
let bestJudge: CustomWidgetJudgeResult | null = null;
|
||||
let bestCalledTools: string[] = [];
|
||||
|
||||
for (let attempt = 1; attempt <= args.maxLoops; attempt += 1) {
|
||||
process.stdout.write(` assistant attempt ${attempt}/${args.maxLoops}\n`);
|
||||
let run: Awaited<ReturnType<typeof runAssistantAttempt>>;
|
||||
try {
|
||||
run = await runAssistantAttempt({
|
||||
testCase: args.testCase,
|
||||
apiKey: args.apiKey,
|
||||
model: args.generatorModel ?? DEFAULT_GENERATOR_MODEL,
|
||||
feedback,
|
||||
});
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : "Unknown assistant evaluation error";
|
||||
errors.push(`Attempt ${attempt}: ${message}`);
|
||||
feedback.splice(0, feedback.length, message);
|
||||
continue;
|
||||
}
|
||||
await writeFile(path.join(caseDirectory, `trace-${attempt}.json`), JSON.stringify(run.messages, null, 2), "utf8");
|
||||
const lifecycleIssues = getAssistantEvaluationLifecycleIssues(run.state);
|
||||
if (!run.state.createdWidget || lifecycleIssues.length > 0) {
|
||||
const issues = [
|
||||
...lifecycleIssues,
|
||||
...(run.state.createdWidget ? [] : ["The assistant did not create the final tested preview."]),
|
||||
];
|
||||
errors.push(`Attempt ${attempt}: lifecycle failed — ${issues.join(" ")}`);
|
||||
feedback.splice(0, feedback.length, ...issues);
|
||||
continue;
|
||||
}
|
||||
const deterministicIssues = getDeterministicEvaluationIssues(args.testCase, run.state.createdWidget);
|
||||
if (deterministicIssues.length > 0) {
|
||||
const issues = deterministicIssues.map(({ message }) => message);
|
||||
errors.push(`Attempt ${attempt}: deterministic checks failed — ${issues.join("; ")}`);
|
||||
feedback.splice(0, feedback.length, ...issues);
|
||||
continue;
|
||||
}
|
||||
|
||||
await writeFile(
|
||||
path.join(caseDirectory, `widget-${attempt}.json`),
|
||||
JSON.stringify(run.state.createdWidget, null, 2),
|
||||
"utf8",
|
||||
);
|
||||
let judgeRaw: string;
|
||||
try {
|
||||
judgeRaw = await requestCustomWidgetJudge({
|
||||
testCase: args.testCase,
|
||||
widget: run.state.createdWidget,
|
||||
apiKey: args.apiKey,
|
||||
judgeModel: args.judgeModel,
|
||||
});
|
||||
await writeFile(path.join(caseDirectory, `judge-${attempt}.json`), judgeRaw, "utf8");
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : "Unknown judge error";
|
||||
errors.push(`Attempt ${attempt}: judge request failed — ${message}`);
|
||||
feedback.splice(0, feedback.length, "Return a complete polished widget for another independent review.");
|
||||
continue;
|
||||
}
|
||||
let judgeResult: CustomWidgetJudgeResult;
|
||||
try {
|
||||
judgeResult = parseJudgeResult(judgeRaw);
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : "Unknown judge parse error";
|
||||
errors.push(`Attempt ${attempt}: judge response invalid — ${message}`);
|
||||
feedback.splice(0, feedback.length, "Return a complete polished widget for another independent review.");
|
||||
continue;
|
||||
}
|
||||
if (!bestJudge || judgeResult.total > bestJudge.total) {
|
||||
bestWidget = run.state.createdWidget;
|
||||
bestJudge = judgeResult;
|
||||
bestCalledTools = run.state.calledTools;
|
||||
}
|
||||
if (judgePasses(judgeResult)) {
|
||||
return {
|
||||
caseId: args.testCase.id,
|
||||
attempts: attempt,
|
||||
widget: run.state.createdWidget,
|
||||
judge: judgeResult,
|
||||
outputDirectory: caseDirectory,
|
||||
errors,
|
||||
calledTools: run.state.calledTools,
|
||||
};
|
||||
}
|
||||
const issues = judgeResult.highestImpactFixes.length ? judgeResult.highestImpactFixes : judgeResult.problems;
|
||||
errors.push(`Attempt ${attempt}: judge ${judgeResult.total}/100 — ${issues.join("; ")}`);
|
||||
feedback.splice(0, feedback.length, ...issues);
|
||||
}
|
||||
|
||||
return {
|
||||
caseId: args.testCase.id,
|
||||
attempts: args.maxLoops,
|
||||
widget: bestWidget,
|
||||
judge: bestJudge,
|
||||
outputDirectory: caseDirectory,
|
||||
errors,
|
||||
calledTools: bestCalledTools,
|
||||
};
|
||||
}
|
||||
@@ -3,6 +3,25 @@ export interface CustomWidgetAiEvaluationCase {
|
||||
request: string;
|
||||
documentationUrl: string;
|
||||
apiNotes: string;
|
||||
sampleResponse?: unknown;
|
||||
previewResponses?: Array<{
|
||||
pathIncludes: string;
|
||||
response: unknown;
|
||||
}>;
|
||||
minimumPreviewCycles?: number;
|
||||
expectations?: {
|
||||
sourceBaseUrl: string;
|
||||
sourceAuth: "none" | "apiKeyQuery";
|
||||
sourceAuthName?: string;
|
||||
requests: Array<{
|
||||
kind: "query" | "action";
|
||||
method: "GET" | "POST";
|
||||
pathIncludes: string;
|
||||
trigger?: "load" | "manual";
|
||||
queryIncludes?: Record<string, string>;
|
||||
}>;
|
||||
templateIncludes?: string[];
|
||||
};
|
||||
}
|
||||
|
||||
export const CUSTOM_WIDGET_AI_EVALUATION_CASES: readonly CustomWidgetAiEvaluationCase[] = [
|
||||
@@ -10,9 +29,49 @@ export const CUSTOM_WIDGET_AI_EVALUATION_CASES: readonly CustomWidgetAiEvaluatio
|
||||
id: "pokedex",
|
||||
documentationUrl: "https://pokeapi.co/docs/v2",
|
||||
request:
|
||||
"Create a polished Pokédex browser with searchable Pokémon, a responsive result grid, manual detail loading, sprites, types, abilities, base-stat progress bars, and clear loading, empty, and failure states.",
|
||||
"Create a complete Pokédex widget using PokéAPI. Make at least three deliberate refinement passes through validation and preview testing. It must have a substantial polished UI with a direct name lookup, a searchable loaded list that stays visible while details open in a separate responsive result area, sprites, types, abilities, base-stat progress bars, and clear initial, loading, empty, failure, and success states. Use a restrained accent and flexible tile sizing rather than repetitive nested cards or a fixed-height list. Completeness must come from tested useful interactions, not filler JSX.",
|
||||
apiNotes:
|
||||
"Use GET /api/v2/pokemon?limit=<option>&offset=<option> for the list and GET /api/v2/pokemon/{param:name} as a manual detail query. PokeAPI needs no authentication.",
|
||||
"Use GET /api/v2/pokemon?limit=<option>&offset=<option> for the list and GET /api/v2/pokemon/{param:name} as a manual detail query for both direct typed lookup and selected list items. Keep the list visible while rendering detail output. Prefer sprites.other.official-artwork.front_default from the tested detail response rather than parsing an ID from a list URL. PokeAPI needs no authentication.",
|
||||
previewResponses: [
|
||||
{
|
||||
pathIncludes: "/api/v2/pokemon/{param:name}",
|
||||
response: {
|
||||
id: 25,
|
||||
name: "pikachu",
|
||||
height: 4,
|
||||
weight: 60,
|
||||
sprites: { other: { "official-artwork": { front_default: "https://example.test/pikachu.png" } } },
|
||||
types: [{ slot: 1, type: { name: "electric" } }],
|
||||
abilities: [{ ability: { name: "static" }, is_hidden: false }],
|
||||
stats: [
|
||||
{ base_stat: 35, stat: { name: "hp" } },
|
||||
{ base_stat: 90, stat: { name: "speed" } },
|
||||
],
|
||||
},
|
||||
},
|
||||
{
|
||||
pathIncludes: "/api/v2/pokemon",
|
||||
response: {
|
||||
count: 1_302,
|
||||
next: "https://pokeapi.co/api/v2/pokemon?offset=20&limit=20",
|
||||
previous: null,
|
||||
results: [
|
||||
{ name: "bulbasaur", url: "https://pokeapi.co/api/v2/pokemon/1/" },
|
||||
{ name: "pikachu", url: "https://pokeapi.co/api/v2/pokemon/25/" },
|
||||
],
|
||||
},
|
||||
},
|
||||
],
|
||||
minimumPreviewCycles: 3,
|
||||
expectations: {
|
||||
sourceBaseUrl: "https://pokeapi.co",
|
||||
sourceAuth: "none",
|
||||
requests: [
|
||||
{ kind: "query", method: "GET", pathIncludes: "/api/v2/pokemon", trigger: "load" },
|
||||
{ kind: "query", method: "GET", pathIncludes: "/api/v2/pokemon/{param:name}", trigger: "manual" },
|
||||
],
|
||||
templateIncludes: ["TextInput", "SubFetch", "Image", "types", "abilities", "stats", "Progress"],
|
||||
},
|
||||
},
|
||||
{
|
||||
id: "portainer-containers",
|
||||
@@ -46,4 +105,157 @@ export const CUSTOM_WIDGET_AI_EVALUATION_CASES: readonly CustomWidgetAiEvaluatio
|
||||
apiNotes:
|
||||
"Use bearer auth. GET /api/states/{option:sensorEntity} and /api/states/{option:lightEntity} load entity state. POST /api/services/light/turn_on and /turn_off accept a body with entity_id from an option and should invalidate the light query.",
|
||||
},
|
||||
{
|
||||
id: "fake-service-health",
|
||||
documentationUrl: "https://status.example.test/docs",
|
||||
request:
|
||||
"Using the supplied fake API contract, create a polished service-health widget for a small homelab. Prioritize the overall state, incident count, latency, last check time, and a compact list of services. It must be immediately useful in narrow and wide dashboard tiles.",
|
||||
apiNotes:
|
||||
"This is a deterministic fake API. Use https://status.example.test with no authentication. GET /v1/health returns the exact sample response. Do not invent additional endpoints or actions.",
|
||||
sampleResponse: {
|
||||
status: "degraded",
|
||||
checkedAt: "2026-08-10T12:00:00Z",
|
||||
latencyMs: 184,
|
||||
openIncidents: 1,
|
||||
services: [
|
||||
{ id: "media", name: "Media", status: "operational", latencyMs: 42 },
|
||||
{ id: "photos", name: "Photos", status: "degraded", latencyMs: 331 },
|
||||
{ id: "dns", name: "DNS", status: "operational", latencyMs: 8 },
|
||||
],
|
||||
},
|
||||
expectations: {
|
||||
sourceBaseUrl: "https://status.example.test",
|
||||
sourceAuth: "none",
|
||||
requests: [{ kind: "query", method: "GET", pathIncludes: "/v1/health" }],
|
||||
templateIncludes: ["checkedAt", "latencyMs", "openIncidents", "services", "RefreshButton"],
|
||||
},
|
||||
},
|
||||
{
|
||||
id: "coinmarketcap-keyless",
|
||||
documentationUrl: "https://coinmarketcap.com/api/documentation/pro-api-reference/keyless-public-api",
|
||||
request:
|
||||
"Create a compact but premium cryptocurrency watchlist for Bitcoin, Ethereum, and Solana. Show price, 24-hour change, market cap, volume, clear positive/negative styling, refresh context, and excellent narrow-tile behavior.",
|
||||
apiNotes:
|
||||
"Use the current keyless CoinMarketCap public API. The source is https://pro-api.coinmarketcap.com with no authentication. GET /public-api/v3/cryptocurrency/quotes/latest with id=1,1027,5426 and convert=USD so similarly named symbols cannot add unrelated assets. The response object has a data array; each asset has name, symbol, is_active, cmc_rank, quote[0].price, quote[0].percent_change_24h, quote[0].market_cap, quote[0].volume_24h, and last_updated.",
|
||||
sampleResponse: {
|
||||
data: [
|
||||
{
|
||||
id: 1,
|
||||
name: "Bitcoin",
|
||||
symbol: "BTC",
|
||||
is_active: 1,
|
||||
cmc_rank: 1,
|
||||
quote: [
|
||||
{
|
||||
symbol: "USD",
|
||||
price: 118_240.42,
|
||||
percent_change_24h: 2.41,
|
||||
market_cap: 2_354_000_000_000,
|
||||
volume_24h: 48_700_000_000,
|
||||
},
|
||||
],
|
||||
last_updated: "2026-08-10T12:00:00Z",
|
||||
},
|
||||
{
|
||||
id: 1027,
|
||||
name: "Ethereum",
|
||||
symbol: "ETH",
|
||||
is_active: 1,
|
||||
cmc_rank: 2,
|
||||
quote: [
|
||||
{
|
||||
symbol: "USD",
|
||||
price: 4_210.18,
|
||||
percent_change_24h: -1.13,
|
||||
market_cap: 508_000_000_000,
|
||||
volume_24h: 21_300_000_000,
|
||||
},
|
||||
],
|
||||
last_updated: "2026-08-10T12:00:00Z",
|
||||
},
|
||||
{
|
||||
id: 5426,
|
||||
name: "Solana",
|
||||
symbol: "SOL",
|
||||
is_active: 1,
|
||||
cmc_rank: 6,
|
||||
quote: [
|
||||
{
|
||||
symbol: "USD",
|
||||
price: 181.72,
|
||||
percent_change_24h: 4.02,
|
||||
market_cap: 98_400_000_000,
|
||||
volume_24h: 5_900_000_000,
|
||||
},
|
||||
],
|
||||
last_updated: "2026-08-10T12:00:00Z",
|
||||
},
|
||||
],
|
||||
status: { timestamp: "2026-08-10T12:00:01Z", error_code: "0", error_message: "", elapsed: 6 },
|
||||
},
|
||||
expectations: {
|
||||
sourceBaseUrl: "https://pro-api.coinmarketcap.com",
|
||||
sourceAuth: "none",
|
||||
requests: [
|
||||
{
|
||||
kind: "query",
|
||||
method: "GET",
|
||||
pathIncludes: "/public-api/v3/cryptocurrency/quotes/latest",
|
||||
trigger: "load",
|
||||
queryIncludes: { id: "1,1027,5426", convert: "USD" },
|
||||
},
|
||||
],
|
||||
templateIncludes: ["?.data", "percent_change_24h", "market_cap", "volume_24h", "last_updated", "RefreshButton"],
|
||||
},
|
||||
},
|
||||
{
|
||||
id: "bored-activity",
|
||||
documentationUrl: "https://bored-api.appbrewery.com/",
|
||||
request:
|
||||
"Create a delightful activity discovery widget. It should load one random activity, make the suggestion and practical constraints easy to scan, link to the activity when available, and provide a safe manual way to fetch another suggestion. It must feel useful in both a small dashboard tile and a wide one.",
|
||||
apiNotes:
|
||||
"Use https://bored-api.appbrewery.com with no authentication. GET /random returns one object with activity, availability, type, participants, price, accessibility, duration, kidFriendly, link, and key. The service is rate limited, so use a sensible query cache and do not create duplicate load requests.",
|
||||
sampleResponse: {
|
||||
activity: "Learn Express.js",
|
||||
availability: 0.25,
|
||||
type: "education",
|
||||
participants: 1,
|
||||
price: 0.1,
|
||||
accessibility: "Few to no challenges",
|
||||
duration: "hours",
|
||||
kidFriendly: true,
|
||||
link: "https://expressjs.com/",
|
||||
key: "3943506",
|
||||
},
|
||||
expectations: {
|
||||
sourceBaseUrl: "https://bored-api.appbrewery.com",
|
||||
sourceAuth: "none",
|
||||
requests: [{ kind: "query", method: "GET", pathIncludes: "/random", trigger: "load" }],
|
||||
templateIncludes: ["activity", "participants", "price", "accessibility", "duration", "RefreshButton"],
|
||||
},
|
||||
},
|
||||
{
|
||||
id: "agify-name",
|
||||
documentationUrl: "https://agify.io/documentation/api/reference",
|
||||
request:
|
||||
"Create a polished age-estimation lookup widget powered by Agify. Let the user enter a full name, manually run the prediction, optionally scope it with a two-letter country code, and clearly explain the estimate and evidence count without presenting it as certainty. Include thoughtful initial, loading, no-result, error, and success states.",
|
||||
apiNotes:
|
||||
"Use https://api.agify.io. Authentication is an API key supplied through the source as apiKeyQuery with query parameter name apikey; never embed it. The manual GET / request receives name from {param:name} and an optional country_id from {param:country}. A successful response has name, age (integer or null), count, and optionally country_id.",
|
||||
sampleResponse: { name: "michael", age: 58, count: 108_496, country_id: "US" },
|
||||
expectations: {
|
||||
sourceBaseUrl: "https://api.agify.io",
|
||||
sourceAuth: "apiKeyQuery",
|
||||
sourceAuthName: "apikey",
|
||||
requests: [
|
||||
{
|
||||
kind: "query",
|
||||
method: "GET",
|
||||
pathIncludes: "/",
|
||||
trigger: "manual",
|
||||
queryIncludes: { name: "$param:name", country_id: "$param:country" },
|
||||
},
|
||||
],
|
||||
templateIncludes: ["TextInput", "SubFetch", "age", "count"],
|
||||
},
|
||||
},
|
||||
] as const;
|
||||
@@ -4,13 +4,16 @@ import path from "node:path";
|
||||
import { z } from "zod/v4";
|
||||
|
||||
import { buildCustomWidgetAiPrompt } from "../src/core/ai-prompt";
|
||||
import { getCustomWidgetSkillContent } from "../src/core/authoring-resources";
|
||||
import { customWidgetDefinitionSchema } from "../src/core/custom-jsx-schema";
|
||||
import type { HomarrCustomWidgetV2 } from "../src/core/custom-jsx-schema";
|
||||
import { formatCustomWidgetImportIssues, parseCustomWidgetAiResponse } from "../src/core/import";
|
||||
import type { CustomWidgetAiEvaluationCase } from "./ai-evaluation-cases";
|
||||
|
||||
export const DEFAULT_GENERATOR_MODEL = "deepseek/deepseek-v4-pro";
|
||||
export const DEFAULT_JUDGE_MODEL = "deepseek/deepseek-v4-flash";
|
||||
// OpenRouter exposes the rolling "latest" route with a leading tilde. The non-tilde
|
||||
// deepseek/deepseek-v4-flash-latest alias is rejected by the chat-completions API.
|
||||
export const DEFAULT_GENERATOR_MODEL = "~deepseek/deepseek-v4-flash-latest";
|
||||
export const DEFAULT_JUDGE_MODEL = "~deepseek/deepseek-v4-flash-latest";
|
||||
export const MAX_AI_EVALUATION_LOOPS = 10;
|
||||
|
||||
const scoreSchema = z.number().int().min(0).max(100);
|
||||
@@ -85,13 +88,101 @@ interface OpenRouterResponse {
|
||||
export function buildEvaluationPrompt(testCase: CustomWidgetAiEvaluationCase): string {
|
||||
return buildCustomWidgetAiPrompt(
|
||||
undefined,
|
||||
null,
|
||||
getEvaluationResponseFixtureText(testCase),
|
||||
null,
|
||||
`${testCase.request}\n\nVerified API notes:\n${testCase.apiNotes}`,
|
||||
testCase.documentationUrl,
|
||||
);
|
||||
}
|
||||
|
||||
export function getEvaluationResponseFixtureText(testCase: CustomWidgetAiEvaluationCase): string | null {
|
||||
if (testCase.previewResponses?.length) {
|
||||
return testCase.previewResponses
|
||||
.map(({ pathIncludes, response }) => `${pathIncludes}:\n${JSON.stringify(response, null, 2)}`)
|
||||
.join("\n\n");
|
||||
}
|
||||
return testCase.sampleResponse === undefined ? null : JSON.stringify(testCase.sampleResponse, null, 2);
|
||||
}
|
||||
|
||||
const getExpectedBindingValue = (value: unknown): string => {
|
||||
if (typeof value === "string") return value;
|
||||
if (typeof value !== "object" || value === null || Array.isArray(value)) return JSON.stringify(value);
|
||||
if ("$param" in value && typeof value.$param === "string") return `$param:${value.$param}`;
|
||||
if ("$option" in value && typeof value.$option === "string") return `$option:${value.$option}`;
|
||||
return JSON.stringify(value);
|
||||
};
|
||||
|
||||
export function getDeterministicEvaluationIssues(
|
||||
testCase: CustomWidgetAiEvaluationCase,
|
||||
widget: HomarrCustomWidgetV2,
|
||||
): Array<{ path?: Array<string | number>; message: string }> {
|
||||
const expectations = testCase.expectations;
|
||||
if (!expectations) return [];
|
||||
const issues: Array<{ path?: Array<string | number>; message: string }> = [];
|
||||
const source = widget.sources.default;
|
||||
if (!source || source.baseUrl !== expectations.sourceBaseUrl) {
|
||||
issues.push({
|
||||
path: ["sources", "default", "baseUrl"],
|
||||
message: `Use the verified source URL ${expectations.sourceBaseUrl}.`,
|
||||
});
|
||||
}
|
||||
const authType = typeof source?.auth === "string" ? source.auth : source?.auth.type;
|
||||
if (authType !== expectations.sourceAuth) {
|
||||
issues.push({
|
||||
path: ["sources", "default", "auth"],
|
||||
message: `Use the verified ${expectations.sourceAuth} authentication mode.`,
|
||||
});
|
||||
}
|
||||
if (
|
||||
expectations.sourceAuth === "apiKeyQuery" &&
|
||||
typeof source?.auth === "object" &&
|
||||
source.auth.name !== expectations.sourceAuthName
|
||||
) {
|
||||
issues.push({
|
||||
path: ["sources", "default", "auth", "name"],
|
||||
message: `Use '${expectations.sourceAuthName}' as the verified API-key query parameter.`,
|
||||
});
|
||||
}
|
||||
const matchedRequestIds = new Set<string>();
|
||||
for (const expected of expectations.requests) {
|
||||
const match = Object.entries(widget.requests).find(([requestId, request]) => {
|
||||
if (matchedRequestIds.has(requestId)) return false;
|
||||
if (
|
||||
request.kind !== expected.kind ||
|
||||
request.method !== expected.method ||
|
||||
!request.path.includes(expected.pathIncludes) ||
|
||||
(expected.trigger !== undefined && request.trigger !== expected.trigger)
|
||||
) {
|
||||
return false;
|
||||
}
|
||||
return Object.entries(expected.queryIncludes ?? {}).every(
|
||||
([key, value]) => getExpectedBindingValue(request.query?.[key]) === value,
|
||||
);
|
||||
});
|
||||
if (!match) {
|
||||
issues.push({
|
||||
path: ["requests"],
|
||||
message:
|
||||
`Define the verified ${expected.trigger ?? ""} ${expected.kind} request ${expected.method} ${expected.pathIncludes} with its documented query bindings.`.replace(
|
||||
/\s+/gu,
|
||||
" ",
|
||||
),
|
||||
});
|
||||
} else {
|
||||
matchedRequestIds.add(match[0]);
|
||||
}
|
||||
}
|
||||
for (const requiredText of expectations.templateIncludes ?? []) {
|
||||
if (!widget.template.includes(requiredText)) {
|
||||
issues.push({
|
||||
path: ["template"],
|
||||
message: `Render or use the required '${requiredText}' capability from the verified response and request.`,
|
||||
});
|
||||
}
|
||||
}
|
||||
return issues;
|
||||
}
|
||||
|
||||
export function buildRepairPrompt(
|
||||
originalPrompt: string,
|
||||
previousResponse: string,
|
||||
@@ -106,12 +197,20 @@ export function buildRepairPrompt(
|
||||
export function buildJudgePrompt(testCase: CustomWidgetAiEvaluationCase, widget: HomarrCustomWidgetV2): string {
|
||||
return `You are a hostile-but-fair product review panel evaluating a safe dashboard widget. Most competent drafts should score 55-75, not 90. Judge only evidence present in the manifest and JSX. Never reward unsupported capabilities, invented API routes, aspirational claims, or code that merely validates.
|
||||
|
||||
The installed Homarr skill and runtime references below are authoritative. In particular, request state is exposed as status.<requestId> with loading/ok/status/error fields while successful payloads are exposed as data.<requestId>. RefreshButton is an installed runtime helper that manually refreshes load queries. Every component in this already-validated template exists in the installed release. Do not penalize those documented facts. The widget has already passed Homarr's real schema and JSX analyzer, which proves syntax and component compatibility but does not prove API correctness, visual quality, usefulness, or accessibility.
|
||||
|
||||
Installed Homarr Custom Widget authoring contract:
|
||||
${getCustomWidgetSkillContent()}
|
||||
|
||||
Request:
|
||||
${testCase.request}
|
||||
|
||||
Verified API notes:
|
||||
${testCase.apiNotes}
|
||||
|
||||
Representative API response used by the preview test:
|
||||
${getEvaluationResponseFixtureText(testCase) ?? "Not supplied."}
|
||||
|
||||
Validated widget:
|
||||
${JSON.stringify(widget, null, 2)}
|
||||
|
||||
@@ -128,8 +227,10 @@ Required review behavior:
|
||||
- Judge visual quality, not component count: hierarchy, density, whitespace, typography, restrained color, scanability, and avoidance of repetitive nested cards.
|
||||
- Judge daily usefulness: information priority, interaction cost, refresh behavior, narrow-tile usability, and whether the widget is pleasant rather than demo-like.
|
||||
- Judge complexity discipline: penalize duplicate requests/options, unnecessary controls, excessive JSX, cleverness, and UI chrome that does not help the goal. Complexity must earn its place.
|
||||
- Recommend only interactions supported by the installed authoring contract. Do not suggest portals, modals, arbitrary event handlers, or other blocked capabilities. Prefer a responsive in-widget detail area when separation is useful.
|
||||
- A visually generic but valid widget should normally score below 75 for visualQuality. A widget that is attractive but inconvenient should score below 75 for dailyUsefulness.
|
||||
- Give a concrete evidence sentence for every category. List all score-capping issues under fatalProblems.
|
||||
- Return an empty fatalProblems array when there are no fatal problems; never put "none" or an explanation of their absence in that array.
|
||||
|
||||
Homarr computes the weighted total and final verdict itself. Your total and verdict are advisory, but must be internally honest. A pass requires a weighted total of at least 90, every category at least 75, goalFulfillment/visualQuality/dailyUsefulness at least 85, complexityDiscipline at least 80, no fatal problem, and dailyUseDecision="would-use-daily". Return only the requested structured object.`;
|
||||
}
|
||||
@@ -158,6 +259,30 @@ export function getJudgeResponseFormat() {
|
||||
};
|
||||
}
|
||||
|
||||
export async function requestCustomWidgetJudge(args: {
|
||||
testCase: CustomWidgetAiEvaluationCase;
|
||||
widget: HomarrCustomWidgetV2;
|
||||
apiKey: string;
|
||||
judgeModel?: string;
|
||||
}) {
|
||||
return callOpenRouter({
|
||||
apiKey: args.apiKey,
|
||||
model: args.judgeModel ?? DEFAULT_JUDGE_MODEL,
|
||||
prompt: buildJudgePrompt(args.testCase, args.widget),
|
||||
purpose: "judge",
|
||||
});
|
||||
}
|
||||
|
||||
export async function judgeCustomWidgetCase(args: {
|
||||
testCase: CustomWidgetAiEvaluationCase;
|
||||
widget: HomarrCustomWidgetV2;
|
||||
apiKey: string;
|
||||
judgeModel?: string;
|
||||
}) {
|
||||
const raw = await requestCustomWidgetJudge(args);
|
||||
return { raw, result: parseJudgeResult(raw) };
|
||||
}
|
||||
|
||||
export async function evaluateCustomWidgetCase(args: {
|
||||
testCase: CustomWidgetAiEvaluationCase;
|
||||
apiKey: string;
|
||||
@@ -200,17 +325,25 @@ export async function evaluateCustomWidgetCase(args: {
|
||||
}
|
||||
|
||||
const canonical = customWidgetDefinitionSchema.parse(parsed.widget);
|
||||
const deterministicIssues = getDeterministicEvaluationIssues(args.testCase, canonical);
|
||||
if (deterministicIssues.length > 0) {
|
||||
errors.push(
|
||||
`Attempt ${attempt}: deterministic scenario checks failed — ${deterministicIssues.map((issue) => issue.message).join("; ")}`,
|
||||
);
|
||||
prompt = buildRepairPrompt(originalPrompt, response, deterministicIssues);
|
||||
continue;
|
||||
}
|
||||
await writeWidgetFiles(caseDirectory, canonical, `attempt-${attempt}`);
|
||||
let judge: CustomWidgetJudgeResult;
|
||||
try {
|
||||
const judgeRaw = await callOpenRouter({
|
||||
const { raw: judgeRaw, result } = await judgeCustomWidgetCase({
|
||||
testCase: args.testCase,
|
||||
widget: canonical,
|
||||
apiKey: args.apiKey,
|
||||
model: args.judgeModel ?? DEFAULT_JUDGE_MODEL,
|
||||
prompt: buildJudgePrompt(args.testCase, canonical),
|
||||
purpose: "judge",
|
||||
judgeModel: args.judgeModel,
|
||||
});
|
||||
await writeFile(path.join(caseDirectory, `judge-${attempt}.json`), judgeRaw, "utf8");
|
||||
judge = parseJudgeResult(judgeRaw);
|
||||
judge = result;
|
||||
} catch (error) {
|
||||
errors.push(
|
||||
`Attempt ${attempt}: judge response invalid — ${error instanceof Error ? error.message : "Unknown error"}`,
|
||||
@@ -253,14 +386,22 @@ export async function evaluateCustomWidgetCase(args: {
|
||||
}
|
||||
|
||||
export function parseJudgeResult(raw: string): CustomWidgetJudgeResult {
|
||||
const parsed: unknown = JSON.parse(raw);
|
||||
const normalizedRaw = raw
|
||||
.trim()
|
||||
.replace(/^```(?:json)?\s*/iu, "")
|
||||
.replace(/\s*```$/u, "");
|
||||
const parsed: unknown = JSON.parse(normalizedRaw);
|
||||
const result = judgeResultSchema.parse(parsed);
|
||||
const fatalProblems = result.fatalProblems.filter(
|
||||
(problem) => !/^(?:none|no fatal problems?)(?:\b|\s*[:.-])/iu.test(problem.trim()),
|
||||
);
|
||||
const weightedTotal = Object.entries(categoryWeights).reduce(
|
||||
(sum, [category, weight]) => sum + result.categories[category as keyof typeof categoryWeights] * weight,
|
||||
0,
|
||||
);
|
||||
const normalized = {
|
||||
...result,
|
||||
fatalProblems,
|
||||
total: Math.round(weightedTotal / 100),
|
||||
};
|
||||
return { ...normalized, verdict: judgePasses(normalized) ? "pass" : "fail" };
|
||||
@@ -290,7 +431,7 @@ async function callOpenRouter(args: {
|
||||
messages: [{ role: "user", content: args.prompt }],
|
||||
temperature: isJudge ? 0 : 0.2,
|
||||
max_tokens: isJudge ? 8_000 : 20_000,
|
||||
reasoning: { effort: "xhigh", exclude: true },
|
||||
reasoning: isJudge ? { enabled: false, exclude: true } : { effort: "high", exclude: true },
|
||||
...(isJudge ? { response_format: getJudgeResponseFormat() } : {}),
|
||||
}),
|
||||
signal: AbortSignal.timeout(180_000),
|
||||
|
||||
@@ -2,6 +2,8 @@ import { mkdir, writeFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
|
||||
import { CUSTOM_WIDGET_AI_EVALUATION_CASES } from "./ai-evaluation-cases";
|
||||
import { evaluateCustomWidgetAssistantCase } from "./ai-assistant-evaluation";
|
||||
import type { CustomWidgetAssistantEvaluationResult } from "./ai-assistant-evaluation";
|
||||
import { DEFAULT_GENERATOR_MODEL, DEFAULT_JUDGE_MODEL, evaluateCustomWidgetCase } from "./ai-evaluation";
|
||||
import type { AiEvaluationResult } from "./ai-evaluation";
|
||||
|
||||
@@ -9,11 +11,19 @@ const apiKey = process.env.OPENROUTER_API_KEY;
|
||||
if (!apiKey) throw new Error("OPENROUTER_API_KEY is required for the live Custom Widget AI evaluation");
|
||||
|
||||
const requestedCase = process.argv.find((value) => value.startsWith("--case="))?.slice("--case=".length);
|
||||
const assistantMode = process.argv.includes("--assistant");
|
||||
const configuredLoops = Number(process.env.CUSTOM_WIDGET_AI_MAX_LOOPS ?? 10);
|
||||
const totalLoopBudget = Number.isInteger(configuredLoops) && configuredLoops > 0 ? Math.min(configuredLoops, 10) : 10;
|
||||
const selectableCases = assistantMode
|
||||
? CUSTOM_WIDGET_AI_EVALUATION_CASES.filter(
|
||||
(testCase) =>
|
||||
(testCase.sampleResponse !== undefined || testCase.previewResponses?.length) &&
|
||||
testCase.expectations !== undefined,
|
||||
)
|
||||
: CUSTOM_WIDGET_AI_EVALUATION_CASES;
|
||||
const selectedCases = requestedCase
|
||||
? CUSTOM_WIDGET_AI_EVALUATION_CASES.filter((testCase) => testCase.id === requestedCase)
|
||||
: CUSTOM_WIDGET_AI_EVALUATION_CASES;
|
||||
: selectableCases;
|
||||
if (selectedCases.length === 0) throw new Error(`Unknown AI evaluation case '${requestedCase}'`);
|
||||
if (totalLoopBudget < selectedCases.length) {
|
||||
throw new Error(`CUSTOM_WIDGET_AI_MAX_LOOPS must be at least ${selectedCases.length} for this run`);
|
||||
@@ -23,20 +33,29 @@ const runId = new Date().toISOString().replaceAll(/[:.]/gu, "-");
|
||||
const outputRoot = path.resolve(process.cwd(), ".ai-evaluations", runId);
|
||||
await mkdir(outputRoot, { recursive: true });
|
||||
|
||||
const results: AiEvaluationResult[] = [];
|
||||
const results: Array<AiEvaluationResult | CustomWidgetAssistantEvaluationResult> = [];
|
||||
let remainingLoopBudget = totalLoopBudget;
|
||||
for (const testCase of selectedCases) {
|
||||
const remainingCases = selectedCases.length - results.length;
|
||||
const maxLoops = Math.max(1, Math.floor(remainingLoopBudget / remainingCases));
|
||||
process.stdout.write(`Evaluating ${testCase.id}...\n`);
|
||||
const result = await evaluateCustomWidgetCase({
|
||||
testCase,
|
||||
apiKey,
|
||||
outputRoot,
|
||||
maxLoops,
|
||||
generatorModel: process.env.OPENROUTER_GENERATOR_MODEL ?? DEFAULT_GENERATOR_MODEL,
|
||||
judgeModel: process.env.OPENROUTER_JUDGE_MODEL ?? DEFAULT_JUDGE_MODEL,
|
||||
});
|
||||
const result = assistantMode
|
||||
? await evaluateCustomWidgetAssistantCase({
|
||||
testCase,
|
||||
apiKey,
|
||||
outputRoot,
|
||||
maxLoops,
|
||||
generatorModel: process.env.OPENROUTER_GENERATOR_MODEL ?? DEFAULT_GENERATOR_MODEL,
|
||||
judgeModel: process.env.OPENROUTER_JUDGE_MODEL ?? DEFAULT_JUDGE_MODEL,
|
||||
})
|
||||
: await evaluateCustomWidgetCase({
|
||||
testCase,
|
||||
apiKey,
|
||||
outputRoot,
|
||||
maxLoops,
|
||||
generatorModel: process.env.OPENROUTER_GENERATOR_MODEL ?? DEFAULT_GENERATOR_MODEL,
|
||||
judgeModel: process.env.OPENROUTER_JUDGE_MODEL ?? DEFAULT_JUDGE_MODEL,
|
||||
});
|
||||
results.push(result);
|
||||
remainingLoopBudget -= result.attempts;
|
||||
process.stdout.write(
|
||||
@@ -48,6 +67,7 @@ for (const testCase of selectedCases) {
|
||||
|
||||
const summary = {
|
||||
generatedAt: new Date().toISOString(),
|
||||
mode: assistantMode ? "assistant-tool-loop" : "manifest",
|
||||
generatorModel: process.env.OPENROUTER_GENERATOR_MODEL ?? DEFAULT_GENERATOR_MODEL,
|
||||
judgeModel: process.env.OPENROUTER_JUDGE_MODEL ?? DEFAULT_JUDGE_MODEL,
|
||||
results: results.map((result) => ({
|
||||
@@ -57,6 +77,7 @@ const summary = {
|
||||
verdict: result.judge?.verdict ?? "fail",
|
||||
categories: result.judge?.categories ?? null,
|
||||
errors: result.errors,
|
||||
calledTools: "calledTools" in result ? result.calledTools : undefined,
|
||||
outputDirectory: path.relative(process.cwd(), result.outputDirectory),
|
||||
})),
|
||||
};
|
||||
|
||||
@@ -2,6 +2,11 @@ import { customJsxRequestSchema } from "../src/core/request-schema";
|
||||
import { executeCustomWidgetRequest } from "../src/server/request-executor";
|
||||
import { renderRequestTarget, resolveCustomWidgetRequestValues } from "../src/server/request-manifest";
|
||||
|
||||
if (process.argv.includes("--public-ai-apis-only")) {
|
||||
await verifyPublicAiEvaluationApis();
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
const pokemonListRequest = customJsxRequestSchema.parse({
|
||||
path: "/api/v2/pokemon",
|
||||
query: { limit: { $option: "limit" }, offset: 0 },
|
||||
@@ -136,3 +141,52 @@ async function runPortainerAction(action: string, containerId: string, ignoreCon
|
||||
throw new Error(`Portainer ${action} returned HTTP ${response.status}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function verifyPublicAiEvaluationApis() {
|
||||
const bored = await fetchJson(new URL("https://bored-api.appbrewery.com/random"), {});
|
||||
if (!isRecord(bored) || typeof bored.activity !== "string" || typeof bored.participants !== "number") {
|
||||
throw new Error("Bored response did not match the evaluation fixture contract");
|
||||
}
|
||||
process.stdout.write("Bored activity request passed\n");
|
||||
|
||||
const coinMarketCapRequest = customJsxRequestSchema.parse({
|
||||
path: "/public-api/v3/cryptocurrency/quotes/latest",
|
||||
query: { id: "1,1027,5426", convert: "USD" },
|
||||
});
|
||||
const coinMarketCapTarget = renderRequestTarget(
|
||||
"https://pro-api.coinmarketcap.com",
|
||||
coinMarketCapRequest,
|
||||
resolveCustomWidgetRequestValues(coinMarketCapRequest, {}),
|
||||
);
|
||||
const response = await fetchJson(coinMarketCapTarget, {});
|
||||
const assets = isRecord(response) ? response.data : undefined;
|
||||
if (
|
||||
!Array.isArray(assets) ||
|
||||
!["BTC", "ETH", "SOL"].every((symbol) =>
|
||||
assets.some((asset) => isRecord(asset) && asset.symbol === symbol && Array.isArray(asset.quote)),
|
||||
)
|
||||
) {
|
||||
throw new Error("CoinMarketCap response did not match the evaluation fixture contract");
|
||||
}
|
||||
process.stdout.write("CoinMarketCap keyless watchlist request passed\n");
|
||||
|
||||
const agifyApiKey = process.env.AGIFY_API_KEY;
|
||||
if (!agifyApiKey) {
|
||||
process.stdout.write("Agify live request skipped (AGIFY_API_KEY is not configured)\n");
|
||||
return;
|
||||
}
|
||||
const agifyTarget = new URL("https://api.agify.io/");
|
||||
agifyTarget.searchParams.set("name", "michael");
|
||||
agifyTarget.searchParams.set("country_id", "US");
|
||||
agifyTarget.searchParams.set("apikey", agifyApiKey);
|
||||
const prediction = await fetchJson(agifyTarget, {});
|
||||
if (
|
||||
!isRecord(prediction) ||
|
||||
typeof prediction.name !== "string" ||
|
||||
!(typeof prediction.age === "number" || prediction.age === null) ||
|
||||
typeof prediction.count !== "number"
|
||||
) {
|
||||
throw new Error("Agify response did not match the evaluation fixture contract");
|
||||
}
|
||||
process.stdout.write("Agify authenticated lookup request passed\n");
|
||||
}
|
||||
@@ -111,10 +111,14 @@ Recommended components: ${RECOMMENDED_COMPONENTS.join(", ")}. This list is not e
|
||||
|
||||
Make the result genuinely attractive: establish clear visual hierarchy, use deliberate spacing, restrained semantic color, responsive layouts, and theme-safe colors. Prefer one strong primary surface over excessive nested cards. Include useful loading, empty, error, and success states. Make narrow and wide tiles both work.
|
||||
|
||||
Treat a supplied sample response as an executable contract. Render every core field requested by the user, guard optional arrays and nested values before indexing them, and do not silently drop sample items. When the response includes a timestamp, show concise freshness context. Pair recoverable load errors and empty states with a clear refresh or retry path.
|
||||
|
||||
Visual quality bar:
|
||||
- Give the widget a purposeful header with title, useful context, and its primary status or action.
|
||||
- When the data supports it, lead with 2–4 scannable summary metrics before detailed rows.
|
||||
- Use responsive SimpleGrid/Grid column objects and let long content wrap on narrow tiles.
|
||||
- Make the initial state actionable with an example, useful hint, or clear next step. Use wrapping groups for variable-length labels and values on narrow tiles.
|
||||
- Avoid unlabeled decorative icons in empty states. Pair an icon with visible explanatory text or give an interactive standalone icon an accessible label.
|
||||
- Use theme tokens and semantic Mantine colors; avoid hard-coded light backgrounds and decorative gradients.
|
||||
|
||||
${compactExample(0)}
|
||||
@@ -129,12 +133,20 @@ export const CUSTOM_WIDGET_MCP_AUTHORING_PROMPT = `Author one Homarr Custom JSX
|
||||
|
||||
First call customWidget_getSkill. Its single lazy response contains the complete installed SKILL.md and every bundled reference; follow all of it. Read the live schema and compact component catalog. Fetch only the named component and example resources needed for the design. Component resources contain component-specific props; if shared props are needed, collect their catalog names and fetch those docs together once with customWidget_getSharedProps. The live resources are authoritative for this Homarr release.
|
||||
|
||||
Keep the documentation phase bounded. Fetch at most eight named component documents before the first validation, or at most four after loading a complete example. Fetch another component only to resolve a concrete validation issue or missing interaction; never walk the catalog component-by-component. Move promptly from the smallest sufficient documentation set to a complete customWidget_validate call.
|
||||
|
||||
Follow this lifecycle without skipping steps: construct one credential-free definition; call customWidget_validate; repair every reported issue; call customWidget_previewCreate; call customWidget_previewQuery once for every query returned by that preview (supplying required manual parameters); inspect the real response shape and HTTP status; adjust paths and template bindings; validate and preview again after changes; open or provide the returned previewPath for visual review; only then call customWidget_createFromPreview with the exact final tested preview session ID. Use customWidget_create only when no reusable preview session exists. Prefer templateLines over a JSON-escaped template string for multiline JSX in validate and preview tool inputs.
|
||||
|
||||
Do not claim a query works when it has not returned a successful preview response. Configure deployment-specific source URLs and request credentials through Homarr when needed; never repeat plaintext. Use simulated preview actions unless the user explicitly enables live preview actions.
|
||||
|
||||
Treat preview data as the binding contract: render every core requested field, guard optional arrays and nested values before indexing, and do not silently drop returned items. Show concise freshness context when the response supplies a timestamp. Give recoverable load errors and empty states a clear refresh or retry path.
|
||||
|
||||
Use the lean keyed sources, requests, and options contract. Bind saved values directly with $option and invocation values with $param. Load queries cannot use $param. Keep the design responsive, accessible, theme-safe, loading-aware, empty-aware, and error-aware.
|
||||
|
||||
Create distinctive polish with one semantic accent, clear surface contrast, purposeful typography, and compact narrow-tile rows rather than decorative gradients or excessive nested cards. Label standalone icons for assistive technology or pair them with visible text.
|
||||
|
||||
Make initial states actionable with an example, useful hint, or clear next step. Use wrapping groups for variable-length labels and values on narrow tiles. Do not use an unlabeled decorative icon as an empty state.
|
||||
|
||||
Use Workshop search/get/install when the user asks for an existing community widget. After installation, configure self-hosted source URLs and credentials with the dedicated source configuration tool or a secure user configuration request.`;
|
||||
|
||||
export function buildCustomWidgetMcpPrompt(request?: string | null, documentationUrl?: string | null) {
|
||||
|
||||
@@ -74,18 +74,26 @@ description: Author safe Homarr Custom JSX v2 widgets from API documentation.
|
||||
|
||||
Author one widget at a time. When connected to Homarr, retrieve \`homarr://custom-widgets/schema\` and only the component resources needed for the design. The live resources are authoritative for the installed release.
|
||||
|
||||
Keep documentation retrieval targeted: fetch at most eight named component documents before the first validation, or at most four after loading a complete example. Fetch another component only when a validation issue or a concrete missing interaction requires it; never load the catalog component-by-component.
|
||||
|
||||
1. Read the API documentation.
|
||||
2. Create one credential-free widget with keyed \`sources\`, keyed \`requests\`, optional keyed \`options\`, and safe JSX \`template\`.
|
||||
3. Validate, preview, test queries and simulated actions, visually inspect, and iterate.
|
||||
4. Configure deployment-specific source URLs and request credentials through Homarr; never repeat plaintext.
|
||||
5. Save only after narrow/wide, light/dark, loading, empty, error, and success states work. When connected through MCP, persist the exact final tested preview with \`customWidget_createFromPreview\`; do not resend a large template through \`customWidget_create\` unless no preview session is available.
|
||||
|
||||
Treat a supplied sample or successful preview response as the binding contract. Render every core requested field, guard optional arrays and nested values before indexing, and do not silently drop returned items. Show concise freshness context when the API supplies a timestamp. Give recoverable load errors and empty states a clear refresh or retry path.
|
||||
|
||||
Use \`{option:name}\` or \`$option\` for saved options. Use \`{param:name}\` or \`$param\` for values supplied by \`SubFetch\`, \`ActionButton\`, or \`ToggleSwitch\`. Load queries cannot use invocation parameters. Templates read \`data\`, \`status\`, \`options\`, and temporary \`inputs\`.
|
||||
|
||||
\`SubFetch\` with \`trigger="manual"\` renders its own load button. Pass a card or image through \`triggerContent\` with \`triggerAriaLabel\` when that content should launch the request. Its callback is \`(result, meta)\`; never author \`onClick\` or a fetch callback.
|
||||
|
||||
Use standard Mantine compound names and deliberate visual hierarchy, spacing, restrained semantic color, and responsive layouts. Avoid excessive nested cards.
|
||||
|
||||
Aim for distinctive polish without decorative gradients or fragile custom styling: use one semantic accent, clear surface contrast, purposeful typography, and compact rows that remain scannable in narrow tiles. Give standalone icons an accessible label or pair them with visible text.
|
||||
|
||||
Make initial states actionable with an example, useful hint, or clear next step. Use wrapping groups for variable-length labels and values on narrow tiles. Do not use an unlabeled decorative icon as an empty state.
|
||||
|
||||
Do not use imports, hooks, refs, raw HTML, raw event callbacks, browser requests, arbitrary functions, eval, bigint, npm packages, authored statement blocks, IIFEs, or recursion. Do not pretend MCP tools exist in an offline chat session.
|
||||
|
||||
To install an existing widget, use \`customWidget_workshopSearch\`, inspect it with \`customWidget_workshopGet\`, then use \`customWidget_workshopInstall\`. Configure each self-hosted source afterward with \`customWidget_sourceConfigure\`, or call \`customWidget_configurationRequestUser\` so the user can enter the server URL and credentials directly in Homarr.
|
||||
|
||||
@@ -0,0 +1,189 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { CUSTOM_WIDGET_AI_EVALUATION_CASES } from "../../scripts/ai-evaluation-cases";
|
||||
import {
|
||||
createAssistantEvaluationState,
|
||||
customWidgetAssistantEvaluationToolDefinitions,
|
||||
executeAssistantEvaluationTool,
|
||||
getAssistantEvaluationPreviewResponse,
|
||||
getAssistantEvaluationRecoveryToolName,
|
||||
getRequiredAssistantEvaluationRequestParams,
|
||||
getAssistantEvaluationLifecycleIssues,
|
||||
} from "../../scripts/ai-assistant-evaluation";
|
||||
|
||||
const widget = {
|
||||
$schema: "homarr-custom-widget-v2",
|
||||
name: "Homelab health",
|
||||
description: "Compact health overview",
|
||||
sources: {
|
||||
default: {
|
||||
baseUrl: "https://status.example.test",
|
||||
networkScope: "public",
|
||||
auth: "none",
|
||||
},
|
||||
},
|
||||
requests: {
|
||||
health: {
|
||||
source: "default",
|
||||
kind: "query",
|
||||
method: "GET",
|
||||
path: "/v1/health",
|
||||
trigger: "load",
|
||||
auth: "inherit",
|
||||
permission: "view",
|
||||
cacheSeconds: 30,
|
||||
},
|
||||
},
|
||||
options: {},
|
||||
templateLines: [
|
||||
'<Stack gap="sm">',
|
||||
' <Group justify="space-between"><Text fw={700}>Homelab health</Text><RefreshButton /></Group>',
|
||||
" {status.health?.loading ? <Skeleton h={64} /> : null}",
|
||||
' {status.health?.ok === false ? <Alert color="red">Health data is unavailable.</Alert> : null}',
|
||||
' {data.health ? <Stack><Text fw={700}>{data.health.status}</Text><Text>{data.health.openIncidents} incidents · {data.health.latencyMs} ms · {data.health.checkedAt}</Text>{(data.health.services ?? []).map(service => <Text key={service.id}>{service.name}: {service.status} · {service.latencyMs} ms</Text>)}</Stack> : <Text c="dimmed">No health data</Text>}',
|
||||
"</Stack>",
|
||||
],
|
||||
};
|
||||
|
||||
describe("Custom Widget assistant live evaluation harness", () => {
|
||||
it("exposes the complete documentation and authoring lifecycle", () => {
|
||||
expect(customWidgetAssistantEvaluationToolDefinitions.map(({ function: definition }) => definition.name)).toEqual(
|
||||
expect.arrayContaining([
|
||||
"customWidget_getSkill",
|
||||
"customWidget_schema",
|
||||
"customWidget_getComponentCatalog",
|
||||
"customWidget_validate",
|
||||
"customWidget_previewCreate",
|
||||
"customWidget_previewQuery",
|
||||
"customWidget_createFromPreview",
|
||||
]),
|
||||
);
|
||||
});
|
||||
|
||||
it("recovers a stalled model at the next required evidence checkpoint", () => {
|
||||
const state = createAssistantEvaluationState();
|
||||
expect(getAssistantEvaluationRecoveryToolName(state)).toBe("customWidget_getSkill");
|
||||
state.calledTools.push("customWidget_getSkill", "customWidget_schema", "customWidget_getComponentCatalog");
|
||||
expect(getAssistantEvaluationRecoveryToolName(state)).toBe("customWidget_validate");
|
||||
});
|
||||
|
||||
it("rejects skipped evidence and persists the exact fixture-tested definition", () => {
|
||||
const testCase = CUSTOM_WIDGET_AI_EVALUATION_CASES.find(({ id }) => id === "fake-service-health");
|
||||
if (!testCase) throw new Error("Fake service-health case is missing");
|
||||
const state = createAssistantEvaluationState();
|
||||
|
||||
expect(
|
||||
executeAssistantEvaluationTool(testCase, state, "customWidget_previewCreate", { definition: widget }),
|
||||
).toEqual({ error: "Validate this exact complete definition before creating its preview." });
|
||||
executeAssistantEvaluationTool(testCase, state, "customWidget_getSkill", {});
|
||||
executeAssistantEvaluationTool(testCase, state, "customWidget_schema", {});
|
||||
executeAssistantEvaluationTool(testCase, state, "customWidget_getComponentCatalog", {});
|
||||
expect(executeAssistantEvaluationTool(testCase, state, "customWidget_validate", { widget })).toMatchObject({
|
||||
valid: true,
|
||||
summary: { requestIds: ["health"] },
|
||||
});
|
||||
expect(
|
||||
executeAssistantEvaluationTool(testCase, state, "customWidget_previewCreate", { definition: widget }),
|
||||
).toMatchObject({
|
||||
success: true,
|
||||
previewSession: { id: "preview-1" },
|
||||
queries: [{ requestId: "health" }],
|
||||
});
|
||||
expect(
|
||||
executeAssistantEvaluationTool(testCase, state, "customWidget_createFromPreview", {
|
||||
previewSessionId: "preview-1",
|
||||
}),
|
||||
).toEqual({ error: "Test every preview query before creation: health" });
|
||||
expect(
|
||||
executeAssistantEvaluationTool(testCase, state, "customWidget_previewQuery", {
|
||||
sessionId: "preview-1",
|
||||
requestId: "health",
|
||||
params: {},
|
||||
}),
|
||||
).toMatchObject({ ok: true, status: 200, data: testCase.sampleResponse });
|
||||
expect(
|
||||
executeAssistantEvaluationTool(testCase, state, "customWidget_createFromPreview", {
|
||||
previewSessionId: "preview-1",
|
||||
}),
|
||||
).toMatchObject({ id: "created-fake-service-health" });
|
||||
expect(state.createdWidget).toMatchObject({ name: widget.name, template: widget.templateLines.join("\n") });
|
||||
expect(getAssistantEvaluationLifecycleIssues(state)).toEqual([]);
|
||||
});
|
||||
|
||||
it("rejects excess parallel component lookups while keeping validation available", () => {
|
||||
const testCase = CUSTOM_WIDGET_AI_EVALUATION_CASES.find(({ id }) => id === "fake-service-health");
|
||||
if (!testCase) throw new Error("Fake service-health case is missing");
|
||||
const state = createAssistantEvaluationState();
|
||||
|
||||
for (let index = 0; index < 8; index += 1) {
|
||||
expect(
|
||||
executeAssistantEvaluationTool(testCase, state, "customWidget_getComponent", { name: "Text" }),
|
||||
).toMatchObject({ name: "Text" });
|
||||
}
|
||||
expect(executeAssistantEvaluationTool(testCase, state, "customWidget_getComponent", { name: "Stack" })).toEqual({
|
||||
error:
|
||||
"The targeted component-document budget is exhausted. Continue with validation using the documentation already loaded.",
|
||||
});
|
||||
expect(executeAssistantEvaluationTool(testCase, state, "customWidget_validate", { widget })).toMatchObject({
|
||||
valid: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("uses four targeted component documents after loading a complete example", () => {
|
||||
const testCase = CUSTOM_WIDGET_AI_EVALUATION_CASES.find(({ id }) => id === "fake-service-health");
|
||||
if (!testCase) throw new Error("Fake service-health case is missing");
|
||||
const state = createAssistantEvaluationState();
|
||||
executeAssistantEvaluationTool(testCase, state, "customWidget_getExample", { name: "service-dashboard" });
|
||||
|
||||
for (let index = 0; index < 4; index += 1) {
|
||||
expect(
|
||||
executeAssistantEvaluationTool(testCase, state, "customWidget_getComponent", { name: "Text" }),
|
||||
).toMatchObject({ name: "Text" });
|
||||
}
|
||||
expect(executeAssistantEvaluationTool(testCase, state, "customWidget_getComponent", { name: "Stack" })).toEqual({
|
||||
error:
|
||||
"The targeted component-document budget is exhausted. Continue with validation using the documentation already loaded.",
|
||||
});
|
||||
});
|
||||
|
||||
it("enforces explicit multi-iteration preview requests before persistence", () => {
|
||||
const baseCase = CUSTOM_WIDGET_AI_EVALUATION_CASES.find(({ id }) => id === "fake-service-health");
|
||||
if (!baseCase) throw new Error("Fake service-health case is missing");
|
||||
const testCase = { ...baseCase, minimumPreviewCycles: 2 };
|
||||
const state = createAssistantEvaluationState();
|
||||
|
||||
executeAssistantEvaluationTool(testCase, state, "customWidget_getSkill", {});
|
||||
executeAssistantEvaluationTool(testCase, state, "customWidget_schema", {});
|
||||
executeAssistantEvaluationTool(testCase, state, "customWidget_getComponentCatalog", {});
|
||||
executeAssistantEvaluationTool(testCase, state, "customWidget_validate", { widget });
|
||||
executeAssistantEvaluationTool(testCase, state, "customWidget_previewCreate", { definition: widget });
|
||||
executeAssistantEvaluationTool(testCase, state, "customWidget_previewQuery", {
|
||||
sessionId: "preview-1",
|
||||
requestId: "health",
|
||||
params: {},
|
||||
});
|
||||
expect(
|
||||
executeAssistantEvaluationTool(testCase, state, "customWidget_createFromPreview", {
|
||||
previewSessionId: "preview-1",
|
||||
}),
|
||||
).toMatchObject({ error: expect.stringContaining("at least 2 validated preview-and-query cycles") });
|
||||
expect(getAssistantEvaluationRecoveryToolName(state, 2)).toBe("customWidget_validate");
|
||||
});
|
||||
|
||||
it("routes multi-request fixtures and requires manual preview parameters", () => {
|
||||
const testCase = CUSTOM_WIDGET_AI_EVALUATION_CASES.find(({ id }) => id === "pokedex");
|
||||
if (!testCase) throw new Error("Pokédex case is missing");
|
||||
const detailRequest = {
|
||||
source: "default",
|
||||
kind: "query" as const,
|
||||
method: "GET" as const,
|
||||
path: "/api/v2/pokemon/{param:name}",
|
||||
trigger: "manual" as const,
|
||||
auth: "inherit" as const,
|
||||
permission: "view" as const,
|
||||
};
|
||||
|
||||
expect(getAssistantEvaluationPreviewResponse(testCase, detailRequest)).toMatchObject({ name: "pikachu" });
|
||||
expect(getRequiredAssistantEvaluationRequestParams(detailRequest)).toEqual(["name"]);
|
||||
});
|
||||
});
|
||||
@@ -3,9 +3,12 @@ import { describe, expect, it } from "vitest";
|
||||
import { CUSTOM_WIDGET_AI_EVALUATION_CASES } from "../../scripts/ai-evaluation-cases";
|
||||
import {
|
||||
buildEvaluationPrompt,
|
||||
buildJudgePrompt,
|
||||
buildRepairPrompt,
|
||||
DEFAULT_GENERATOR_MODEL,
|
||||
DEFAULT_JUDGE_MODEL,
|
||||
getDeterministicEvaluationIssues,
|
||||
getEvaluationResponseFixtureText,
|
||||
getJudgeResponseFormat,
|
||||
judgePasses,
|
||||
parseJudgeResult,
|
||||
@@ -46,9 +49,106 @@ const makeJudgeResult = (score: number) => ({
|
||||
});
|
||||
|
||||
describe("AI authoring evaluation", () => {
|
||||
it("defines five distinct complex scenarios", () => {
|
||||
expect(CUSTOM_WIDGET_AI_EVALUATION_CASES).toHaveLength(5);
|
||||
expect(new Set(CUSTOM_WIDGET_AI_EVALUATION_CASES.map((entry) => entry.id)).size).toBe(5);
|
||||
it("defines distinct complex and public-API scenarios", () => {
|
||||
expect(CUSTOM_WIDGET_AI_EVALUATION_CASES).toHaveLength(9);
|
||||
expect(new Set(CUSTOM_WIDGET_AI_EVALUATION_CASES.map((entry) => entry.id)).size).toBe(9);
|
||||
expect(CUSTOM_WIDGET_AI_EVALUATION_CASES.map(({ id }) => id)).toEqual(
|
||||
expect.arrayContaining([
|
||||
"pokedex",
|
||||
"fake-service-health",
|
||||
"coinmarketcap-keyless",
|
||||
"bored-activity",
|
||||
"agify-name",
|
||||
]),
|
||||
);
|
||||
});
|
||||
|
||||
it("grounds fixture-backed scenarios in verified routes and authentication", () => {
|
||||
const testCase = CUSTOM_WIDGET_AI_EVALUATION_CASES.find(({ id }) => id === "agify-name");
|
||||
if (!testCase) throw new Error("Agify evaluation case is missing");
|
||||
const validWidget = {
|
||||
$schema: "homarr-custom-widget-v2" as const,
|
||||
name: "Agify",
|
||||
sources: {
|
||||
default: {
|
||||
baseUrl: "https://api.agify.io",
|
||||
networkScope: "public" as const,
|
||||
auth: { type: "apiKeyQuery" as const, name: "apikey" },
|
||||
},
|
||||
},
|
||||
requests: {
|
||||
prediction: {
|
||||
source: "default",
|
||||
kind: "query" as const,
|
||||
method: "GET" as const,
|
||||
path: "/",
|
||||
trigger: "manual" as const,
|
||||
query: { name: { $param: "name" }, country_id: { $param: "country" } },
|
||||
auth: "inherit" as const,
|
||||
permission: "view" as const,
|
||||
},
|
||||
},
|
||||
options: {},
|
||||
template:
|
||||
'<Stack><TextInput bind="name" /><SubFetch requestId="prediction" params={{ name: inputs.name, country: "US" }}>{(result) => <Text>{result.age} from {result.count}</Text>}</SubFetch></Stack>',
|
||||
};
|
||||
expect(getDeterministicEvaluationIssues(testCase, validWidget)).toEqual([]);
|
||||
expect(
|
||||
getDeterministicEvaluationIssues(testCase, {
|
||||
...validWidget,
|
||||
sources: { default: { ...validWidget.sources.default, auth: "none" as const } },
|
||||
}),
|
||||
).toContainEqual(expect.objectContaining({ path: ["sources", "default", "auth"] }));
|
||||
});
|
||||
|
||||
it("checks the public API response contract instead of only the endpoint", () => {
|
||||
const testCase = CUSTOM_WIDGET_AI_EVALUATION_CASES.find(({ id }) => id === "coinmarketcap-keyless");
|
||||
if (!testCase) throw new Error("CoinMarketCap evaluation case is missing");
|
||||
const widget = {
|
||||
$schema: "homarr-custom-widget-v2" as const,
|
||||
name: "Crypto",
|
||||
sources: {
|
||||
default: {
|
||||
baseUrl: "https://pro-api.coinmarketcap.com",
|
||||
networkScope: "public" as const,
|
||||
auth: "none" as const,
|
||||
},
|
||||
},
|
||||
requests: {
|
||||
prices: {
|
||||
source: "default",
|
||||
kind: "query" as const,
|
||||
method: "GET" as const,
|
||||
path: "/public-api/v3/cryptocurrency/quotes/latest",
|
||||
trigger: "load" as const,
|
||||
query: { id: "1,1027,5426", convert: "USD" },
|
||||
auth: "inherit" as const,
|
||||
permission: "view" as const,
|
||||
},
|
||||
},
|
||||
options: {},
|
||||
template:
|
||||
"<Stack><RefreshButton />{data.prices?.data?.map(item => <Text key={item.symbol}>{item.quote?.[0]?.percent_change_24h} {item.quote?.[0]?.market_cap} {item.quote?.[0]?.volume_24h} {item.last_updated}</Text>)}</Stack>",
|
||||
};
|
||||
|
||||
expect(getDeterministicEvaluationIssues(testCase, widget)).toEqual([]);
|
||||
expect(
|
||||
getDeterministicEvaluationIssues(testCase, {
|
||||
...widget,
|
||||
requests: { prices: { ...widget.requests.prices, query: { id: "1,1027", convert: "USD" } } },
|
||||
}),
|
||||
).toContainEqual(expect.objectContaining({ path: ["requests"] }));
|
||||
});
|
||||
|
||||
it("includes every path-specific Pokédex fixture in generator and judge context", () => {
|
||||
const testCase = CUSTOM_WIDGET_AI_EVALUATION_CASES.find(({ id }) => id === "pokedex");
|
||||
if (!testCase) throw new Error("Pokédex evaluation case is missing");
|
||||
const fixtureText = getEvaluationResponseFixtureText(testCase);
|
||||
|
||||
expect(fixtureText).toContain("/api/v2/pokemon/{param:name}");
|
||||
expect(fixtureText).toContain('"official-artwork"');
|
||||
expect(fixtureText).toContain('"results"');
|
||||
expect(testCase.minimumPreviewCycles).toBe(3);
|
||||
});
|
||||
|
||||
it("uses the exact clipboard prompt and grounded API notes", () => {
|
||||
@@ -70,8 +170,8 @@ describe("AI authoring evaluation", () => {
|
||||
});
|
||||
|
||||
it("uses the requested DeepSeek models and strict structured judge output", () => {
|
||||
expect(DEFAULT_GENERATOR_MODEL).toBe("deepseek/deepseek-v4-pro");
|
||||
expect(DEFAULT_JUDGE_MODEL).toBe("deepseek/deepseek-v4-flash");
|
||||
expect(DEFAULT_GENERATOR_MODEL).toBe("~deepseek/deepseek-v4-flash-latest");
|
||||
expect(DEFAULT_JUDGE_MODEL).toBe("~deepseek/deepseek-v4-flash-latest");
|
||||
const format = getJudgeResponseFormat();
|
||||
expect(format.type).toBe("json_schema");
|
||||
expect(format.json_schema.strict).toBe(true);
|
||||
@@ -89,6 +189,28 @@ describe("AI authoring evaluation", () => {
|
||||
expect(judgePasses({ ...makeJudgeResult(95), fatalProblems: ["A requested core action is missing."] })).toBe(false);
|
||||
});
|
||||
|
||||
it("gives the harsh judge the authoritative request-state runtime contract", () => {
|
||||
const testCase = CUSTOM_WIDGET_AI_EVALUATION_CASES.find(({ id }) => id === "fake-service-health");
|
||||
if (!testCase) throw new Error("Fake service-health case is missing");
|
||||
const widget = {
|
||||
$schema: "homarr-custom-widget-v2" as const,
|
||||
name: "Health",
|
||||
sources: {
|
||||
default: { baseUrl: "https://status.example.test", networkScope: "public" as const, auth: "none" as const },
|
||||
},
|
||||
requests: {},
|
||||
options: {},
|
||||
template: "<Text>Health</Text>",
|
||||
};
|
||||
const prompt = buildJudgePrompt(testCase, widget);
|
||||
|
||||
expect(prompt).toContain("status.<requestId> with loading/ok/status/error fields");
|
||||
expect(prompt).toContain("RefreshButton is an installed runtime helper");
|
||||
expect(prompt).toContain(JSON.stringify(testCase.sampleResponse, null, 2));
|
||||
expect(prompt).toContain("# Bundled file: references/runtime.md");
|
||||
expect(prompt).toContain("does not prove API correctness, visual quality, usefulness, or accessibility");
|
||||
});
|
||||
|
||||
it("computes the weighted score and refuses inflated advisory verdicts", () => {
|
||||
const parsed = parseJudgeResult(
|
||||
JSON.stringify({
|
||||
@@ -105,4 +227,18 @@ describe("AI authoring evaluation", () => {
|
||||
expect(parsed.total).toBe(91);
|
||||
expect(parsed.verdict).toBe("fail");
|
||||
});
|
||||
|
||||
it("accepts a provider response that wraps structured review JSON in a code fence", () => {
|
||||
const parsed = parseJudgeResult(`\`\`\`json\n${JSON.stringify(makeJudgeResult(90))}\n\`\`\``);
|
||||
expect(parsed.total).toBe(90);
|
||||
expect(parsed.verdict).toBe("pass");
|
||||
});
|
||||
|
||||
it("normalizes a prose 'none' entry out of an otherwise empty fatal-problem list", () => {
|
||||
const parsed = parseJudgeResult(
|
||||
JSON.stringify({ ...makeJudgeResult(90), fatalProblems: ["None: all requested capabilities are present."] }),
|
||||
);
|
||||
expect(parsed.fatalProblems).toEqual([]);
|
||||
expect(parsed.verdict).toBe("pass");
|
||||
});
|
||||
});
|
||||
@@ -47,6 +47,13 @@ describe("AI prompt", () => {
|
||||
expect(CUSTOM_WIDGET_MCP_AUTHORING_PROMPT).toContain("customWidget_previewQuery once for every query");
|
||||
expect(CUSTOM_WIDGET_MCP_AUTHORING_PROMPT).toContain("customWidget_createFromPreview");
|
||||
expect(CUSTOM_WIDGET_MCP_AUTHORING_PROMPT).toContain("customWidget_getSharedProps");
|
||||
expect(CUSTOM_WIDGET_MCP_AUTHORING_PROMPT).toContain("at most eight named component documents");
|
||||
expect(CUSTOM_WIDGET_MCP_AUTHORING_PROMPT).toContain("at most four after loading a complete example");
|
||||
expect(CUSTOM_WIDGET_MCP_AUTHORING_PROMPT).toContain("Treat preview data as the binding contract");
|
||||
expect(CUSTOM_WIDGET_MCP_AUTHORING_PROMPT).toContain("freshness context");
|
||||
expect(CUSTOM_WIDGET_MCP_AUTHORING_PROMPT).toContain("Make initial states actionable");
|
||||
expect(CUSTOM_WIDGET_MCP_AUTHORING_PROMPT).toContain("compact narrow-tile rows");
|
||||
expect(CUSTOM_WIDGET_MCP_AUTHORING_PROMPT).toContain("Label standalone icons");
|
||||
expect(CUSTOM_WIDGET_MCP_AUTHORING_PROMPT).toContain("Prefer templateLines");
|
||||
expect(CUSTOM_WIDGET_MCP_AUTHORING_PROMPT.indexOf("customWidget_previewQuery")).toBeLessThan(
|
||||
CUSTOM_WIDGET_MCP_AUTHORING_PROMPT.indexOf("customWidget_create"),
|
||||
|
||||
@@ -31,6 +31,10 @@ describe("authoring resources", () => {
|
||||
expect(bundledContent).toContain(content.trim());
|
||||
}
|
||||
expect(bundledContent).toContain(skill.skillMd.trim());
|
||||
expect(bundledContent).toContain("at most eight named component documents");
|
||||
expect(bundledContent).toContain("at most four after loading a complete example");
|
||||
expect(bundledContent).toContain("successful preview response as the binding contract");
|
||||
expect(bundledContent).toContain("Do not use an unlabeled decorative icon as an empty state");
|
||||
expect(Buffer.byteLength(JSON.stringify(skill), "utf8")).toBeLessThan(12_000);
|
||||
});
|
||||
|
||||
|
||||
Reference in new issue
Block a user