Pandore/homarr
Public Access
1
0 Fork 0
Code Actions Activity

Harden Homarr provider integration

Browse Source
This commit is contained in:
ajnart committed 2026-08-11 19:47:11 +02:00
1 parent 165201632d
commit 4ebaea29fd
23 files changed
+427 -159
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

apps/docs/docs/advanced/environment-variables/index.mdx
+14 -11
View File
@@ -25,17 +25,19 @@ This is especially useful for [docker secrets](https://docs.docker.com/engine/sw
Using the `PUID` and `PGID` will require you to set the correct permissions on the mounted volumes and if used the docker socket. See more in the [Running as a different user](/docs/advanced/running-as-different-user) documentation.
:::
| Environment Variable | Description | Possible values | Default |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- | ----------------------------- |
| `PUID` | User ID to run the container as | Any valid user ID | `0` |
| `PGID` | Group ID to run the container as | Any valid group ID | `0` |
| `LOG_LEVEL` | Log level to use | `debug` / `info` / `warn` / `error` | `info` |
| `NO_EXTERNAL_CONNECTION` | Disables Homarr's internet-backed requests, including update checks, weather, and RSS feeds | `true` or `false` | `false` |
| `ENABLE_DNS_CACHING` | Enables dns caching. Enabled by default, but if you experience issues with IPv6 or static IPs, see [#4006](https://github.com/homarr-labs/homarr/issues/4006) or disable this flag | `true` or `false` | `true` |
| `HOMARR_DISABLE_IPV6` | Disables the IPv6 listener of the internal nginx proxy, even when the host supports IPv6. IPv6 is normally disabled automatically when the host has no IPv6 configured. See [#4596](https://github.com/homarr-labs/homarr/issues/4596) | `true` or `false` | `false` |
| `HOMARR_WEBSITE_URL` | Main Homarr documentation/site base used by runtime links. Must be an absolute HTTP(S) URL. | HTTP(S) URL | `https://homarr.dev` |
| `WORKSHOP_API_URL` | PocketBase API used by Homarr and standalone documentation. | HTTP(S) URL | `HOMARR_WEBSITE_URL` |
| `WORKSHOP_WEB_URL` | Public Workshop page opened from Homarr. | HTTP(S) URL | `HOMARR_WEBSITE_URL/workshop` |
| Environment Variable | Description | Possible values | Default |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- | ------------------------------ |
| `PUID` | User ID to run the container as | Any valid user ID | `0` |
| `PGID` | Group ID to run the container as | Any valid group ID | `0` |
| `LOG_LEVEL` | Log level to use | `debug` / `info` / `warn` / `error` | `info` |
| `NO_EXTERNAL_CONNECTION` | Disables Homarr's internet-backed requests, including update checks, weather, and RSS feeds | `true` or `false` | `false` |
| `ENABLE_DNS_CACHING` | Enables dns caching. Enabled by default, but if you experience issues with IPv6 or static IPs, see [#4006](https://github.com/homarr-labs/homarr/issues/4006) or disable this flag | `true` or `false` | `true` |
| `HOMARR_DISABLE_IPV6` | Disables the IPv6 listener of the internal nginx proxy, even when the host supports IPv6. IPv6 is normally disabled automatically when the host has no IPv6 configured. See [#4596](https://github.com/homarr-labs/homarr/issues/4596) | `true` or `false` | `false` |
| `HOMARR_WEBSITE_URL` | Main Homarr documentation/site base used by runtime links. Must be an absolute HTTP(S) URL. | HTTP(S) URL | `https://homarr.dev` |
| `WORKSHOP_API_URL` | PocketBase API used by Homarr and standalone documentation. | HTTP(S) URL | `HOMARR_WEBSITE_URL` |
| `WORKSHOP_WEB_URL` | Public Workshop page opened from Homarr. | HTTP(S) URL | `HOMARR_WEBSITE_URL/workshop` |
| `HOMARR_AI_DAILY_REQUEST_LIMIT` | Initial Homarr provider request allowance stored for each Workshop user. Existing per-user limits are retained. | Integer from `1` to `100000` | `50` |
| `HOMARR_AI_OPENROUTER_BASE_URL` | OpenRouter-compatible upstream used by the Workshop Homarr provider proxy. | HTTP(S) URL | `https://openrouter.ai/api/v1` |
## Authentication
@@ -114,6 +116,7 @@ The advanced deployments environment variables should only be used if you know w
| `DB_MIGRATIONS_DISABLED` | Disable db migrations. For example for helm charts. |
| `WORKSHOP_PUBLIC_ORIGIN` | Origin used in PocketBase-generated links and email. It must be set explicitly in production and must match the externally reachable Workshop origin. |
| `PB_ALLOWED_ORIGINS` | Browser origins accepted by PocketBase. Use `*` for the central public community so arbitrary self-hosted Homarr instances can use the API and OAuth; private staging may use a comma-separated origin allowlist. |
| `OPENROUTER_API_KEY` | Server-only OpenRouter key used by the Workshop Homarr provider proxy. The provider is unavailable when empty. |
All public URL variables reject non-HTTP protocols, embedded credentials, query strings, and fragments, and normalize
trailing slashes when Homarr or Docusaurus starts. The Docusaurus-only `WORKSHOP_URL` name is accepted as a deprecated alias for
apps/docs/docs/management/custom-widgets/agent-authoring.mdx
+15 -2
View File
@@ -125,6 +125,18 @@ To exercise the real Homarr Assistant lifecycle instead of accepting a returned
OPENROUTER_API_KEY=... pnpm --filter @homarr/custom-widgets test:assistant-ai
```
The same suites can target the Homarr provider with a Workshop PocketBase token instead of an OpenRouter key:
```sh
AI_PROVIDER_BASE_URL=https://homarr.dev/api/ai/v1 \
AI_PROVIDER_API_KEY=<workshop-token> \
AI_PROVIDER_MODEL=homarr/deepseek-v4-flash-latest \
pnpm --filter @homarr/custom-widgets test:assistant-ai
```
`AI_PROVIDER_JUDGE_MODEL` can select a separate judge when the configured endpoint offers one. The Homarr provider
offers exactly one model, so generation and review both use `homarr/deepseek-v4-flash-latest`.
The assistant test exposes the same lazy skill, live schema, component catalog, targeted component references,
validation, preview-query, and create-from-preview tools as the product. It refuses to save a definition that was not
validated exactly or whose preview queries were not all tested. Multi-request fixtures return a different payload for
@@ -137,8 +149,9 @@ tool traces, validated manifests, raw reviews, and detailed category scores unde
they consume a configured provider account and exercise a nondeterministic model. The repository's ordinary tests
cover the lifecycle enforcement, fixture routing, score calculation, and failure behavior without network access.
Each scenario stops as soon as it passes and has a hard limit of ten generation attempts. Override the models or
per-scenario limit with `OPENROUTER_GENERATOR_MODEL`, `OPENROUTER_JUDGE_MODEL`, and
Each scenario stops as soon as it passes and has a hard limit of ten generation attempts. Override the endpoint,
models, or per-scenario limit with `AI_PROVIDER_BASE_URL`, `AI_PROVIDER_API_KEY`, `AI_PROVIDER_MODEL`,
`AI_PROVIDER_JUDGE_MODEL`, and
`CUSTOM_WIDGET_AI_MAX_LOOPS`. Generated artifacts are local and ignored by Git; promote a reviewed, validated result
into the bundled-widget source deliberately.
apps/docs/docs/management/homarr-provider.mdx
+9 -5
View File
@@ -57,12 +57,16 @@ share the prompt, API source, and screenshots so other Homarr users can improve
The open-source proxy is part of the Workshop image. Operators configure it with server-only environment variables:
| Variable | Purpose | Default |
| --- | --- | --- |
| `OPENROUTER_API_KEY` | Upstream key used only by the Workshop proxy | Disabled when empty |
| `HOMARR_AI_DAILY_REQUEST_LIMIT` | Initial daily limit stored for each new user | `50` |
| `HOMARR_AI_OPENROUTER_BASE_URL` | OpenRouter-compatible upstream base URL | `https://openrouter.ai/api/v1` |
| Variable | Purpose | Default |
| ------------------------------- | -------------------------------------------- | ------------------------------ |
| `OPENROUTER_API_KEY` | Upstream key used only by the Workshop proxy | Disabled when empty |
| `HOMARR_AI_DAILY_REQUEST_LIMIT` | Initial daily limit stored for each new user | `50` |
| `HOMARR_AI_OPENROUTER_BASE_URL` | OpenRouter-compatible upstream base URL | `https://openrouter.ai/api/v1` |
Changing the global default affects new quota records. Existing per-user limits remain unchanged so administrators can
support future account plans. Run `pnpm test:workshop` to verify authentication, streaming, UTC reset, atomic quota
enforcement, tool-call accounting, private records, anonymous public activity, realtime delivery, and migration rollback.
The Custom Widget AI evaluation suites can use this proxy by setting `AI_PROVIDER_BASE_URL` to this service's
`/api/ai/v1` endpoint and `AI_PROVIDER_API_KEY` to a disposable Workshop user token. See
[Agent authoring](./custom-widgets/agent-authoring#evaluate-the-copied-prompt) for the complete command.
apps/docs/docs/management/workshop/index.mdx
+10 -7
View File
@@ -111,13 +111,16 @@ SMTP for comment, report, and removal notices.
The public URL contract is:
| Variable | Purpose | Default |
| ------------------------ | ------------------------------------------------- | ----------------------------- |
| `HOMARR_WEBSITE_URL` | Documentation/site base used by Homarr links | `https://homarr.dev` |
| `WORKSHOP_API_URL` | PocketBase API used by Homarr and standalone docs | `HOMARR_WEBSITE_URL` |
| `WORKSHOP_WEB_URL` | Public Workshop page linked from Homarr | `HOMARR_WEBSITE_URL/workshop` |
| `WORKSHOP_PUBLIC_ORIGIN` | PocketBase-generated links and email | Required in production |
| `PB_ALLOWED_ORIGINS` | PocketBase browser origins | `*` for the central community |
| Variable | Purpose | Default |
| ------------------------------- | ------------------------------------------------- | ------------------------------ |
| `HOMARR_WEBSITE_URL` | Documentation/site base used by Homarr links | `https://homarr.dev` |
| `WORKSHOP_API_URL` | PocketBase API used by Homarr and standalone docs | `HOMARR_WEBSITE_URL` |
| `WORKSHOP_WEB_URL` | Public Workshop page linked from Homarr | `HOMARR_WEBSITE_URL/workshop` |
| `WORKSHOP_PUBLIC_ORIGIN` | PocketBase-generated links and email | Required in production |
| `PB_ALLOWED_ORIGINS` | PocketBase browser origins | `*` for the central community |
| `OPENROUTER_API_KEY` | Server-only key for the Homarr provider proxy | Provider disabled when empty |
| `HOMARR_AI_DAILY_REQUEST_LIMIT` | Initial per-user daily provider allowance | `50` |
| `HOMARR_AI_OPENROUTER_BASE_URL` | OpenRouter-compatible proxy upstream | `https://openrouter.ai/api/v1` |
The deprecated docs-only `WORKSHOP_URL` alias remains for one release; `WORKSHOP_API_URL` takes precedence. The
combined Workshop image writes these public URLs into its browser runtime configuration whenever the container starts;
apps/docs/src/components/assistant-provider/assistant-provider-activity.tsx
+12 -5
View File
@@ -23,9 +23,11 @@ const statusLabel = {
failed: "Failed",
};
const activityDate = (value: string) => new Date(value.includes("T") ? value : value.replace(" ", "T"));
const mergeActivity = (current: WorkshopAssistantActivity[], next: WorkshopAssistantActivity) =>
[next, ...current.filter((item) => item.id !== next.id)]
.toSorted((left, right) => Date.parse(right.created) - Date.parse(left.created))
.toSorted((left, right) => activityDate(right.created).getTime() - activityDate(left.created).getTime())
.slice(0, 10);
export const AssistantProviderActivity = ({ compact = false }: { compact?: boolean }) => {
@@ -44,13 +46,18 @@ export const AssistantProviderActivity = ({ compact = false }: { compact?: boole
try {
const initial = await backend.listAssistantActivity(10);
if (active) setActivities(initial);
unsubscribe = await backend.pocketBase.collection("assistant_activity").subscribe("*", (event) => {
const nextUnsubscribe = await backend.pocketBase.collection("assistant_activity").subscribe("*", (event) => {
const parsed = workshopAssistantActivitySchema.safeParse(event.record);
if (!active || !parsed.success) return;
setConnected(true);
setActivities((current) => mergeActivity(current, parsed.data));
});
if (active) setConnected(true);
if (active) {
unsubscribe = nextUnsubscribe;
setConnected(true);
} else {
nextUnsubscribe();
}
} catch {
if (active) setConnected(false);
} finally {
@@ -93,8 +100,8 @@ export const AssistantProviderActivity = ({ compact = false }: { compact?: boole
return (
<tr key={activity.id}>
<td>
<time dateTime={activity.created} title={new Date(activity.created).toLocaleString()}>
{new Date(activity.created).toLocaleTimeString([], { hour: "2-digit", minute: "2-digit" })}
<time dateTime={activity.created} title={activityDate(activity.created).toLocaleString()}>
{activityDate(activity.created).toLocaleTimeString([], { hour: "2-digit", minute: "2-digit" })}
</time>
</td>
<td>
apps/docs/src/components/pages/home/hero/widgets/assistant-provider-widget.tsx
+2 -1
View File
@@ -1,10 +1,11 @@
import { AssistantProviderActivity } from "@site/src/components/assistant-provider/assistant-provider-activity";
import clsx from "clsx";
import type { CommonWidgetProps } from "./card";
import { WidgetCard } from "./card";
export const AssistantProviderWidget = ({ className }: CommonWidgetProps) => (
<WidgetCard width={2} className={className}>
<WidgetCard width={2} className={clsx("!items-stretch !justify-stretch overflow-hidden !p-0", className)}>
<AssistantProviderActivity compact />
</WidgetCard>
);
apps/nextjs/src/app/api/assistant/chat/assistant-provider-options.ts
+3 -3
View File
@@ -1,3 +1,5 @@
import { assistantHomarrProviderTokenHeader } from "@homarr/definitions";
/**
* The AI SDK resolves provider options by the provider name registered on the
* language model. The @ai-sdk/openai-compatible provider converts the raw
@@ -16,8 +18,7 @@ export const toProviderOptionsKey = (name: string) =>
})
.join("");
export const getHomarrProviderBaseUrl = (workshopApiUrl: string) =>
`${workshopApiUrl.replace(/\/+$/u, "")}/api/ai/v1`;
export const getHomarrProviderBaseUrl = (workshopApiUrl: string) => `${workshopApiUrl.replace(/\/+$/u, "")}/api/ai/v1`;
export const resolveHomarrProviderToken = ({
provider,
@@ -37,4 +38,3 @@ export const resolveHomarrProviderToken = ({
const token = headers.get(assistantHomarrProviderTokenHeader)?.trim();
return token && token.length <= 4096 ? token : null;
};
import { assistantHomarrProviderTokenHeader } from "@homarr/definitions";
apps/nextjs/src/components/assistant/assistant-panel.tsx
+23 -12
View File
@@ -117,7 +117,7 @@ import remarkGfm from "remark-gfm";
import { useRouter } from "next/navigation";
import { clientApi, fetchApi } from "@homarr/api/client";
import { useTimeAgo } from "@homarr/common/hooks";
import { useTimeAgo } from "@homarr/common";
import { assistantProviderIds, assistantProviderPresets, assistantReasoningModes } from "@homarr/definitions";
import type { AssistantProvider } from "@homarr/definitions";
import { showErrorNotification, showSuccessNotification } from "@homarr/notifications";
@@ -153,7 +153,7 @@ import { useAssistantReasoningState } from "./assistant-reasoning-state";
import { getNearestTriggerScrollTop } from "./assistant-trigger-scroll";
import { getToolResultPresentation } from "./assistant-tool-result";
import { getAssistantIconSearchQuery } from "./assistant-tool-label";
import { getAssistantProviderQuotaLevel } from "./assistant-provider-quota";
import { getAssistantProviderQuotaLevel, isAssistantProviderUnavailable } from "./assistant-provider-quota";
import { getAssistantToolTraceTarget } from "./assistant-tool-trace";
import { getSafeAssistantHttpUrl } from "./assistant-url";
@@ -2581,12 +2581,7 @@ const HomarrProviderQuota = () => {
return (
<Popover position="top" width={290} shadow="md" withinPortal>
<Popover.Target>
<UnstyledButton
className={classes.providerQuotaTrigger}
data-level={level}
type="button"
aria-label={label}
>
<UnstyledButton className={classes.providerQuotaTrigger} data-level={level} type="button" aria-label={label}>
{preferences.quotaLoading && !preferences.quota ? <Loader size={14} /> : <Icon size={16} aria-hidden />}
<Text component="span" size="xs" fw={650}>
{preferences.quota?.remaining ?? "–"}
@@ -2644,7 +2639,10 @@ const HomarrProviderQuota = () => {
</Box>
<Text size="xs" c="dimmed">
{t("providerQuota.reset", { relative: resetRelative })}{" "}
<time dateTime={preferences.quota.resetsAt} title={new Date(preferences.quota.resetsAt).toLocaleString()}>
<time
dateTime={preferences.quota.resetsAt}
title={new Date(preferences.quota.resetsAt).toLocaleString()}
>
{new Date(preferences.quota.resetsAt).toLocaleTimeString([], {
hour: "2-digit",
minute: "2-digit",
@@ -2679,8 +2677,15 @@ const HomarrProviderQuota = () => {
const Composer = (props: ComposerProps) => {
const t = useScopedI18n("common.assistant");
const preferences = useAssistantPreferences();
const running = useAuiState((state) => state.thread.isRunning);
const hasPendingAction = props.pendingAction !== undefined;
const providerUnavailable = isAssistantProviderUnavailable({
provider: preferences.provider,
signedIn: preferences.providerUser !== null,
remaining: preferences.quota?.remaining,
});
const sendBlocked = hasPendingAction || providerUnavailable;
const composerInputRef = useRef<HTMLDivElement>(null);
const composerLabel = t("composerPlaceholder");
@@ -2701,7 +2706,7 @@ const Composer = (props: ComposerProps) => {
className={classes.composer}
data-pending-action={hasPendingAction || undefined}
onSubmit={(event) => {
if (hasPendingAction) event.preventDefault();
if (sendBlocked) event.preventDefault();
}}
>
<ComposerTriggers />
@@ -2747,8 +2752,14 @@ const Composer = (props: ComposerProps) => {
color="red"
variant="filled"
size="lg"
aria-label={hasPendingAction ? t("pendingAction.sendBlocked") : t("send")}
disabled={hasPendingAction}
aria-label={
hasPendingAction
? t("pendingAction.sendBlocked")
: providerUnavailable
? t("providerQuota.unavailableDescription")
: t("send")
}
disabled={sendBlocked}
>
<IconArrowUp size={18} />
</ActionIcon>
apps/nextjs/src/components/assistant/assistant-provider-quota.spec.ts
+8 -1
View File
@@ -1,6 +1,6 @@
import { describe, expect, test } from "vitest";
import { getAssistantProviderQuotaLevel } from "./assistant-provider-quota";
import { getAssistantProviderQuotaLevel, isAssistantProviderUnavailable } from "./assistant-provider-quota";
describe("getAssistantProviderQuotaLevel", () => {
test.each([
@@ -12,4 +12,11 @@ describe("getAssistantProviderQuotaLevel", () => {
] as const)("maps %o to %s", (usage, expected) => {
expect(getAssistantProviderQuotaLevel(usage)).toBe(expected);
});
test("blocks only unavailable Homarr provider sessions", () => {
expect(isAssistantProviderUnavailable({ provider: "homarr", signedIn: false, remaining: undefined })).toBe(true);
expect(isAssistantProviderUnavailable({ provider: "homarr", signedIn: true, remaining: 0 })).toBe(true);
expect(isAssistantProviderUnavailable({ provider: "homarr", signedIn: true, remaining: undefined })).toBe(false);
expect(isAssistantProviderUnavailable({ provider: "openrouter", signedIn: false, remaining: 0 })).toBe(false);
});
});
apps/nextjs/src/components/assistant/assistant-provider-quota.ts
+11
View File
@@ -1,4 +1,5 @@
import type { WorkshopAssistantUsage } from "@homarr/workshop/schema";
import type { AssistantProvider } from "@homarr/definitions";
export type AssistantProviderQuotaLevel = "ok" | "warning" | "bad" | "dead";
@@ -11,3 +12,13 @@ export const getAssistantProviderQuotaLevel = (
if (remainingRatio <= 0.5) return "warning";
return "ok";
};
export const isAssistantProviderUnavailable = ({
provider,
signedIn,
remaining,
}: {
provider: AssistantProvider | null;
signedIn: boolean;
remaining: number | undefined;
}) => provider === "homarr" && (!signedIn || remaining === 0);
apps/nextjs/src/components/assistant/assistant-provider.tsx
+41 -20
View File
@@ -29,6 +29,7 @@ import { createWorkshopClient } from "~/components/workshop/workshop-client";
import { AssistantContext, AssistantPreferencesContext, useAssistantPreferences } from "./assistant-context";
import { shouldAutomaticallyContinueAssistant } from "./assistant-auto-submit";
import { isAssistantProviderUnavailable } from "./assistant-provider-quota";
import { AssistantAutoApprovalProvider } from "./assistant-auto-approval";
import { getRunningAssistantPartType } from "./assistant-activity-state";
import { prepareAssistantRequestBody } from "./assistant-attachment-payload";
@@ -169,7 +170,7 @@ const AssistantPreferencesProvider = ({ children }: PropsWithChildren) => {
reasoning: current.reasoning,
};
}, []);
const getRequestHeaders = useCallback(() => {
const getRequestHeaders = useCallback((): Record<string, string> => {
if (data?.provider !== "homarr") return {};
const token = workshopClient.authToken;
return token ? { [assistantHomarrProviderTokenHeader]: token } : {};
@@ -412,18 +413,19 @@ const AssistantThreadRuntime = () => {
const localThreadId = useAuiState((state) => state.threadListItem.id);
const threadId = useAuiState((state) => state.threadListItem.remoteId);
const preferences = useAssistantPreferences();
const { getRequestBody, getRequestHeaders } = preferences;
const transport = useMemo(
() =>
new AssistantChatTransport({
api: "/api/assistant/chat",
body: preferences.getRequestBody,
body: getRequestBody,
fetch: (input, init) => {
const headers = new Headers(init?.headers);
for (const [name, value] of Object.entries(preferences.getRequestHeaders())) headers.set(name, value);
for (const [name, value] of Object.entries(getRequestHeaders())) headers.set(name, value);
return globalThis.fetch(input, { ...init, headers, body: prepareAssistantRequestBody(init?.body) });
},
}),
[preferences.getRequestBody, preferences.getRequestHeaders],
[getRequestBody, getRequestHeaders],
);
const history = useMemo(() => createHistoryAdapter(threadId), [threadId]);
const selectedModel = preferences.models.find((model) => model.id === preferences.modelId);
@@ -447,17 +449,14 @@ const AssistantThreadRuntime = () => {
}),
[t, threadId],
);
const onError = useCallback(
() => {
void preferences.refreshQuota();
showErrorNotification({
title: t("responseError.title"),
message: t("responseError.description"),
autoClose: 10_000,
});
},
[preferences, t],
);
const onError = useCallback(() => {
void preferences.refreshQuota();
showErrorNotification({
title: t("responseError.title"),
message: t("responseError.description"),
autoClose: 10_000,
});
}, [preferences, t]);
const chat = useChat<AssistantUIMessage>({
id: localThreadId,
@@ -650,6 +649,11 @@ const EnabledAssistantProvider = ({ children }: PropsWithChildren) => {
);
const pendingAction = getPendingAssistantAction(latestAssistantMessage);
const assistantIsRunning = isRunning || queuedPrompt !== null;
const providerUnavailable = isAssistantProviderUnavailable({
provider: preferences.provider,
signedIn: preferences.providerUser !== null,
remaining: preferences.quota?.remaining,
});
const notificationKey = getNotificationKey(latestAssistantMessage);
const notificationStateRef = useRef(initialAssistantNotificationState);
const mutationRefreshStateRef = useRef<{ conversationId: string | null; toolCallIds: Set<string> }>({
@@ -680,6 +684,15 @@ const EnabledAssistantProvider = ({ children }: PropsWithChildren) => {
(prompt: string) => {
const text = prompt.trim();
if (text.length === 0) return false;
if (providerUnavailable) {
showWarningNotification({
title: t("providerQuota.title"),
message: preferences.providerUser
? t("providerQuota.exhaustedDescription")
: t("providerQuota.signInDescription"),
});
return false;
}
if (assistantIsRunning || latestStatus?.type === "requires-action") {
showWarningNotification({
title: t("busy.title"),
@@ -694,7 +707,7 @@ const EnabledAssistantProvider = ({ children }: PropsWithChildren) => {
}
return sendPromptThroughRuntime(aui, text);
},
[assistantIsRunning, aui, isLoading, latestStatus?.type, t],
[assistantIsRunning, aui, isLoading, latestStatus?.type, preferences.providerUser, providerUnavailable, t],
);
const selectModel = useCallback(
(modelId: string) => {
@@ -712,10 +725,14 @@ const EnabledAssistantProvider = ({ children }: PropsWithChildren) => {
useEffect(() => {
if (isLoading || isRunning || queuedPrompt === null) return;
if (providerUnavailable) {
setQueuedPrompt(null);
return;
}
const prompt = queuedPrompt;
setQueuedPrompt(null);
sendPromptThroughRuntime(aui, prompt);
}, [aui, isLoading, isRunning, queuedPrompt]);
}, [aui, isLoading, isRunning, providerUnavailable, queuedPrompt]);
useEffect(() => {
if (isLoading) return;
@@ -757,12 +774,16 @@ const EnabledAssistantProvider = ({ children }: PropsWithChildren) => {
useHotkeys([[hotkeys.openAssistant, open, { preventDefault: true }]]);
const spotlightItem = useMemo(() => {
const canSend = !assistantIsRunning && latestStatus?.type !== "requires-action";
const canSend = !providerUnavailable && !assistantIsRunning && latestStatus?.type !== "requires-action";
return {
id: "homarr-assistant",
name: t("spotlight"),
icon: "/logo/logo.png",
description: canSend ? t("spotlightDescription") : t("busy.description"),
description: providerUnavailable
? t("providerQuota.unavailableDescription")
: canSend
? t("spotlightDescription")
: t("busy.description"),
unavailable: !canSend,
alwaysVisible: true,
interaction: (query: string) =>
@@ -773,7 +794,7 @@ const EnabledAssistantProvider = ({ children }: PropsWithChildren) => {
canSend,
}),
};
}, [assistantIsRunning, latestStatus?.type, sendPrompt, t]);
}, [assistantIsRunning, latestStatus?.type, providerUnavailable, sendPrompt, t]);
useRegisterSpotlightContextResults("homarr-assistant", [spotlightItem], [spotlightItem]);
const value = useMemo(
apps/workshop/homarr_provider.go
+93 -70
View File
@@ -20,13 +20,14 @@ import (
)
const (
homarrProviderModelID = "homarr/deepseek-v4-flash-latest"
openRouterModelID = "~deepseek/deepseek-v4-flash-latest"
defaultDailyLimit = 50
maxChatRequestBytes = 8 << 20
maxUpstreamErrorBytes = 1 << 20
openRouterDefaultURL = "https://openrouter.ai/api/v1"
providerRequestTimeout = 5 * time.Minute
homarrProviderModelID = "homarr/deepseek-v4-flash-latest"
openRouterModelID = "~deepseek/deepseek-v4-flash-latest"
defaultDailyLimit = 50
maxProviderRequestUnits = 1000
maxChatRequestBytes = 8 << 20
maxUpstreamErrorBytes = 1 << 20
openRouterDefaultURL = "https://openrouter.ai/api/v1"
providerRequestTimeout = 5 * time.Minute
)
type homarrProvider struct {
@@ -139,7 +140,7 @@ func (provider *homarrProvider) models(event *core.RequestEvent) error {
}
func (provider *homarrProvider) usage(event *core.RequestEvent) error {
snapshot, err := provider.quota(event.App, event.Auth, 0)
snapshot, err := provider.quota(event.App, event.Auth)
if err != nil {
return event.InternalServerError("The Homarr provider allowance could not be loaded.", nil)
}
@@ -165,16 +166,8 @@ func (provider *homarrProvider) chat(event *core.RequestEvent) error {
}
requestUnits := countRequestUnits(requestBody.Messages)
snapshot, err := provider.quota(event.App, event.Auth, requestUnits)
if err != nil {
var quotaErr *quotaExceededError
if errors.As(err, &quotaErr) {
event.Response.Header().Set("X-Homarr-Quota-Limit", strconv.Itoa(quotaErr.Snapshot.Limit))
event.Response.Header().Set("X-Homarr-Quota-Remaining", "0")
event.Response.Header().Set("X-Homarr-Quota-Reset", quotaErr.Snapshot.ResetsAt.Format(time.RFC3339))
return event.JSON(http.StatusTooManyRequests, openAIError("Your daily Homarr provider allowance is exhausted."))
}
return event.JSON(http.StatusInternalServerError, openAIError("The Homarr provider allowance could not be reserved."))
if requestUnits > maxProviderRequestUnits {
return event.JSON(http.StatusBadRequest, openAIError("A single Homarr provider request can use at most 1000 request units."))
}
var payload map[string]any
@@ -188,8 +181,15 @@ func (provider *homarrProvider) chat(event *core.RequestEvent) error {
return event.JSON(http.StatusBadRequest, openAIError("The request body could not be prepared."))
}
privateRequest, publicActivity, err := provider.startActivity(event.App, event.Auth, requestUnits)
snapshot, privateRequest, publicActivity, err := provider.startRequest(event.App, event.Auth, requestUnits)
if err != nil {
var quotaErr *quotaExceededError
if errors.As(err, &quotaErr) {
event.Response.Header().Set("X-Homarr-Quota-Limit", strconv.Itoa(quotaErr.Snapshot.Limit))
event.Response.Header().Set("X-Homarr-Quota-Remaining", "0")
event.Response.Header().Set("X-Homarr-Quota-Reset", quotaErr.Snapshot.ResetsAt.Format(time.RFC3339))
return event.JSON(http.StatusTooManyRequests, openAIError("Your daily Homarr provider allowance is exhausted."))
}
return event.JSON(http.StatusInternalServerError, openAIError("The Homarr provider request could not be started."))
}
startedAt := provider.now().UTC()
@@ -262,76 +262,99 @@ func countRequestUnits(messages []providerMessage) int {
return units
}
func (provider *homarrProvider) quota(app core.App, user *core.Record, reserve int) (quotaSnapshot, error) {
func (provider *homarrProvider) quota(app core.App, user *core.Record) (quotaSnapshot, error) {
now := provider.now().UTC()
today := now.Format(time.DateOnly)
resetsAt := time.Date(now.Year(), now.Month(), now.Day()+1, 0, 0, 0, 0, time.UTC)
var snapshot quotaSnapshot
err := app.RunInTransaction(func(txApp core.App) error {
quota, err := txApp.FindFirstRecordByFilter("assistant_quotas", "user = {:user}", dbx.Params{"user": user.Id})
if err != nil && !errors.Is(err, sql.ErrNoRows) {
quota, current, err := provider.loadQuota(txApp, user, now)
if err != nil {
return err
}
if quota == nil {
collection, err := txApp.FindCollectionByNameOrId("assistant_quotas")
if err != nil {
return err
}
quota = core.NewRecord(collection)
quota.Set("user", user.Id)
quota.Set("dailyLimit", provider.dailyLimit)
}
if quota.GetString("day") != today {
quota.Set("day", today)
quota.Set("used", 0)
quota.Set("inputTokens", 0)
quota.Set("outputTokens", 0)
quota.Set("totalTokens", 0)
}
limit := quota.GetInt("dailyLimit")
used := quota.GetInt("used")
snapshot = quotaSnapshot{Limit: limit, Used: used, Remaining: max(0, limit-used), ResetsAt: resetsAt}
if reserve > snapshot.Remaining {
return &quotaExceededError{Snapshot: snapshot}
}
if reserve > 0 {
used += reserve
quota.Set("used", used)
snapshot.Used = used
snapshot.Remaining = max(0, limit-used)
}
snapshot = current
return txApp.Save(quota)
})
return snapshot, err
}
func (provider *homarrProvider) startActivity(app core.App, user *core.Record, requestUnits int) (*core.Record, *core.Record, error) {
privateCollection, err := app.FindCollectionByNameOrId("assistant_requests")
if err != nil {
return nil, nil, err
func (provider *homarrProvider) loadQuota(app core.App, user *core.Record, now time.Time) (*core.Record, quotaSnapshot, error) {
quota, err := app.FindFirstRecordByFilter("assistant_quotas", "user = {:user}", dbx.Params{"user": user.Id})
if err != nil && !errors.Is(err, sql.ErrNoRows) {
return nil, quotaSnapshot{}, err
}
publicCollection, err := app.FindCollectionByNameOrId("assistant_activity")
if err != nil {
return nil, nil, err
if quota == nil {
collection, err := app.FindCollectionByNameOrId("assistant_quotas")
if err != nil {
return nil, quotaSnapshot{}, err
}
quota = core.NewRecord(collection)
quota.Set("user", user.Id)
quota.Set("dailyLimit", provider.dailyLimit)
}
privateRequest := core.NewRecord(privateCollection)
publicActivity := core.NewRecord(publicCollection)
for _, record := range []*core.Record{privateRequest, publicActivity} {
record.Set("status", "processing")
record.Set("model", homarrProviderModelID)
record.Set("requestUnits", requestUnits)
if quota.GetString("day") != now.Format(time.DateOnly) {
quota.Set("day", now.Format(time.DateOnly))
quota.Set("used", 0)
quota.Set("inputTokens", 0)
quota.Set("outputTokens", 0)
quota.Set("totalTokens", 0)
}
privateRequest.Set("user", user.Id)
privateRequest.Set("publicActivity", publicActivity.Id)
limit := quota.GetInt("dailyLimit")
used := quota.GetInt("used")
return quota, quotaSnapshot{
Limit: limit,
Used: used,
Remaining: max(0, limit-used),
ResetsAt: time.Date(now.Year(), now.Month(), now.Day()+1, 0, 0, 0, 0, time.UTC),
}, nil
}
func (provider *homarrProvider) startRequest(
app core.App,
user *core.Record,
requestUnits int,
) (quotaSnapshot, *core.Record, *core.Record, error) {
var snapshot quotaSnapshot
var privateRequest *core.Record
var publicActivity *core.Record
err := app.RunInTransaction(func(txApp core.App) error {
quota, current, err := provider.loadQuota(txApp, user, provider.now().UTC())
if err != nil {
return err
}
if requestUnits > current.Remaining {
return &quotaExceededError{Snapshot: current}
}
privateCollection, err := txApp.FindCollectionByNameOrId("assistant_requests")
if err != nil {
return err
}
publicCollection, err := txApp.FindCollectionByNameOrId("assistant_activity")
if err != nil {
return err
}
privateRequest = core.NewRecord(privateCollection)
publicActivity = core.NewRecord(publicCollection)
for _, record := range []*core.Record{privateRequest, publicActivity} {
record.Set("status", "processing")
record.Set("model", homarrProviderModelID)
record.Set("requestUnits", requestUnits)
}
privateRequest.Set("user", user.Id)
err = app.RunInTransaction(func(txApp core.App) error {
if err := txApp.Save(publicActivity); err != nil {
return err
}
privateRequest.Set("publicActivity", publicActivity.Id)
return txApp.Save(privateRequest)
if err := txApp.Save(privateRequest); err != nil {
return err
}
current.Used += requestUnits
current.Remaining = max(0, current.Limit-current.Used)
quota.Set("used", current.Used)
snapshot = current
return txApp.Save(quota)
})
return privateRequest, publicActivity, err
return snapshot, privateRequest, publicActivity, err
}
func (provider *homarrProvider) finishActivity(
apps/workshop/tests/workshop.integration.mjs
+41 -6
View File
@@ -154,6 +154,30 @@ if (
throw new Error(`Unexpected initial Homarr provider allowance: ${JSON.stringify(initialUsage)}`);
}
await expectStatus(
"/api/ai/v1/chat/completions",
{
method: "POST",
headers: authorSession.headers,
body: JSON.stringify({
model: "homarr/deepseek-v4-flash-latest",
messages: [
{ role: "user", content: "too many tools" },
...Array.from({ length: 1000 }, (_, index) => ({
role: "tool",
tool_call_id: `tool-${index}`,
content: "{}",
})),
],
}),
},
400,
);
const usageAfterRejectedRequest = await request("/api/ai/usage", { headers: authorSession.headers });
if (usageAfterRejectedRequest.used !== 0) {
throw new Error("A rejected oversized request-unit batch consumed allowance");
}
await expectStatus(
"/api/ai/v1/chat/completions",
{
@@ -257,6 +281,18 @@ if (
) {
throw new Error("Public assistant activity must be anonymous and include completed token totals");
}
await expectStatus(
"/api/collections/assistant_activity/records",
{
method: "POST",
body: JSON.stringify({
status: "completed",
model: "forged",
requestUnits: 1,
}),
},
403,
);
await expectStatus("/api/collections/assistant_requests/records", { headers: authorSession.headers }, 403);
await expectStatus("/api/collections/assistant_quotas/records", { headers: authorSession.headers }, 403);
@@ -296,13 +332,12 @@ const allowanceProbe = () =>
stream: false,
}),
});
if (!(await allowanceProbe()).ok || !(await allowanceProbe()).ok) {
throw new Error("The per-user allowance rejected a permitted request");
}
const exhaustedResponse = await allowanceProbe();
const concurrentAllowanceResponses = await Promise.all([allowanceProbe(), allowanceProbe(), allowanceProbe()]);
const concurrentStatuses = concurrentAllowanceResponses.map((response) => response.status).toSorted();
const exhaustedResponse = concurrentAllowanceResponses.find((response) => response.status === 429);
if (
exhaustedResponse.status !== 429 ||
exhaustedResponse.headers.get("x-homarr-quota-remaining") !== "0" ||
concurrentStatuses.join(",") !== "200,200,429" ||
exhaustedResponse?.headers.get("x-homarr-quota-remaining") !== "0" ||
!exhaustedResponse.headers.get("x-homarr-quota-reset")
) {
throw new Error("The per-user allowance was not enforced atomically");
packages/api/src/router/assistant.ts
+1 -1
View File
@@ -480,7 +480,7 @@ export const assistantRouter = createTRPCRouter({
const configuration = await getConfigurationAsync(ctx.db);
if (isDemoMode) {
return {
provider: "homarr",
provider: "custom" as const,
defaultModelId: "homarr-assistant",
models: [
{
packages/api/src/router/test/assistant.spec.ts
+67
View File
@@ -121,6 +121,30 @@ describe("assistantRouter.updateConnection", () => {
modelId: "example/model",
});
});
test("never stores administrator credentials for the Homarr provider", async () => {
const db = await createConfiguredAssistantAsync();
const caller = assistantRouter.createCaller({ db, deviceType: undefined, session: adminSession });
await caller.updateConnection({
provider: "homarr",
baseUrl: "https://homarr.dev/api/ai/v1",
modelDiscoveryPath: "/models",
apiKey: "must-not-be-stored",
customHeaders: { "X-Secret": "must-not-be-stored" },
clearApiKey: false,
clearCustomHeaders: false,
});
const [configuration] = await db.select().from(assistantConfigurations);
expect(configuration).toMatchObject({
provider: "homarr",
encryptedApiKey: null,
encryptedHeaders: null,
enabled: false,
modelId: null,
});
});
});
describe("assistantRouter.updateConfiguration", () => {
@@ -170,6 +194,49 @@ describe("assistantRouter.updateConfiguration", () => {
expect(configuration?.webSearchEnabled).toBe(true);
});
test("allows the Homarr OpenRouter proxy to enable web search", async () => {
const db = await createConfiguredAssistantAsync();
await db
.update(assistantConfigurations)
.set({
provider: "homarr",
baseUrl: "https://homarr.dev/api/ai/v1",
encryptedApiKey: null,
encryptedHeaders: null,
modelId: "homarr/deepseek-v4-flash-latest",
})
.where(eq(assistantConfigurations.id, "default"));
const caller = assistantRouter.createCaller({ db, deviceType: undefined, session: adminSession });
vi.stubGlobal(
"fetch",
vi.fn().mockResolvedValue(
Response.json({
data: [
{
id: "homarr/deepseek-v4-flash-latest",
name: "DeepSeek V4 Flash Latest",
supported_parameters: ["tools"],
architecture: { output_modalities: ["text"] },
},
],
}),
),
);
try {
await caller.updateConfiguration({
enabled: true,
modelId: "homarr/deepseek-v4-flash-latest",
webSearchEnabled: true,
});
} finally {
vi.unstubAllGlobals();
}
const [configuration] = await db.select().from(assistantConfigurations);
expect(configuration?.webSearchEnabled).toBe(true);
});
test("rejects OpenRouter server tools for an incompatible provider", async () => {
const db = await createConfiguredAssistantAsync();
await db
packages/custom-widgets/scripts/ai-assistant-evaluation.ts
+11 -3
View File
@@ -21,6 +21,7 @@ import { getCustomWidgetJsonSchema } from "../src/core/schema";
import type { CustomWidgetAiEvaluationCase } from "./ai-evaluation-cases";
import {
DEFAULT_GENERATOR_MODEL,
getAiProviderChatCompletionsUrl,
getDeterministicEvaluationIssues,
judgePasses,
parseJudgeResult,
@@ -441,12 +442,13 @@ function buildAssistantPrompt(testCase: CustomWidgetAiEvaluationCase, feedback:
async function callAssistantStep(args: {
apiKey: string;
baseUrl?: string;
model: string;
messages: OpenRouterMessage[];
tools: ToolDefinition[];
toolChoice?: string;
}): Promise<OpenRouterAssistantMessage> {
const response = await fetch("https://openrouter.ai/api/v1/chat/completions", {
const response = await fetch(getAiProviderChatCompletionsUrl(args.baseUrl), {
method: "POST",
headers: {
Authorization: `Bearer ${args.apiKey}`,
@@ -467,10 +469,10 @@ async function callAssistantStep(args: {
});
const payload = (await response.json()) as OpenRouterResponse;
if (!response.ok) {
throw new Error(`OpenRouter request failed (${response.status}): ${payload.error?.message ?? "Unknown error"}`);
throw new Error(`AI provider request failed (${response.status}): ${payload.error?.message ?? "Unknown error"}`);
}
const message = payload.choices?.[0]?.message;
if (!message) throw new Error("OpenRouter returned no assistant message");
if (!message) throw new Error("AI provider returned no assistant message");
return message;
}
@@ -498,6 +500,7 @@ export const getAssistantEvaluationRecoveryToolName = (state: AssistantAttemptSt
async function callAssistantStepWithRecovery(args: {
apiKey: string;
baseUrl?: string;
model: string;
messages: OpenRouterMessage[];
tools: ToolDefinition[];
@@ -530,6 +533,7 @@ async function callAssistantStepWithRecovery(args: {
async function runAssistantAttempt(args: {
testCase: CustomWidgetAiEvaluationCase;
apiKey: string;
baseUrl?: string;
model: string;
feedback: readonly string[];
}) {
@@ -553,6 +557,7 @@ async function runAssistantAttempt(args: {
: customWidgetAssistantEvaluationToolDefinitions;
const message = await callAssistantStepWithRecovery({
apiKey: args.apiKey,
baseUrl: args.baseUrl,
model: args.model,
messages,
tools,
@@ -592,6 +597,7 @@ async function runAssistantAttempt(args: {
export async function evaluateCustomWidgetAssistantCase(args: {
testCase: CustomWidgetAiEvaluationCase;
apiKey: string;
baseUrl?: string;
outputRoot: string;
maxLoops: number;
generatorModel?: string;
@@ -612,6 +618,7 @@ export async function evaluateCustomWidgetAssistantCase(args: {
run = await runAssistantAttempt({
testCase: args.testCase,
apiKey: args.apiKey,
baseUrl: args.baseUrl,
model: args.generatorModel ?? DEFAULT_GENERATOR_MODEL,
feedback,
});
@@ -651,6 +658,7 @@ export async function evaluateCustomWidgetAssistantCase(args: {
testCase: args.testCase,
widget: run.state.createdWidget,
apiKey: args.apiKey,
baseUrl: args.baseUrl,
judgeModel: args.judgeModel,
});
await writeFile(path.join(caseDirectory, `judge-${attempt}.json`), judgeRaw, "utf8");
packages/custom-widgets/scripts/ai-evaluation.ts
+27 -3
View File
@@ -14,7 +14,24 @@ import type { CustomWidgetAiEvaluationCase } from "./ai-evaluation-cases";
// deepseek/deepseek-v4-flash-latest alias is rejected by the chat-completions API.
export const DEFAULT_GENERATOR_MODEL = "~deepseek/deepseek-v4-flash-latest";
export const DEFAULT_JUDGE_MODEL = "~deepseek/deepseek-v4-flash-latest";
export const DEFAULT_AI_PROVIDER_BASE_URL = "https://openrouter.ai/api/v1";
export const MAX_AI_EVALUATION_LOOPS = 10;
export const getAiProviderChatCompletionsUrl = (baseUrl = DEFAULT_AI_PROVIDER_BASE_URL) =>
`${baseUrl.replace(/\/+$/u, "")}/chat/completions`;
export const resolveAiEvaluationProviderConfig = (environment: Record<string, string | undefined>) => {
const baseUrl = (environment.AI_PROVIDER_BASE_URL ?? DEFAULT_AI_PROVIDER_BASE_URL).replace(/\/+$/u, "");
const homarrProvider = baseUrl.endsWith("/api/ai/v1");
const providerDefaultModel = homarrProvider ? "homarr/deepseek-v4-flash-latest" : DEFAULT_GENERATOR_MODEL;
return {
apiKey: environment.AI_PROVIDER_API_KEY ?? environment.OPENROUTER_API_KEY,
baseUrl,
generatorModel: environment.AI_PROVIDER_MODEL ?? environment.OPENROUTER_GENERATOR_MODEL ?? providerDefaultModel,
judgeModel:
environment.AI_PROVIDER_JUDGE_MODEL ??
environment.OPENROUTER_JUDGE_MODEL ??
(homarrProvider ? providerDefaultModel : DEFAULT_JUDGE_MODEL),
};
};
const scoreSchema = z.number().int().min(0).max(100);
const judgeCategoriesSchema = z.strictObject({
@@ -263,10 +280,12 @@ export async function requestCustomWidgetJudge(args: {
testCase: CustomWidgetAiEvaluationCase;
widget: HomarrCustomWidgetV2;
apiKey: string;
baseUrl?: string;
judgeModel?: string;
}) {
return callOpenRouter({
apiKey: args.apiKey,
baseUrl: args.baseUrl,
model: args.judgeModel ?? DEFAULT_JUDGE_MODEL,
prompt: buildJudgePrompt(args.testCase, args.widget),
purpose: "judge",
@@ -277,6 +296,7 @@ export async function judgeCustomWidgetCase(args: {
testCase: CustomWidgetAiEvaluationCase;
widget: HomarrCustomWidgetV2;
apiKey: string;
baseUrl?: string;
judgeModel?: string;
}) {
const raw = await requestCustomWidgetJudge(args);
@@ -286,6 +306,7 @@ export async function judgeCustomWidgetCase(args: {
export async function evaluateCustomWidgetCase(args: {
testCase: CustomWidgetAiEvaluationCase;
apiKey: string;
baseUrl?: string;
outputRoot: string;
maxLoops: number;
generatorModel?: string;
@@ -305,6 +326,7 @@ export async function evaluateCustomWidgetCase(args: {
try {
response = await callOpenRouter({
apiKey: args.apiKey,
baseUrl: args.baseUrl,
model: args.generatorModel ?? DEFAULT_GENERATOR_MODEL,
prompt,
purpose: "generation",
@@ -340,6 +362,7 @@ export async function evaluateCustomWidgetCase(args: {
testCase: args.testCase,
widget: canonical,
apiKey: args.apiKey,
baseUrl: args.baseUrl,
judgeModel: args.judgeModel,
});
await writeFile(path.join(caseDirectory, `judge-${attempt}.json`), judgeRaw, "utf8");
@@ -413,12 +436,13 @@ async function writeWidgetFiles(directory: string, widget: HomarrCustomWidgetV2,
async function callOpenRouter(args: {
apiKey: string;
baseUrl?: string;
model: string;
prompt: string;
purpose: "generation" | "judge";
}): Promise<string> {
const isJudge = args.purpose === "judge";
const response = await fetch("https://openrouter.ai/api/v1/chat/completions", {
const response = await fetch(getAiProviderChatCompletionsUrl(args.baseUrl), {
method: "POST",
headers: {
Authorization: `Bearer ${args.apiKey}`,
@@ -438,8 +462,8 @@ async function callOpenRouter(args: {
});
const payload = (await response.json()) as OpenRouterResponse;
if (!response.ok)
throw new Error(`OpenRouter request failed (${response.status}): ${payload.error?.message ?? "Unknown error"}`);
throw new Error(`AI provider request failed (${response.status}): ${payload.error?.message ?? "Unknown error"}`);
const content = payload.choices?.[0]?.message?.content;
if (!content) throw new Error("OpenRouter returned no message content");
if (!content) throw new Error("AI provider returned no message content");
return content;
}
packages/custom-widgets/scripts/evaluate-ai-authoring.ts
+14 -9
View File
@@ -4,11 +4,13 @@ import path from "node:path";
import { CUSTOM_WIDGET_AI_EVALUATION_CASES } from "./ai-evaluation-cases";
import { evaluateCustomWidgetAssistantCase } from "./ai-assistant-evaluation";
import type { CustomWidgetAssistantEvaluationResult } from "./ai-assistant-evaluation";
import { DEFAULT_GENERATOR_MODEL, DEFAULT_JUDGE_MODEL, evaluateCustomWidgetCase } from "./ai-evaluation";
import { evaluateCustomWidgetCase, resolveAiEvaluationProviderConfig } from "./ai-evaluation";
import type { AiEvaluationResult } from "./ai-evaluation";
const apiKey = process.env.OPENROUTER_API_KEY;
if (!apiKey) throw new Error("OPENROUTER_API_KEY is required for the live Custom Widget AI evaluation");
const { apiKey, baseUrl: providerBaseUrl, generatorModel, judgeModel } = resolveAiEvaluationProviderConfig(process.env);
if (!apiKey) {
throw new Error("AI_PROVIDER_API_KEY or OPENROUTER_API_KEY is required for the live Custom Widget AI evaluation");
}
const requestedCase = process.argv.find((value) => value.startsWith("--case="))?.slice("--case=".length);
const assistantMode = process.argv.includes("--assistant");
@@ -43,18 +45,20 @@ for (const testCase of selectedCases) {
? await evaluateCustomWidgetAssistantCase({
testCase,
apiKey,
baseUrl: providerBaseUrl,
outputRoot,
maxLoops,
generatorModel: process.env.OPENROUTER_GENERATOR_MODEL ?? DEFAULT_GENERATOR_MODEL,
judgeModel: process.env.OPENROUTER_JUDGE_MODEL ?? DEFAULT_JUDGE_MODEL,
generatorModel,
judgeModel,
})
: await evaluateCustomWidgetCase({
testCase,
apiKey,
baseUrl: providerBaseUrl,
outputRoot,
maxLoops,
generatorModel: process.env.OPENROUTER_GENERATOR_MODEL ?? DEFAULT_GENERATOR_MODEL,
judgeModel: process.env.OPENROUTER_JUDGE_MODEL ?? DEFAULT_JUDGE_MODEL,
generatorModel,
judgeModel,
});
results.push(result);
remainingLoopBudget -= result.attempts;
@@ -68,8 +72,9 @@ for (const testCase of selectedCases) {
const summary = {
generatedAt: new Date().toISOString(),
mode: assistantMode ? "assistant-tool-loop" : "manifest",
generatorModel: process.env.OPENROUTER_GENERATOR_MODEL ?? DEFAULT_GENERATOR_MODEL,
judgeModel: process.env.OPENROUTER_JUDGE_MODEL ?? DEFAULT_JUDGE_MODEL,
providerBaseUrl,
generatorModel,
judgeModel,
results: results.map((result) => ({
caseId: result.caseId,
attempts: result.attempts,
packages/custom-widgets/src/test/ai-evaluation.spec.ts
+20
View File
@@ -7,11 +7,13 @@ import {
buildRepairPrompt,
DEFAULT_GENERATOR_MODEL,
DEFAULT_JUDGE_MODEL,
getAiProviderChatCompletionsUrl,
getDeterministicEvaluationIssues,
getEvaluationResponseFixtureText,
getJudgeResponseFormat,
judgePasses,
parseJudgeResult,
resolveAiEvaluationProviderConfig,
} from "../../scripts/ai-evaluation";
import type { CustomWidgetJudgeResult } from "../../scripts/ai-evaluation";
@@ -178,6 +180,24 @@ describe("AI authoring evaluation", () => {
expect(format.json_schema.schema).toMatchObject({ type: "object", additionalProperties: false });
});
it("can run the same evaluation against the Homarr provider endpoint", () => {
expect(getAiProviderChatCompletionsUrl("https://homarr.dev/api/ai/v1/")).toBe(
"https://homarr.dev/api/ai/v1/chat/completions",
);
expect(getAiProviderChatCompletionsUrl()).toBe("https://openrouter.ai/api/v1/chat/completions");
expect(
resolveAiEvaluationProviderConfig({
AI_PROVIDER_BASE_URL: "https://homarr.dev/api/ai/v1/",
AI_PROVIDER_API_KEY: "workshop-token",
}),
).toEqual({
apiKey: "workshop-token",
baseUrl: "https://homarr.dev/api/ai/v1",
generatorModel: "homarr/deepseek-v4-flash-latest",
judgeModel: "homarr/deepseek-v4-flash-latest",
});
});
it("requires exceptional goal, design, practicality, and complexity scores", () => {
expect(judgePasses(makeJudgeResult(90))).toBe(true);
expect(
packages/db/schema/mysql.ts
+1
View File
@@ -488,6 +488,7 @@ export const assistantConfigurations = mysqlTable("assistant_configuration", {
webSearchEnabled: boolean().notNull().default(false),
provider: varchar({ length: 32 })
.$type<
| "homarr"
| "openrouter"
| "openai"
| "anthropic"
packages/db/schema/postgresql.ts
+1
View File
@@ -486,6 +486,7 @@ export const assistantConfigurations = pgTable("assistant_configuration", {
webSearchEnabled: boolean().notNull().default(false),
provider: varchar({ length: 32 })
.$type<
| "homarr"
| "openrouter"
| "openai"
| "anthropic"
packages/db/schema/sqlite.ts
+1
View File
@@ -471,6 +471,7 @@ export const assistantConfigurations = sqliteTable("assistant_configuration", {
webSearchEnabled: int({ mode: "boolean" }).notNull().default(false),
provider: text()
.$type<
| "homarr"
| "openrouter"
| "openai"
| "anthropic"
packages/translation/src/lang/en.json
+2
View File
@@ -2091,6 +2091,8 @@
"loading": "Loading Homarr provider allowance",
"signInRequired": "Workshop sign-in required for the Homarr provider",
"signInDescription": "Sign in with GitHub through the Community Workshop to receive your separate daily allowance.",
"exhaustedDescription": "Your Homarr provider allowance is used for today. It resets at midnight UTC.",
"unavailableDescription": "Sign in to the Workshop or wait for your Homarr provider allowance to reset.",
"signIn": "Sign in with GitHub",
"reset": "Resets {relative} at",
"toolCalls": "Each model request and completed tool call uses one request.",