Repository navigation
Autoscale the AI Gateway by default - #119
Conversation
The gateway Deployment had a fixed replica count and no HPA, while the AWS module's gateway ECS service autoscales 2-6 tasks on CPU and memory. Add aiGateway.autoscaling, a standard autoscaling/v2 HPA on ai-gateway container CPU (70%) and memory (75%) with the same bounds and scale windows, and an optional aiGateway.podDisruptionBudget. Both default to disabled, so existing releases render unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 59131f4486
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| metadata: | ||
| name: {{ .Values.aiGateway.name }} | ||
| namespace: {{ include "braintrust.namespace" . }} | ||
| {{- with (merge (deepCopy .Values.aiGateway.labels) .Values.global.labels) }} |
There was a problem hiding this comment.
Add cross-template label-isolation coverage for both new resources: the current test only checks that disjoint component and global labels appear on one HPA, while neither the HPA nor PDB is tested for mutation leakage or conflicting-label precedence against other rendered resources. The repository explicitly requires a corresponding isolation test for every new template that merges labels.
AGENTS.md reference: AGENTS.md:L32-L35
Useful? React with 👍 / 👎.
Without the resource metrics API the HPA holds the gateway at minReplicas, which matches the previous fixed count of 2, so a default-on HPA does not regress clusters without metrics-server. Releases that already run the gateway briefly drop to 1 replica on the upgrade that removes the fixed Deployment replica count; the README documents how to avoid that. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Alex R (alexr17)
changed the title
Drop the disruption budget, HPA annotations, configurable scale windows, and the defaults/validation helper. Defaults are read inline with dig so --reuse-values upgrades from older charts still render. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Alex R (alexr17)
changed the title
Cover global/gateway label precedence on the HPA and confirm gateway labels do not reach another rendered component, per the label-merge test rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Codex (@codex) review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0443689873
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| type: Utilization | ||
| averageUtilization: {{ dig "cpu" "targetAverageUtilization" 70 $autoscaling }} |
There was a problem hiding this comment.
When an existing gateway configuration removes both aiGateway.resources.requests.cpu and .memory—for example, with null values retained by --reuse-values—these default-on utilization metrics have no request denominator. Kubernetes documents that utilization is undefined without the corresponding request and the autoscaler takes no action for that metric, so this HPA cannot scale the gateway under load even when metrics-server is healthy. Require the requests while autoscaling is enabled, use absolute-value targets, or preserve fixed replicas for this configuration, and add null/reused-values coverage. Kubernetes HPA resource metrics documentation
AGENTS.md reference: AGENTS.md:L14-L17
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 76d4062: rendering fails when the CPU or memory request is missing.
Fail rendering when autoscaling.minReplicas is below 1 or the gateway has no CPU or memory request, since utilization targets need both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Codex (@codex) review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bfc0edb057
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
The API HPA has neither check, the default values set requests, and Kubernetes rejects minReplicas 0 on its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fail rendering when replica bounds or utilization targets are ones Kubernetes rejects, or the gateway has no CPU or memory request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Codex (@codex) review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 76d4062841
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| - it: rejects zero minReplicas | ||
| set: | ||
| aiGateway.enabled: true | ||
| aiGateway.autoscaling.minReplicas: 0 |
There was a problem hiding this comment.
The validation tests cover only one input shape per setting, leaving null/empty replica bounds, null utilization targets, and partial nested CPU/memory maps on reused values untested. These exercise different Helm coalescing and dig paths from the covered zero/scalar cases, so the existing tests do not establish their intended rendering or rejection behavior. Add explicit cases for these shapes; the repository requires null/missing/empty/zero and partial reused-value coverage and explicitly treats unchecked input shapes as release blockers.
AGENTS.md reference: AGENTS.md:L14-L17
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
The new validation looks right. One upgrade issue is worth fixing (inline), plus a README note.
Not blocking: scale-in is new and the chart has no preStop hook, so in-flight requests can be cut when a pod stops. Worth a follow-up to confirm the gateway drains before exit.
| {{- $autoscaling := .Values.aiGateway.autoscaling | default dict }} | ||
| {{- if and .Values.aiGateway.enabled (dig "enabled" true $autoscaling) }} | ||
| {{- $minReplicas := int (dig "minReplicas" 2 $autoscaling) }} | ||
| {{- $maxReplicas := int (dig "maxReplicas" 6 $autoscaling) }} |
There was a problem hiding this comment.
A custom aiGateway.replicas is still replaced by these 2–6 defaults. One GKE install we run uses replicas: 4 and would settle at 2; a gateway at 10 would be capped at 6. We're setting autoscaling.minReplicas for that install. To cover others, take minReplicas and maxReplicas out of values.yaml and default them here to replicas and max(6, replicas). While 2 and 6 are in values.yaml, Helm supplies them on every normal install and upgrade, so a dig default of replicas never runs.
There was a problem hiding this comment.
Done in 34f88b3. minReplicas/maxReplicas are out of values.yaml and now default to aiGateway.replicas and max(6, minReplicas). I used minReplicas in the max so an explicit minReplicas above 6 also renders. replicas: 4 gives 4-6, and replicas: 10 gives 10-10. Explicit bounds still win.
|
|
||
| The AI Gateway autoscales by default. An HPA runs 2-6 replicas at 70% CPU and 75% memory, the same as the AWS module. It needs metrics-server, which GKE and AKS include; without it the gateway stays at 2. Set `aiGateway.autoscaling.enabled: false` to use a fixed `aiGateway.replicas`. | ||
|
|
||
| Turning on autoscaling for a running gateway briefly drops it to 1 replica until the HPA scales it back up. |
There was a problem hiding this comment.
This dip is roughly 30–60 seconds on one pod: up to about 15 seconds for the HPA to restore the count, then image pull and the readiness delay. Setting autoscaling.enabled: false only postpones it until autoscaling is turned on.
There was a problem hiding this comment.
Updated in 34f88b3. The README and PR description now say roughly 30-60s, and that autoscaling.enabled: false only postpones the dip.
minReplicas now defaults to aiGateway.replicas and maxReplicas to max(6, minReplicas), so a custom replica count is no longer reset to 2-6. The bounds leave values.yaml so the template default applies. README notes the 30-60s dip on enabling autoscaling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Codex (@codex) review |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback". |
Summary
Autoscale the AI Gateway by default, the same way the AWS module's gateway ECS service does: 70% CPU or 75% memory, scaling up after 60s and down after 300s.
minReplicasdefaults toaiGateway.replicas(2) andmaxReplicastomax(6, minReplicas), so a custom replica count stays the floor.ai-gateway-hpa.yaml(autoscaling/v2, ContainerResource on theai-gatewaycontainer).replicas.aiGateway.autoscaling.enabled: falserestores fixedaiGateway.replicas.dig, so--reuse-valuesupgrades from older charts still render.Without metrics-server (for example, stock EKS), the HPA holds
minReplicas, the same count as today.Input matrix
replicasreplicas, no boundsminReplicas=replicas,maxReplicas=max(6, replicas)replicasnullautoscaling.enabled: falsereplicas--reuse-valuesfrom an older chart, with or without a partial--setminReplicasorreplicas< 1 or empty,maxReplicas<minReplicas, zero or empty targets, missing CPU or memory requestEach row has a test in
ai-gateway-autoscaling_test.yaml. HPA label isolation tests are inlabels-merge-autoscaling_test.yaml.Upgrade notes
On a release already running the gateway, the upgrade drops it to 1 replica for roughly 30–60s: up to about 15s for the HPA to restore the count, then image pull and the readiness delay.
autoscaling.enabled: falseonly postpones this until autoscaling is turned on.Chart.yamlisn't bumped.Validation
./test.shpasses (466/466).kubeconform -strictagainst Kubernetes 1.30 passed on aws, azure, and google at 76d4062 (not rerun for the replicas-default change). Not tested on a live cluster.🤖 Generated with Claude Code
Co-authored by StarfolkAI (@starfolkai)[bot]