[shurkus]

Type of Change

• New Feature
• Bug Fix
• Documentation
• Performance Improvement
• Test/CI
• Refactor
• Other:

Related Issues

Closes #289

Summary of Changes

The operator now manages one PodDisruptionBudget per Tenant pool, so node drains and autoscaler scale-downs cannot evict more pool Pods than the pool's erasure sets can lose while keeping write quorum for the STANDARD storage class.

• New pod_disruption_budget reconcile step after the pool StatefulSets. The budget is named <tenant>-<pool>, owned by the Tenant, carries the pool labels and selects the pool Pods.
• minAvailable = servers minus the servers that can be down while every erasure set keeps write quorum. It is derived from the layout RustFS reports in /rustfs/admin/v3/info (drive pool_index, set_index, endpoint host, state) and the reported standard parity. When pools resolve different parities RustFS reports none, and each set uses the automatic parity for its size, as RustFS does.
• Offline drives count against the allowance, since Pods stay Ready with offline drives. When they use it up, or when RustFS cannot report the layout, the budget is set to minAvailable = servers, and the layout is re-read every 60s.
• Pools that cannot lose a server even with all drives online (single-server, two-server, or dense layouts such as 3 servers x 4 volumes with EC:2) get no budget.
• Budgets are created or updated only while all pools are settled (rolled out, ready, no decommission).
• Ownership: a budget is created with POST and updated with a forced server-side apply only when the Tenant owns it. A same-name budget without the Tenant owner, including one created concurrently (409 on create), is left unchanged and reported with a PodDisruptionBudgetConflict warning event.
• New optional field spec.disablePodDisruptionBudget (default: budgets on). Setting it deletes only the budgets owned by the Tenant; budgets of removed pools are deleted as well.
• rustfs-admin: RustfsServerInfo now deserializes servers[].drives[].
• RBAC (chart and deploy/k8s-dev): policy/poddisruptionbudgets get, list, watch, create, patch, delete. The controller watches owned budgets.
• CRD regenerated, user guide (EN and zh-CN) and CHANGELOG updated, including the storage class scope and the upgrade procedure.

Checklist

• I have read and followed the CONTRIBUTING.md guidelines
• Passed make pre-commit (fmt-check + clippy + test + console-lint + console-fmt-check)
• Added/updated necessary tests
• Documentation updated (if needed)
• CHANGELOG.md updated under [Unreleased] (if user-visible change)
• CI/CD passed (if applicable)

Impact

• Breaking change (CRD/API compatibility)
• Requires doc/config/deployment update
• Other impact: budgets are enabled by default and change drain behavior for existing Tenants

Upgrade procedure (also in the user guide and CHANGELOG):

1. Upgrade the chart first: the CRD gains disablePodDisruptionBudget and the operator ClusterRole gains PodDisruptionBudget permissions.
2. Tenants whose budgets are managed by the user: set spec.disablePodDisruptionBudget: true before the new operator image starts.
3. Otherwise remove user-created budgets that select pool Pods. Kubernetes rejects the eviction of a Pod selected by more than one budget, so with both in place a drain cannot evict any Pod of the pool. A same-name user budget is never modified or adopted.

Verification

make pre-commit
cargo test --lib pod_disruption_budget

Additional Notes

• Storage class scope: the guarantee covers STANDARD objects at the current parity. REDUCED_REDUNDANCY objects (EC:1 by default) and objects written with a lower standard parity before a configuration change are not covered; including them would allow at most one drive per set and leave most multi-drive layouts without a budget. This is documented in the user guide.
• Reconcile tests run the step against a mocked API server: create, update of an owned budget, heterogeneous parity from the real admin response shape (no standardSCParity), a pool without spare servers, disabling (only Tenant-owned budgets, uid precondition), a removed pool while not settled, an unreadable layout and an offline drive tightening the budget to the server count, a same-name budget without the Tenant owner, and a budget created concurrently by someone else.
• Unit tests cover 3x1, 4x1 EC:2, 2x1, 8x2 in one 16-drive set with EC:4 (2 servers, 1 with an offline drive), 3x4 with EC:4 and EC:2, single-server pools, automatic per-set parity, offline and missing drives, and other pools' drives.
• There is no "majority of servers" fallback when the layout is unknown: majority is not safe when a server holds several drives of one set or parity is below the default (8x2 with EC:4 tolerates 2 servers, majority would allow 3).
• The operator deletes Pods directly, not through the eviction API, so the budgets do not block its own Pod cleanup or StatefulSet rolling updates.

[GatewayJ]

Per-pool PodDisruptionBudget management fits the existing reconcile flow. Please address the following correctness and resource-ownership cases before merging.

1. [P1] Tighten the budget when drive failures consume the remaining fault tolerance. In src/reconcile/pod_disruption_budget.rs:104-113, an unhealthy drive causes the existing budget to be retained. With 8 Pods, 2 drives per Pod, one erasure set, and EC:4, the budget is minAvailable: 6. After one drive fails, all Pods can remain Ready while 15 drives are online. The unchanged budget still permits eviction of two healthy Pods, leaving 11 drives against a write quorum of 12. RustFS 1.0.0 readiness permits drive failures while quorum remains available, and Kubernetes counts Pod readiness rather than healthy drives. Please account for existing drive failures when calculating the remaining disruption allowance, and tighten the affected budget when a safe allowance cannot be established. The current unhealthy_drive_keeps_the_pool_budget_and_retries test preserves this unsafe behavior. References: RustFS readiness, Kubernetes PDB healthiness.

2. [P1] Account for supported storage classes and existing object layouts. At src/reconcile/pod_disruption_budget.rs:259-264, the allowance uses only STANDARD parity. For 8 Pods with one drive each and STANDARD EC:4, minAvailable: 5 allows two Pods to be unavailable. An existing REDUCED_REDUNDANCY object written with the default EC:1 requires 7 data shards, so its GET fails with only 6 drives available even though the PDB is satisfied. Please define the storage classes covered by the availability guarantee and calculate the required quorum accordingly, including existing objects with lower parity. References: default RRS parity, persisted object quorum.

3. [P2] Support heterogeneous per-pool parity in the actual admin response. At src/reconcile/pod_disruption_budget.rs:157-161, a missing scalar standardSCParity rejects the entire layout. RustFS 1.0.0 returns no scalar when pool parity values differ. For example, automatically configured sets of 4 and 8 drives use parity 2 and 4. This valid configuration repeatedly requeues every 60 seconds without creating budgets for any pool. Please obtain each pool's effective parity and add a test using the real heterogeneous admin response shape. Reference: RustFS homogeneous parity calculation.

4. [P2] Verify ownership before applying a same-name PDB. At src/reconcile/pod_disruption_budget.rs:118-120, ctx.apply() can add a Tenant owner reference to an existing user-managed PDB when its name, minAvailable, and selector already match. SSA permits shared ownership of equal field values, so omitting force does not prevent this. Subsequently setting disablePodDisruptionBudget: true, or deleting the Tenant, can delete that user-managed PDB. This violates the documented promise to leave externally created budgets untouched. Please verify ownership before updating an existing object, handle creation races with the same ownership check, and test a same-name, matching-spec PDB without a Tenant owner. Reference: Kubernetes SSA field ownership.

Upgrade compatibility needs explicit attention. The optional CRD field preserves existing manifest parsing, while default enablement changes drain behavior for existing Tenants. A user-managed PDB with a different name and an overlapping selector causes Kubernetes to reject evictions. The user guide already mentions migration; please make the upgrade procedure explicit: update the CRD and RBAC, provide a way to retain user-managed budgets through the opt-out before automatic creation, and preserve external resource ownership. Please record this operational compatibility impact in the PR/release notes. Reference: overlapping PDB selectors.

Reviewed base 30f695a018f27dffefdedb52263a1876d0ee0c61 and head e44d1242d68b764641ab20e8a480ee15ed625be8, with server behavior checked against the default RustFS 1.0.0 image source. The 14 focused tests passed locally with cargo test --locked --lib pod_disruption_budget -j 6; git diff --check passed. GitHub Test and Lint passed; Docker Build was skipped. Real Kubernetes eviction and SSA integration scenarios were not executed. Correctness and compatibility findings are listed above; the overall module placement is appropriate. Regression coverage is needed for the reported cases.

[shurkus]

Addressed in 10532dd (rebased onto main):

1. [P1] Drive failures. Offline drives now count against the allowance: each set uses its online drives minus write quorum, divided by the drives one server holds. When offline drives use up the allowance, minAvailable equals the server count, so no Pod of the pool can be evicted; the layout is re-read every 60s while drives are offline. If RustFS cannot report the layout, existing budgets are tightened the same way instead of being kept. Tests: offline_drives_reduce_the_allowance (your 8x2 EC:4 case: 2 servers healthy, 1 with one offline drive), offline_drive_that_consumes_the_allowance_blocks_evictions, unreadable_layout_blocks_evictions_through_existing_budgets.
2. [P1] Storage classes. The guarantee is defined as STANDARD at its current parity, and the user guide now says that REDUCED_REDUNDANCY objects (EC:1 by default) and objects written with a lower standard parity before a configuration change can become unreadable while a drain keeps the pool at minAvailable. Covering RRS would cap every set at one drive and leave most multi-drive layouts (for example 8x2) without any budget.
3. [P2] Heterogeneous parity. A missing standardSCParity no longer rejects the layout. RustFS reports no scalar only when pools resolve different parities, which happens only with automatic parity (an explicit EC:N applies to every pool), so each set then uses default_parity_count for its size, as lookup_config_for_pools does. Tests: missing_parity_uses_the_automatic_parity_of_each_set and heterogeneous_pool_parity_from_the_admin_response, which parses the 1.0.0 response shape with "standardSCParity": null and expects 3 for a 4-drive pool and 5 for an 8-drive pool.
4. [P2] Ownership. A budget is now created with POST and updated with a forced server-side apply only when the Tenant owns it. A same-name budget without the Tenant owner is never patched; the same check runs after a 409 on create. It is reported with a PodDisruptionBudgetConflict warning event. Tests: same_name_budget_without_tenant_owner_is_left_unchanged (matching spec) and budget_created_concurrently_by_someone_else_is_left_unchanged.

Upgrade compatibility. The user guide, CHANGELOG and the PR Impact section now describe the procedure: upgrade the chart (CRD and RBAC) first; set spec.disablePodDisruptionBudget: true before the new image starts on Tenants whose budgets are managed by the user; otherwise remove user budgets that select pool Pods, since overlapping selectors make Kubernetes reject evictions.

make pre-commit passes (825 Rust tests); CI is green.