Skip to content

peering: settlement failures retry forever with no attempt count or operator signal #1133

Open

Description

@Ferryx349

Follow‑up from the re‑review of #1087 (N1).

Problem

Since #1087, a peering invoice that is COMPLETED with confirmed_at = null is selected by findUnconfirmedPeeringInvoices (src/repositories/invoice-repository.ts:71-92) and retried on every maintenance pass, ahead of the pending page (src/app/maintenance-worker.ts:199). That is the intended fix for the stranded‑settlement bug.

What is missing is failure handling. The worker catches a settlement error, logs it, and moves on (src/app/maintenance-worker.ts:303). Nothing increments an attempt counter and nothing is enqueued. settlePeeringInvoice throws deterministically for a missing peer row or a missing term snapshot (src/services/payments-service.ts:426-434), and only the underpaid branch writes last_error.

Net effect: an invoice whose settlement can never succeed is retried indefinitely, every pass, with no operator‑visible state. Before this PR such an invoice was stranded once and at least looked stuck; now it is actively retried but still invisible. Operators cannot distinguish "will succeed on a later pass" from "broken forever", and the only symptom is log volume.

Proposed fix

Persist settlement attempts on the invoice (e.g. settlement_attempts + last_settlement_error), incremented when the peering settlement path throws.
After a configurable threshold N (default 5), enqueue an operator notification through the notification outbox — e.g. OperatorNotificationEventType.PEERING_PAYMENT_FAILED — carrying invoiceId, pubkey, peer id, attempt count and the last error. Emit it once per invoice at the threshold, not every pass.
Keep retrying (or back off) after the alert; do not discard the invoice.
Make sure the idempotent duplicate path (src/services/payments-service.ts:437-440) is not counted as a failure.

Activity

  1. self-assigned this
    on Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingpeering

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions