[xgreenx]

Same change as #3325, targeted at master instead of release/v0.48.2. #3325 exists so the fix can ship on the deployed v0.48.2 line; this PR keeps master correct. Cherry-picked cleanly — the three touched files are byte-identical between v0.48.2 and master.

Problem

Fuel mainnet's public p2p sentries run 10-45 blocks behind the block producer, while every internal-only sentry sits at 0-8. Measured against a healthy sentry over 579 common blocks: p50 16.6-21.5s behind, p95 up to 42s, max 54s, with 72-75% of wall clock spent inside a stalled range.

The cycle, from production logs:

00:50:24.520  Committed block 61196157
00:50:36.669  RequestResponse outbound error for peer: PeerId("16Uiu2HAmPYbwn…") error: Io
00:50:36.669  Failed to get headers … "Hit the end of buffer, expected more data"
00:50:36.669  Failed to import range of blocks: 61196158..=61196159
   … repeats at 00:50:56.676, 00:51:16.686, 00:51:36.694 — exactly 20.000s apart, same peer …
00:51:38.147  Committed block 61196158        <- 73.6s later

100% of header-fetch failures paired within 1ms with a RequestResponse outbound error naming one external peer. Two root causes:

1. Transport failures never penalize the peer. crates/services/sync/src/import.rs does .trace_err("Failed to get headers").ok(), discarding the error, then returns Batch::new(None, range, vec![]), so report_peer is called with peer_id: None and silently no-ops. The peer id is available in the OutboundFailure handler in p2p_service.rs, which only logged it. Net effect: appScore is ~0.0 for every peer, including one responsible for ~100% of failures, one 21,599 blocks behind, and one 1,831,510 blocks behind — all still connected.

The contrast is stark: a peer that answers "here are 3 of your 10 headers" is banned forever (MissingBlockHeaders, -100), while a peer that times out 248 times is untouched.

2. Peer selection ignores reputation and gives reserved peers no preference. get_peer_id_with_height chained non_reserved_connected_peers with reserved_connected_peers and did a uniform .choose(), with an explicit // TODO: Optimize the selection of the peer. The bad peer advertised the highest height, so it always passed the height filter and was re-drawn indefinitely.

Changes

(A) handle_request_response_event reports the peer from the OutboundFailure arm via a new is_peer_at_fault helper. Penalized: Timeout, ConnectionClosed, Io(_). Not penalized: UnsupportedProtocols (already handled by disconnect — avoids double-punishing) and DialFailure (may be our own connectivity; the peer never received the request). The match is exhaustive so a libp2p bump forces a decision. Reserved peers remain exempt — update_app_score only touches non_reserved_connected_peers; that was previously incidental and is now documented and test-pinned.

(B) get_peer_id_with_height picks uniformly among qualifying reserved peers, falling back to uniformly among qualifying non-reserved peers. Height-filter semantics unchanged (extracted to a peers_with_height helper).

Penalty magnitude: -48

Decay is 0.9 applied every second against a MIN_APP_SCORE of -50. A peer failing once per 20s --request-timeout converges to p/(1 - 0.9²⁰) = 1.14·p, so anything weaker than -43.9 can never ban — reusing the heartbeat's -5 would settle at -5.7 forever and ship a no-op. Reusing -100 bans on a single transient blip. The feasible band is 43.9 < |p| ≤ 50.

-48 gives: one failure → no ban, back above -0.5 in 44s; ban requires two failures within 30.2s, ~50% margin over the observed 20s cadence. (-44/-45 also nominally work but leave <1s of margin, which jitter would erase.)

Tests

cargo test -p fuel-core-p2p --features test-helpers → 63 passed, 0 failed, 2 ignored on this base.

Added: reserved preference over 100 draws; fallback + None when nothing qualifies; reserved score untouched while non-reserved is scored and banned; a single failure doesn't ban and decays away; two failures at the 20s cadence do ban; a per-variant fault table for is_peer_at_fault. Also extended req_res_outbound_timeout_works to assert the timed-out peer's score goes negative — it stayed at 0.0 before, so the existing test was quietly encoding the bug.

Risk

With block_stream_buffer_size: 10, a single 20s blip can produce two concurrent failures against the same peer and ban it permanently for the process lifetime — libp2p::allow_block_list has no TTL and there is no unblock_peer anywhere in the codebase. A follow-up giving banned peers a route back (time-based unban, or score-driven removal from the block list) is worth doing separately.

Note also that the culprit peer rotated within a day in production, which is why (B) matters more than (A): it makes the class of problem irrelevant regardless of which peer misbehaves.

🤖 Generated with Claude Code

[cursor]

PR Summary

Medium Risk
Changes P2P sync peer selection and banning thresholds; repeated failures can permanently block-list a peer for the process lifetime, though reserved operator peers are protected.

Overview
Fixes block sync stalls where a misbehaving public peer could be re-selected forever after transport-level header fetch failures, while reputation stayed near zero.

Request/response reputation: On OutboundFailure, non-reserved peers that time out or return unreadable I/O responses get a -48 app score via report_peer (p2p_request_response). is_peer_at_fault skips dial failures, unsupported protocols, and ConnectionClosed (disconnect ordering makes scoring a no-op). Reserved peers remain exempt from update_app_score.

Block-fetch peer choice: get_peer_id_with_height now picks randomly among reserved peers at the height first, then falls back to non-reserved peers (peers_with_height helper). Docs and tests cover reserved preference, fallback, penalty decay/ban behavior, and the outbound timeout integration test now expects a negative score after timeout.

^{Reviewed by Cursor Bugbot for commit ae425bc. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ae425bc. Configure here.}