fix(gridfs): close remaining upload races and stream append safely

.github/workflows/gridfs-fix-validation.yml

@@ -42,7 +42,10 @@ jobs:
         run: |
           node --test --test-concurrency=1 \
             .generated/test-dist/test/unit/gridfs/gridfs.test.js \
+            .generated/test-dist/test/unit/gridfs/stream-chunks.test.js \
             .generated/test-dist/test/integration/gridfs/gridfs.test.js \
             .generated/test-dist/test/integration/gridfs/gridfs-scan-regressions.test.js \
             .generated/test-dist/test/integration/gridfs/gridfs-ownership-regressions.test.js \
-            .generated/test-dist/test/integration/gridfs/gridfs-stale-recovery.test.js
+            .generated/test-dist/test/integration/gridfs/gridfs-stale-recovery.test.js \
+            .generated/test-dist/test/integration/gridfs/gridfs-remaining-issues.test.js \
+            .generated/test-dist/test/integration/gridfs/gridfs-inspect-orphans.test.js

config/package-budget.json

@@ -1,8 +1,8 @@
 {
-  "comment": "The 3.5.3 bundle ceilings were 881096 CJS and 878096 ESM. The GridFS cross-operation ownership, durable session CAS/abort fencing, delete protection and idle-TTL migration add ~4.4/4.7 KiB over that ceiling (CI build 2026-11-12: 885536 CJS / 882819 ESM). Raise to 890000 / 887000, keeping ~4 KiB reviewed headroom; no dependency change. Enforce these ceilings in release preflight.",
+  "comment": "GridFS session identity checks bring the 2026-11-13 build to 897830 CJS and 895065 ESM bytes, with no dependency change. Release preflight enforces these ceilings.",
   "artifacts": {
-    "dist/cjs/index.cjs": 890000,
-    "dist/esm/index.mjs": 887000,
+    "dist/cjs/index.cjs": 898000,
+    "dist/esm/index.mjs": 895500,
     "dist/cjs/cli/data-task.cjs": 50000
   }
 }

docs/en/gridfs-recovery.md

@@ -6,7 +6,14 @@
 
 The ownership registry is the per-bucket `<bucket>.uploads` collection. A reservation excludes ordinary upload, uploadStream, resumable append/finalize/abort, delete, rename and orphan cleanup for the same ObjectId. A missing file document **does not** prove its chunks are orphaned: an ordinary upload may still be writing, or GridFS driver delete may have removed the files document before chunks. Recovering while an old writer can resume can destroy new data, even with a token-CAS. The maintenance process cannot prove cluster/process shutdown on its own.
 
-The script `scripts/maintenance/gridfs-recover.cjs` uses a default read-only preview, exact state/token/timestamp checks, a persisted `recovering` ownership claim, verified preservation of committed visible chunks, and an append-only-by-intent audit collection `<bucket>.ownership_recovery_audit`. It never performs a time-based lock steal.
+The script `scripts/maintenance/gridfs-recover.cjs` uses a default read-only preview, exact state/token/timestamp checks, a persisted `recovering` ownership claim, verified preservation of committed visible chunks, and an append-only-by-intent audit collection `<bucket>.ownership_recovery_audit`. It never performs a time-based lock steal. Dry-run plans may use a pre-claim visibility snapshot; **apply always re-reads `files` after the exclusive claim**. If visibility changed, or `verifyVisibleFile` cannot confirm a visible file that the plan required, the tool fails closed and leaves the `recovering` owner in place.
+
+To survey leftover chunks with ObjectId file IDs, use the inspect-only reporter `scripts/maintenance/gridfs-inspect-orphans.cjs`. It walks `<bucket>.chunks` in index order in bounded batches, re-checks `files` after finding neither a file nor an owner for a `files_id`, and reports `visible_file`, `owned`, or `candidate_orphan`. It has no apply mode, does not delete, and does not use document age as a deletion criterion. It is not part of the public `GridFSBucket` API. The CLI emits one NDJSON finding per line plus a final summary object. `--max-findings` (default 1000, cap 100000) bounds memory; a truncated scan returns `resumeAfter.filesId` for the next `--after-files-id`. Summary counters accumulate incrementally.
+
+```sh
+export MONSQLIZE_RECOVERY_MONGODB_URI='<MongoDB URI from the secrets manager>'
+node scripts/maintenance/gridfs-inspect-orphans.cjs --db=my_database --bucket=fs --page-size=200 --max-findings=1000
+```
 
 ## Deployment and rollback gates (required)
 
@@ -54,6 +61,7 @@ Every apply attempt includes an audit entry. A completed visible-file verificati
 | `kind:direct,status:uploading` | Verify chunk sequence, per-chunk length and total length. Enforce SHA only when the owner records a pre-verified `upload()` expected digest; otherwise record any metadata mismatch as an audit warning | `uploadStream()` and arbitrary `metadata.sha256` are untrusted claims; visible-file chunks are never deleted |
 | `kind:delete,status:deleting` | Complete the interrupted delete of file and chunks before releasing ownership | No new upload may claim the same ID until both steps finish |
 | `kind:cleanup,status:cleaning` | Check visible file; delete chunks only if no file exists; release | Never delete a visible file's chunks |
+| `kind:resumable,status:initializing` | Preserve visible chunks or remove invisible leftovers, then release the initialization claim | Retry `beginResumableUpload` after recovery if no file is visible |
 | `kind:rename,status:renaming` | Verify the resulting visible file has complete chunks, then release; missing or corrupt files remain locked | The tool does not choose or replay a rename target; the operator must inspect the intended filename |
 | `status:pending,activeAppend` | Recompute the contiguous chunk prefix, clear only the observed append claim, restore seven-day idle expiry | Existing chunks are preserved; caller replays/checks the data |
 | `status:finalizing` without file | Restore pending with expiry; retry normal finalize, which revalidates bytes | Never discard chunks to unlock finalize |

docs/en/gridfs.md

@@ -46,7 +46,7 @@ Calling either method before `connect()` throws `NOT_CONNECTED`.
 `upload` defaults to `ifExists: 'error'`.
 
 - When `sha256` is set, the conflict key is `metadata.sha256`. The digest must be 64 lowercase hex characters. `Buffer` and `Uint8Array` payloads are hashed before any GridFS write. A mismatch throws `GRIDFS_INTEGRITY_MISMATCH` and leaves no visible file.
-- A `Readable` passed with `sha256` is buffered completely, then hashed, before any write. Large files should use the resumable API instead.
+- A `Readable` passed with `sha256` is buffered completely, then hashed, before any write. That is required to honor the contract that a failed verification leaves no visible file. `upload()` without `sha256` can stream, but the whole payload is still a single commit. For large files, use the resumable API with `appendChunks` and a `Readable` whose `highWaterMark` stays modest; do not pass a multi-gigabyte `Buffer` or a fully-buffered `upload({ sha256 })`.
 - `skip` with `sha256` returns the newest visible file with that digest. `skip` without `sha256` returns the newest visible file with the same filename. `duplicate` always creates another file.
 - A `metadata.sha256` value supplied inside `metadata` is caller data and is not verified. To request byte verification with `upload()`, use the top-level `sha256` option.
 - `uploadStream` stores its top-level `sha256` option in metadata only. It does not check the bytes and it does not accept `ifExists`.
@@ -75,13 +75,13 @@ const info = await bucket.finalizeResumableUpload(session.uploadId);
 await bucket.abortResumableUpload(session.uploadId);
 ```
 
-`beginResumableUpload` requires `length` and `sha256`. An id that already exists in `files` or `uploads` throws `GRIDFS_FILE_EXISTS`.
+`beginResumableUpload` requires `length` and `sha256`. An id already present in `files` or `uploads` throws `GRIDFS_FILE_EXISTS`. The insert claims the id in `initializing` without expiry. It rechecks `files` and cleans old chunks before making the session `pending`. While initialization runs, other writes to that id are blocked. If a file became visible before the recheck, the call preserves its chunks and throws `GRIDFS_FILE_EXISTS`. An interrupted initialization can be cleared through operator recovery after the writer has stopped.
 
 Chunk bytes must follow the session size:
 
 - Every chunk except the last has length `chunkSizeBytes`.
 - The last chunk has length `length % chunkSizeBytes`, or `chunkSizeBytes` when the length divides evenly.
-- A payload that does not split into those sizes throws `INVALID_ARGUMENT`. The whole payload is checked before any chunk is inserted.
+- A payload that does not split into those sizes throws `INVALID_ARGUMENT`. For a `Buffer` or `Uint8Array`, the whole payload is planned and checked before any chunk is inserted. For a `Readable`, complete chunks are written as they arrive; a trailing partial or oversize remainder is rejected during consumption, after any complete prefix chunks have been stored. A failed append throws and does not return `nextChunk`. Resume from the last successfully acknowledged position; if the stored prefix has moved, `GRIDFS_RESUME_MISMATCH` includes `expectedNextChunk`. A source error during automatic destruction after EOF also rejects the append. Memory use is bounded by the current stream buffer, the partial chunk, and a fixed-size batch of stored chunk indexes. Use a modest `highWaterMark` for large files.
 
 `appendChunks` is addressed by `fromChunk`:
 
@@ -92,9 +92,11 @@ Chunk bytes must follow the session size:
 - A payload that contains any new chunk must start at `nextChunk`. Starting earlier throws `GRIDFS_RESUME_MISMATCH` and writes nothing.
 - A duplicate `{ files_id, n }` insert is accepted only when the stored bytes match.
 
-`finalizeResumableUpload` requires a contiguous prefix, a total length equal to the session length, and a sha256 equal to the session digest. It then inserts the files document, sets `metadata.sha256`, and deletes the session. Missing chunk indexes throw `GRIDFS_UPLOAD_INCOMPLETE` with `details[0].missingChunks`. A bad digest or a chunk of the wrong size throws `GRIDFS_INTEGRITY_MISMATCH`. The files document is not inserted in those cases.
+`finalizeResumableUpload` requires exactly one chunk at each expected index starting at zero, a total length equal to the session length, and a sha256 equal to the session digest. It then inserts the files document, sets `metadata.sha256`, and deletes the session. Missing chunk indexes throw `GRIDFS_UPLOAD_INCOMPLETE`. `details[0].missingChunks` lists the first 1,024 missing indexes at most. When truncated, `details[0].missingChunkCount` gives the total. An invalid, duplicate, or out-of-range chunk index, a bad digest, or a chunk of the wrong size throws `GRIDFS_INTEGRITY_MISMATCH`. The files document is not inserted in those cases.
 
-Retrying `finalizeResumableUpload` is idempotent while the session still exists: if a file with the same id is already visible and its `metadata.sha256` and length match the session, the call returns that file and deletes the session. A file with a different sha256 or length still throws `GRIDFS_FILE_EXISTS`. Once the session is gone (a finalize that fully succeeded, an abort, or the TTL), there is no digest to compare, so a retry throws `GRIDFS_FILE_NOT_FOUND`; use `getFile(uploadId)` to check the result.
+Retrying `finalizeResumableUpload` is idempotent while the session still exists: if a file with the same id is already visible, the library re-hashes the actual stored chunks and returns the file only when that digest, the file length, and the chunk size match the session. A different sha256 or length throws `GRIDFS_FILE_EXISTS`. A chunk-size mismatch throws `GRIDFS_INTEGRITY_MISMATCH` and retains the existing finalizing session. A retry that observes a different upload under the same ID returns `GRIDFS_UPLOAD_BUSY`. Once the session is gone (a finalize that fully succeeded, an abort, or the TTL), there is no digest to compare, so a retry throws `GRIDFS_FILE_NOT_FOUND`. Callers must persist `uploadId`, `length`, and `sha256` themselves. After a vanished session, use `getFile(uploadId)` to see whether a file is visible, then re-hash the **actual** bytes (for example `download(uploadId)` or a walk of `<bucket>.chunks`) and compare them to the persisted length and digest. Never treat `metadata.sha256` alone as proof: `uploadStream()` and a raw `metadata.sha256` can store an unverified claim. This package does not add a separate public verification API.
+
+A `finalizeResumableUpload` that fails with `GRIDFS_FILE_EXISTS` or `GRIDFS_INTEGRITY_MISMATCH` after it has claimed `finalizing` rolls the session back to `pending` so `abortResumableUpload` (and then a delete of a conflicting visible file) can proceed through the public API. Ambiguous driver errors during the files insert still leave `finalizing` for explicit recovery.
 
 `abortResumableUpload` atomically claims a pending session before deleting chunks. It fails with `GRIDFS_UPLOAD_BUSY` while append or finalize owns the session. It deletes the session and, when no files document is visible, that upload's chunks. A missing or already aborted session resolves, and any chunks left under that id without a visible file are still deleted. Aborting an id that already has a visible file does not delete that file's chunks.
 
@@ -136,7 +138,7 @@ A blocked call throws `INVALID_OPERATION` before any GridFS write. `details[0]`
 |------|------|
 | `GRIDFS_FILE_NOT_FOUND` | Download, delete, rename, or resumable session target is missing |
 | `GRIDFS_FILE_EXISTS` | `ifExists: 'error'` finds a conflict, or the chosen id is already present |
-| `GRIDFS_INTEGRITY_MISMATCH` | Bytes do not match `sha256`, or a finalized chunk has the wrong size |
+| `GRIDFS_INTEGRITY_MISMATCH` | Bytes do not match `sha256`, or finalize found an invalid chunk index or size |
 | `GRIDFS_UPLOAD_INCOMPLETE` | Finalize found gaps. `details[0].missingChunks` lists them |
 | `GRIDFS_RESUME_MISMATCH` | `fromChunk` is ahead of the contiguous prefix, or a replayed chunk differs |
 | `GRIDFS_UPLOAD_BUSY` | Another append/finalize/abort/upload/delete operation owns the ID; do not retry until it finishes or is explicitly recovered |

docs/en/support-matrix.md

@@ -18,7 +18,7 @@
 | MongoDB Driver | 6.x / 7.x | `npm run test:server-matrix` |
 | MongoDB Server | 7.x / 8.x | strict `mongodb-memory-server` 7.0.37 / 8.0.26 single + replica set matrix |
 
-GridFS is a client convention on `<bucket>.files` and `<bucket>.chunks`. Its integration tests run on `mongodb-memory-server` 7.0.37. That does not add MongoDB 4.x, or any server outside 7.x / 8.x, to this matrix.
+GridFS is a client convention on `<bucket>.files` and `<bucket>.chunks`. Its integration tests run on `mongodb-memory-server` 7.0.37 and 8.0.26 in the current GridFS safety / release-gate jobs. That does not add MongoDB 4.x, or any server outside 7.x / 8.x, to this matrix. Search and Vector Search against Atlas, atlas-local, or mongot remain UNVERIFIED in this checkout; those deployments are not implied by the memory-server matrix.
 
 ## Default verification method
 

docs/zh/gridfs-recovery.md

@@ -8,6 +8,16 @@
 
 任何强制恢复必须先从集群所有节点停止并排空旧版和新版 GridFS 写入进程、队列消费者、定时任务与自动重启，确认没有仍可恢复的旧写入。恢复脚本无法在 MongoDB 层直接阻止旧进程已发出的 chunks 写入；只有外部进程停机隔离后才安全。不得混用忽略 ownership 的旧版与新版本。
 
+dry-run 可以使用领取所有权之前的可见性快照；**apply 必须在独占 claim 之后重新读取 `files`**。若可见性已经变化，或在需要可见文件时 `verifyVisibleFile` 无法确认，脚本失败并保留 `recovering` 所有权。
+
+排查使用 ObjectId 的残留 chunks 时，使用只读巡检脚本 `scripts/maintenance/gridfs-inspect-orphans.cjs`。它按已有 `{ files_id, n }` 索引分批扫描 `<bucket>.chunks`。分类时若未查到 `files` 和 `uploads` 记录，会再次核对 `files`，报告 `visible_file`、`owned` 或 `candidate_orphan`。没有 apply 模式，不会删除，也不会按文档年龄删除。它不是公开的 `GridFSBucket` API。CLI 按行输出 NDJSON 结果及一条摘要。`--max-findings`（默认 1000，上限 100000）限制单次内存；截断时用 `resumeAfter.filesId` 作为下次的 `--after-files-id`。摘要计数在扫描过程中增量累计。
+
+```sh
+export MONSQLIZE_RECOVERY_MONGODB_URI='<从密钥管理系统读取的 MongoDB URI>'
+node scripts/maintenance/gridfs-inspect-orphans.cjs --db=database --bucket=fs --page-size=200 --max-findings=1000
+```
+
+
 ## 上线和回滚步骤
 
 1. 冻结所有 GridFS 写入，禁止新请求和自动重启，核查所有旧版 Writer 已排空；保留相关运行实例、进程、队列、版本和时间证据。
@@ -49,6 +59,7 @@ node scripts/maintenance/gridfs-recover.cjs \
 | direct/uploading | 验证 chunks 连续性、每块长度和总长度；只有 owner 记录了 upload() 写入前验证过的 expected SHA 时才强制校验摘要，否则将 metadata 不一致记为审计 warning | uploadStream() 与任意 metadata.sha256 都是不可信声明；绝不删除可见文件 chunks |
 | delete/deleting | 继续删除 files 和 chunks，之后才释放同 ID 的所有权 | 删除未完成时禁止替换上传 |
 | cleanup/cleaning | 可见文件保留 chunks，否则清理孤儿 chunks | 先查 files |
+| resumable/initializing | 保留可见文件的分片，或清理不可见的残留分片，再释放初始化占位 | 文件不可见时，恢复完成后可重试 `beginResumableUpload` |
 | rename/renaming | 先核验可见文件及完整 chunks，缺失或损坏时保留锁；运维核查文件名 | 工具不猜测目标或重新执行 rename |
 | pending + activeAppend | 根据 chunks 的连续前缀恢复 nextChunk，清除中断令牌，重新设置七天闲置过期 | 保留已写 chunks，后续需验证重传 |
 | finalizing，文件不可见 | 退回 pending，正常 finalize 重新校验 | 不删已有 chunks |
@@ -61,4 +72,4 @@ Direct owner 会保存摘要来源：`upload()` 的 `options.sha256` 在写入
 
 ## 失败和验收边界
 
-恢复脚本位于源码 `scripts/maintenance/gridfs-recover.cjs`，不是对外公开 API。没有绝对可靠的在线“僵尸 Writer 自动判断”，因此不能靠超时解锁。必须以确定性测试覆盖异常中断后状态、错误令牌拒绝、错误时间戳拒绝、可见文件 chunks 绝不误删、finalize 重验 SHA、abort/delete 幂等续接、审计结果及两版 MongoDB。验证结果未全绿前不能把 PR #12 标记 Ready 或合入 main。
+恢复脚本位于源码 `scripts/maintenance/gridfs-recover.cjs`，不是对外公开 API。没有绝对可靠的在线“僵尸 Writer 自动判断”，因此不能靠超时解锁。必须以确定性测试覆盖异常中断后状态、错误令牌拒绝、错误时间戳拒绝、可见文件 chunks 绝不误删、finalize 重验 SHA、abort/delete 幂等续接、审计结果及两版 MongoDB。验证结果未全绿前不能把 PR #12 标记 Ready 或合入 main。