Skip to content

docs(toolchain): fix builds and clarify import recovery - #570

Open
imbajin wants to merge 5 commits into
apache:masterfrom
hugegraph:docs/toolchain-workflows-20261115
Open

imbajin wants to merge 5 commits into
apache:masterfrom
hugegraph:docs/toolchain-workflows-20261115

Conversation

@imbajin

@imbajin imbajin commented Nov 16, 2026

Copy link
Copy Markdown
Member

Purpose of the PR

Fix Toolchain instructions that fail against current source builds and clarify what import recovery actually preserves.

  • Read the locked SDK revision from the verifier and reuse one isolated Maven build procedure across Tools, Loader, Client and Spark Connector.
  • Correct current Java/Spark/Scala requirements, nullable GraphSpace quota updates and storage measurement.
  • Distinguish FILE/HDFS positions, Kafka offsets and JDBC/GRAPH rereads; replace old help-output dumps with the installed binary's help.

Validation: executable shell/JSON checks, locked-SDK lookup, source-contract review and independent artifact checks passed. Desktop/mobile checks verified complete code copying, native tables, keyboard focus and touch behavior. Legacy anchors are preserved.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: The locked SDK lookup, the master rejection in checkout-server.sh, Spark 3.5.8/Scala 2.12.18/Guava 30.0-jre, nullable compute quotas, graph-clone and per-source Loader resume text match Toolchain and Server master. One version gap in the GraphSpace section, in both languages.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: The locked SDK lookup, candidate build flags, Spark 3.5.8/Scala 2.12.18/Guava 30.0-jre, nullable compute quotas, graph-clone, help <command> and per-source Loader resume text match Toolchain and Server master, and the earlier 8.3 version comment is fixed in prose. The Spark 3.2 test command also runs the Client API tests, which clear a hardcoded local graph, and the getStorageUsedKb() table row still lacks its version note.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Score: 9/10. Summary: The locked SDK lookup, master rejection and the Tools, Loader, Client and Spark Connector commands all build as written against Toolchain master, and the Spark 3.5.8/Scala 2.12.18, nullable compute quota, per-source resume and help <command> text match the source; the three earlier comments are fixed in both languages.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Score: 7.5/10 · Confidence: high

Blocking: yes

Summary

The quota update table promises omission even for a fetched GraphSpace whose fields are already populated. Resending a stale object can overwrite a concurrent quota change.

Evidence

GraphSpaceAPI.get deserializes the response, and GraphSpaceReq copies non-null compute quotas into the update request. The earlier Spark and version comments are addressed at this head.

Validation notes

Source inspection only; no live quota update, full SDK build, Spark deployment, or Loader recovery was run.

Pengzna
Pengzna previously approved these changes Nov 21, 2026

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Score: 9/10. Summary: The quota table now limits omission to unset fields and warns that a getGraphSpace() object resends its fetched compute limits, matching GraphSpaceReq and the Server update handler, and the 1.8.0 dependency version matches Toolchain master; the earlier fetched-object comment is fixed in both languages.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Score: 7.5/10 · Confidence: medium

Blocking: yes

Summary

The Kafka recovery table omits the no-valid-offset fallback, which defaults to latest and can skip existing records; earlier quota, Spark test safety and version issues are addressed.

Evidence

KafkaReader selects auto.offset.reset from from_beginning, whose default is false; the table states only that committed group offsets determine the start.

Validation notes

Source inspection only; Maven builds, live quota updates, Spark deployment, Loader recovery and browser checks were not rerun.

| --- | --- |
| FILE / HDFS | Loader restores the recorded per-file positions from its progress file |
| JDBC / GRAPH | Loader does not restore a source-row or graph-scan position; restarting can read previously imported elements again |
| Kafka | Consumer-group offsets stored by Kafka determine the starting point; Loader's progress file does not replace them |

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Include the missing-offset fallback in recovery guidance

If the group has no valid committed offset (for example, the first commit has not happened or the offset expired), Kafka uses auto.offset.reset instead. Loader defaults from_beginning to false and sets that policy to latest, so a restart can skip existing records. Please qualify this row and link to the from_beginning explanation; the Chinese table needs the same clarification. This is based on source inspection, without a live Kafka recovery test.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants