Skip to content

docs: add FILE/HDFS alternative to Kafka skipped_line bullet - #567

Open
n24q02m wants to merge 2 commits into
apache:masterfrom
n24q02m:docs/hg-3-1.8-audit
Open

n24q02m wants to merge 2 commits into
apache:masterfrom
n24q02m:docs/hg-3-1.8-audit

Conversation

@n24q02m

@n24q02m n24q02m commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Purpose of the PR

Main Changes

  • Update the skipped_line row in the English and Chinese Loader guides to note that when reading from KAFKA, line filtering should be done before publishing to the topic or by loading from a FILE/HDFS source.
  • Remove the duplicate standalone note below the Kafka input options table.

Context

KafkaSource inherits @JsonProperty("skipped_line"), while KafkaReader does not evaluate skipped_line. Folding the note directly into the existing skipped_line bullet keeps the parameter table concise across both languages.

Verification

  • Verified internal links and Markdown table formatting in content/en/docs/quickstart/toolchain/hugegraph-loader.md and content/cn/docs/quickstart/toolchain/hugegraph-loader.md.

- Clarify that Kafka input accepts skipped_line but does not apply it.
- Recommend preprocessing the topic or using FILE/HDFS for line filtering.
- Update both English and Chinese guides.
@n24q02m
n24q02m force-pushed the docs/hg-3-1.8-audit branch from 94c6d6f to 0536c41 Compare November 16, 2026 13:33

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: The note matches KafkaReader, which hands each message to the parser and never reads skippedLine, but both pages already state this in the Kafka skipped_line bullet, so the note duplicates it above the page intro.

search_boost: 1.6
description: "Bulk import graph data into HugeGraph with Loader from files, HDFS, relational databases, Kafka, and other HugeGraph graphs."
---
> **Kafka and `skipped_line`:** HugeGraph Loader accepts `skipped_line` for

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: the skipped_line bullet under 3.3.2.4 Kafka input source already documents this limitation on both pages, and this note renders above the page intro instead of below the Kafka options. Remove the note from the English and Chinese pages, or move the FILE/HDFS suggestion into the existing skipped_line bullet.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Score: 8.0/10 · Confidence: medium

Blocking: no

Summary

The existing inline concern remains: the note duplicates the Kafka option description and sits above the introduction. Consolidating the FILE/HDFS suggestion in that option would keep the guidance together.

Evidence

KafkaSource declares skipped_line and KafkaReader passes each message to the parser without applying it; FileLineFetcher applies the file filter. Both language pages already document the Kafka limitation in their Kafka options.

Validation notes

Source review only; no live Kafka/HDFS test or Hugo build was run.

@n24q02m n24q02m mentioned this pull request Nov 21, 2026
46 of 62 tasks
@n24q02m n24q02m changed the title docs: document Kafka skipped_line limitation docs: add FILE/HDFS alternative to Kafka skipped_line bullet Nov 21, 2026
@n24q02m

n24q02m commented Nov 21, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, agreed. #501 already covers this in the Kafka skipped_line bullet on both pages, and the note rendered above the page intro, not under the Kafka options as I wrote in the description. I removed the note from both pages and moved the FILE/HDFS suggestion into the existing bullet, so the PR is now a one-line change per language. The title and description are updated to match.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Score: 9/10. Summary: The FILE/HDFS alternative in both Kafka skipped_line bullets matches hugegraph-loader master, where LocalFileReader and HDFSFileReader read CSV, TEXT and JSON through FileLineFetcher, which applies skipped_line, and the earlier duplicate-note comment is resolved at this head.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants