Repository navigation
fix(subscription): treat absent schema-evolved TAG columns as NULL and flatten logical chains in filter parser - #19346
Conversation
…d flatten logical chains in filter parser
| final Expression expression) { | ||
| if (expression instanceof LogicalExpression | ||
| && ((LogicalExpression) expression).getOperator() == operator) { | ||
| terms.addAll(((LogicalExpression) expression).getTerms()); |
There was a problem hiding this comment.
[P2] Avoid quadratic copying when flattening a chain. The grammar is left-recursive, so for a OR b OR ..., each parent reaches this branch with all prior terms in the left child. addAll recopies those terms for every predicate, and LogicalExpression copies the new list again. Parsing a length-N flat column filter therefore allocates O(N^2) references. The column-filter path has no size/depth cap, so a sufficiently large valid topic filter can cause a large CPU/heap spike while refreshing its matcher. Please collect same-operator terms in one pass or impose a limit.
There was a problem hiding this comment.
Thanks for catching this! Fixed in 7a9280d62e:
- Replaced the bottom-up recursive
appendLogicalTerms/addAllflattening inColumnFilterParser.AstBuilder#visitLogicalBinarywith an iterative single-pass walk using an explicitArrayDeque<BooleanExpressionContext>stack (also peeling parenthesized same-operator sub-expressions viaunwrapParenthesized). - Same-operator chains are now flattened into a single
LogicalExpressionin strictO(N)time andO(N)allocations without intermediateLogicalExpressioncopies or Java recursion depth growth. - Strengthened
ColumnFilterParserTest#testFlattenedOrChainParsesIntoSingleLogicalExpression(500-termORchain checking exact left-to-right order) and addedtestParenthesizedSameOperatorFlattensWhilePreservingMixedPrecedence.
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
2 open issues
What changed in this PR
Fixes correctness and scalability issues in subscription filter evaluation by handling schema-evolved missing TAG columns as SQL NULL (when schema binding is available) and by flattening logical AND/OR chains to avoid excessive AST depth.
Changes:
- Treat missing, schema-evolved TAG columns in older
Tablets as SQLNULLwhen table schema binding is present. - Flatten same-operator logical chains in
ColumnFilterParserto preventMAX_AST_DEPTHviolations for wideAND/ORexpressions. - Add regression tests for schema evolution behavior and flattened logical-chain parsing/evaluation.
| File | Description |
|---|---|
| iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/tagfilter/TabletTagFilter.java | Makes schema binding return a boolean and uses it to treat absent TAG columns as NULL instead of throwing. |
| iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/columnfilter/ColumnFilterParser.java | Flattens AND/OR logical chains into n-ary LogicalExpression terms to reduce AST depth. |
| iotdb-core/datanode/src/test/java/org/apache/iotdb/db/subscription/tagfilter/TagFilterParserTest.java | Adds tests ensuring flat logical chains don’t exceed max AST depth and precedence is preserved. |
| iotdb-core/datanode/src/test/java/org/apache/iotdb/db/subscription/tagfilter/TabletTagFilterTest.java | Adds tests for schema-evolved missing TAG columns treated as NULL when schema-bound. |
| iotdb-core/datanode/src/test/java/org/apache/iotdb/db/subscription/columnfilter/ColumnFilterParserTest.java | Adds test validating flattened OR chains parse/evaluate as a single LogicalExpression. |
Review effort: Lite
| final LogicalExpression.Operator operator = | ||
| Objects.nonNull(context.AND()) | ||
| ? LogicalExpression.Operator.AND | ||
| : LogicalExpression.Operator.OR; | ||
| final List<Expression> terms = new ArrayList<>(); | ||
| appendLogicalTerms(terms, operator, visit(context.booleanExpression(0))); | ||
| appendLogicalTerms(terms, operator, visit(context.booleanExpression(1))); | ||
| return new LogicalExpression(operator, terms); |
There was a problem hiding this comment.
Thanks for the note. For the tag-filter path, TagFilterValidator already enforces MAX_AST_NODES = 256 across all flattened terms (visitLogicalExpression invokes process() -> accountNode() on every element in node.getTerms()), so wide AND/OR chains remain strictly bounded by MAX_AST_NODES. ColumnFilterValidator did not enforce MAX_AST_DEPTH/MAX_AST_NODES previously, and the quadratic flattening cost in ColumnFilterParser has now been eliminated in 7a9280d62e.
| final List<Expression> terms = new ArrayList<>(); | ||
| appendLogicalTerms(terms, operator, visit(context.booleanExpression(0))); | ||
| appendLogicalTerms(terms, operator, visit(context.booleanExpression(1))); | ||
| return new LogicalExpression(operator, terms); | ||
| } | ||
|
|
||
| private static void appendLogicalTerms( | ||
| final List<Expression> terms, | ||
| final LogicalExpression.Operator operator, | ||
| final Expression expression) { | ||
| if (expression instanceof LogicalExpression | ||
| && ((LogicalExpression) expression).getOperator() == operator) { | ||
| terms.addAll(((LogicalExpression) expression).getTerms()); | ||
| } else { | ||
| terms.add(expression); | ||
| } |
There was a problem hiding this comment.
Fixed in 7a9280d62e: ColumnFilterParser.AstBuilder#visitLogicalBinary now walks same-operator LogicalBinaryContext subtrees iteratively with an explicit ArrayDeque stack and collects all terms into a single LogicalExpression in O(N) time without quadratic addAll copying.


Summary
Fixes two correctness issues in table-model subscription tag/column filtering:
TAGcolumns in olderTablets as SQLNULL(TabletTagFilter):ALTER TABLE ... ADD COLUMN new_tag STRING TAG, historical TsFiles and pre-alter WALTablets do not includenew_tagin their column schema.TabletTagFilter#resolveReferencedTagColumnsunconditionally threwTagFilterEvaluationException.schemaBinding("Referenced TAG column is missing")even whenvalidateBindinghad already proven againstTsTablethatnew_tagis a validTAGcolumn.schemaBound == true, covering both literal and bound wildcard topics), absent referencedTAGcolumns now evaluate as SQLNULL(handled bygetTagValuewhencolumnIndex == null, preserving SQL three-valued logic). Unbound matchers without aTsTablebinding continue to fail closed.AND/ORchains inColumnFilterParser:ColumnFilterParser.AstBuilder#visitLogicalBinarypreviously constructed left-skewed binary trees viaLogicalExpression.and(left, right)/.or(left, right), causing flat chains of >31 predicates (e.g., 40ORterms) to exceedTagFilterValidator.MAX_AST_DEPTH = 32.LogicalExpressionchildren are now flattened into a single n-aryLogicalExpression(matching howTagFilterValidator,TagFilterEvaluator,ColumnFilterValidator, andColumnFilterEvaluatoralready iteratenode.getTerms()), while preserving operator precedence ((a OR b) AND c) and still rejecting genuinely deep nesting (>32 levels ofNOT (...)).Concrete Before-and-After Examples
ALTER TABLE weather ADD COLUMN new_tag STRING TAG) with a pre-alterTablet(region, device, temperature):new_tag IS NULLornew_tag = "v1" OR region = "north"threwTagFilterEvaluationException: Referenced TAG column is missing.new_tagevaluates as SQLNULL—new_tag IS NULLretains all rows,new_tag = "v1"andNOT (new_tag = "v1")drop all rows under 3VL, andnew_tag = "v1" OR region = "north"retains"north"rows; unbound matchers still throw on missing columns.region = "r0" OR region = "r1" OR ... OR region = "r39"):Tag-filter AST depth exceeds maximum of 32.LogicalExpression(OR, 40 terms)of depth 2 and evaluated accurately; 33-level nestedNOT (NOT (...))is still rejected.Verification
mvn test -pl iotdb-core/datanode -am -Dsurefire.failIfNoSpecifiedTests=false -Dtest=TagFilterParserTest,TabletTagFilterTest,ColumnFilterParserTest,SubscriptionFilterSnapshotTest: 28/28 passed (7 + 8 + 9 + 4)mvn test-compile -pl iotdb-core/datanode -am -DskipTests: exit 0mvn test-compile -pl iotdb-core/datanode -am -P with-zh-locale -DskipTests: exit 0mvn spotless:check -pl iotdb-core/datanode: exit 0