Skip to content

docs(specs): NULL semantics of restriction and its complement; current system requirements - #307

Open
dimitri-yatsenko wants to merge 3 commits into
mainfrom
docs/restriction-complement
Open

dimitri-yatsenko wants to merge 3 commits into
mainfrom
docs/restriction-complement

Conversation

@dimitri-yatsenko

Copy link
Copy Markdown
Member

Specifies the behavior that datajoint/datajoint-python#1663 ships in 2.3.7, and updates the system requirements.

Spec (reference/specs/query-algebra.md)

  • New §2.10, NULL semantics. Restriction by a query expression matches on all shared attributes, using the same equality as join. NULL never matches. So A & B and its complement A - B partition A. String and dict conditions follow SQL's three-valued logic and carry no such guarantee. A "Changed in 2.3.7" note describes the old NOT IN behavior.
  • Other changes in this file:
    • The properties table (§2.11) gets a Partition row.
    • §2.5 now says matching uses all shared attributes, not only primary keys.
    • §2.7 calls A - B the complement of A & B.
  • New §13.4.1, compiled SQL.
    • The SQL for A & B (IN) and A - B (correlated NOT EXISTS), checked against the code on MySQL and PostgreSQL.
    • De Morgan pushdown for negated lists and the wrapping of aggregation and union operands.
    • The negated multi-attribute fix.
    • The rule for operands with no shared attributes.
  • Union SQL in §7.5 and §13.7, plus new rows in the §11 and §13.9 tables.
  • Short notes in reference/operators.md, explanation/query-algebra.md, how-to/query-data.md and reference/specs/semantic-matching.md. The last is a one-line change that stays clear of docs(semantic-matching): unrelated namesakes, not missing lineage #305's edits.

System requirements

  • how-to/installation.md:
    • Python 3.10 to 3.15, matching requires-python >=3.10,<3.16.
    • MySQL 8.0.18+ (the 8.0 and 8.4 LTS lines) or PostgreSQL 15+.
    • The database drivers, including datajoint[postgres] for PostgreSQL.
    • Graphviz, as an optional system package for diagrams.
    • Platform support lists the versions CI actually tests: Python 3.10, 3.14 and 3.15, MySQL 8.0 and 8.4, PostgreSQL 15.
  • reference/specs/database-backends.md: the MySQL minimum rises from 8.0.13 to 8.0.18. It explains why: on 8.0.17, DELETE, HAVING and OR with the NOT EXISTS form took more than 60 s, against 0.1 to 0.7 s on 8.0.18. Below 8.0.18 the client warns on connect and keeps working.
  • README.md and how-to/migrate-to-v20.md: the same requirements.

Notes

  • Depends on #1657 to be fully accurate. The text on wrapping Top-limited operands describes the code once datajoint-python#1657 is merged. Until then, A & (B & dj.Top(...)) fails on MySQL with error 1235.
  • Release notes are not included. about/whats-new-*.md is untouched because docs(whats-new): changes in 2.3.5 #304 edits it.
  • Build. mkdocs build --strict ran clean before the final spec fix-up commit, which changes only text inside query-algebra.md.

Specify restriction by a query expression for v2.3.7:

- query-algebra: matching is on all shared attributes, not only the
  primary key; new section 2.10 NULL Semantics (NULL never matches,
  A & B and A - B partition A, consistent with join; string and dict
  conditions keep SQL three-valued logic); a Partition row in the
  algebraic properties; new section 13.4.1 with the compiled SQL
  (IN for A & B, correlated NOT EXISTS for A - B, negation pushed
  through lists, wrapping of aggregation, union and Top operands);
  Union SQL in 7.5 and 13.7; subquery tables in 11 and 13.9.
  Sections 2.10 and 2.11 are renumbered to 2.11 and 2.12.
- MySQL minimum 8.0.18 in installation, database-backends (with the
  reason, and the warning on connect below it), migrate-to-v20 and
  README.
- Short notes in operators, explanation/query-algebra, how-to/query-data
  and the semantic-matching spec.
Python 3.10 - 3.15 (requires-python >=3.10,<3.16), MySQL 8.0.18+ or PostgreSQL 15+, the database
drivers, and Graphviz as an optional system package for diagrams. Platform support lists the Python
versions CI runs (3.10, 3.14, 3.15) and the database versions it tests against.
- 13.4.1: B's own SQL lists columns without a space after the comma, the
  shared attributes follow B's heading order, and the operand wrapper uses
  a lowercase "as", as the code emits them.
- 2.10: under the old NOT IN, a row of A with NULL in a shared attribute
  could be in neither result, not always; state when.
- 13.4.1: state exactly when NOT IN is NULL rather than true.
- 13.4.1: note that a negated multi-attribute dict was wrong before 2.3.7,
  and that an operand with no shared attributes is evaluated by a separate
  query.

The wrapping of a dj.Top-limited operand (sections 11, 13.4.1, 13.9)
depends on datajoint-python#1657; without it, A & B with a limited B
fails on MySQL with error 1235.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant