Repository navigation
Conversation
PostgreSQL clients (psql, pgJDBC/DBeaver, psycopg, Npgsql, Power BI, Excel) can query the table model, read-only. New built-in external service external-service-impl/postgresql, off by default (enable_postgresql_service): - PostgreSQL protocol 3.0 on Netty: simple and extended query protocol, binary results, portals and SQL cursors, cancel, statement_timeout, streaming with a write timeout, connection limit. - TLS 1.2/1.3 only off loopback (key_store_path); password login through SessionManager, offered only inside TLS or on loopback. - Translation of PostgreSQL SQL into table-model SQL (JSqlParser); constructs without an exact equivalent are refused with SQLSTATE 0A000. - pg_catalog and information_schema emulation as information_schema.pg_* and pg_is_* tables, filtered by the user's privileges. Unit and golden tests over SQL recorded from psql, pgJDBC and psycopg, and integration tests (TableSimpleIT).
Catalog queries of PostgreSQL clients (Npgsql's type loading joins seven pg_* tables) were refused at small memory budgets with "There is not enough memory to execute current fragment instance": every sort of a join is estimated at a full sort buffer, however small the inputs are. Queries that read only information_schema tables now get the existing highest priority (so far only information_schema.queries had it), so they use the memory fallback and are scheduled first. QueryExecution also sets the flag on the query context after distribution planning, because table-model analysis decides it after the context was created.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #19350 +/- ##
============================================
+ Coverage 49.44% 49.68% +0.24%
- Complexity 718 1454 +736
============================================
Files 5656 5703 +47
Lines 418222 423371 +5149
Branches 55296 56203 +907
============================================
+ Hits 206769 210347 +3578
- Misses 211453 213024 +1571 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
- IoTDBServicesIT lists the POSTGRESQL built-in service. - IoTDBDatabaseIT expects the information_schema tables that an InformationSchemaExtension (the PostgreSQL catalog) registers. - The PostgreSQL ITs wait until the endpoint listens: the service starts after the DataNode reports itself ready (refused connections on Windows). - Log checks wait for the expected line instead of reading the log once. - The write-timeout IT starts its countdown when the first result bytes arrive, so slow machines that sort the result for a while do not finish the query before the stall.
Contributor
Author
Power BI Desktop 2.149 recorded against PostgreSQL 14: connect (Npgsql type loading), navigator, Import and three DirectQuery visuals, now in the client corpus with the expected translations. Power BI reads information_schema.character_sets on every connection; the catalog now answers it as PostgreSQL 14 does for a UTF8 database with collation C (UTF8 / UCS, default collation pg_catalog.ucs_basic), and pg_collation lists ucs_basic. pg_total_relation_size stays refused. extract.py accepts application names with spaces (Power BI sends "Mashup Engine") and takes the capture date from the server log.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.






Description
This draft adds a PostgreSQL wire protocol service to IoTDB. PostgreSQL clients can then
query the table model, read-only: psql, JDBC tools (pgJDBC, DBeaver), Python (psycopg),
.NET (Npgsql), and BI tools that connect through PostgreSQL connectors, such as Microsoft Power BI
(Import and DirectQuery) and Excel.
It was developed in the ToddySoft DB fork of IoTDB and ported to
masterfor this PR. Thecommunity has asked about a PR, so I'm opening it as a draft, for two reasons:
(see "Differences from the fork" below), and
Example
Recorded with psql 17 against a standalone server from this branch, bound to
0.0.0.0(soTLS is required) with
key_store_pathset (host name and port replaced by placeholders):Every IoTDB database appears as a PostgreSQL schema (
plant.sensorsis tablesensorsofdatabase
plant). Queries are translated into table-model SQL and executed by the normal engine,so filters and aggregations stay in the engine. A construct without an exact translation is
refused with SQLSTATE
0A000and named; it is never answered approximately.Tree-model data
The service reads the table model only. Tree-model data becomes visible through a table view
(the path levels below the prefix become tags, the measurements become fields), for example for
devices
root.factory.<line>.<machine>:The service does not create views itself; one view is needed per device pattern (also in the
module's README).
Design
Module and start. The new module
external-service-impl/postgresqlis a built-in externalservice like REST and MQTT: a
POSTGRESQLentry inBuiltinExternalServices, enabled byenable_postgresql_service(defaultfalse). The jar-with-dependencies goes tolib/of theallandedgedistributions and of the IT nodes. It has to be inlib/and not inext/external_service, because the catalog tables (below) are registered through the class pathwhen the DataNode starts. The only third-party dependency is JSqlParser 5.4 (dual LGPL-2.1 /
Apache-2.0, used under Apache-2.0, added to
LICENSE-binary). Netty comes from the DataNode.Core changes (small):
BuiltinExternalServices.POSTGRESQL, plusIoTDBConfig/IoTDBDescriptor:enable_postgresql_service.ExternalServiceClientSession: anIClientSessionfor clients that come in through anexternal service speaking another protocol.
IClientSession's connection methods arepackage-private, so a plugin cannot subclass it itself.
iotdb-system.properties.template: the "PostgreSQL Service Configuration" section.DataNodeConfigsetters for the service's keys.information_schematables run with the existing highest priority(
Analysis.needSetHighestPriority, so far only forinformation_schema.queries). Catalogqueries of clients join up to seven
pg_*tables, and every sort of a join is estimated at afull sort buffer, so at small memory budgets they were refused with "There is not enough memory
to execute current fragment instance", although their inputs are tiny. With the highest
priority they use the existing memory fallback and are scheduled first. CI showed this on the
Windows runner.
QueryExecutionnow also sets the flag on the query context afterdistribution planning, because table-model analysis decides it after the context was created.
Protocol (
server,protocol,session,execution): PostgreSQL protocol 3.0 on Netty:results;
DECLARE/FETCH/MOVE/CLOSE, as psycopg's server-side cursors use them);57014) andstatement_timeout;SET/RESET/SHOW,BEGIN/COMMIT/ROLLBACK,DISCARD,DEALLOCATE;for
postgresql_service_write_timeout_in_msloses its query and its connection;53300), an idle timeout and a startup timeout;PgErrors).Writes are refused with
25006. The reportedserver_versionis14.0, so clients take theirstandard code paths.
Security:
AEAD suites; a plain connection gets
FATAL 28000 encryption required.key_store_path/key_store_pwd. Without a key store the servicerefuses to start off loopback and logs why.
TLS or on loopback, and the password is checked by
SessionManager.login, so lockout and theother login rules apply unchanged. See "Differences" for why it isn't SCRAM.
Translation (
sql): JSqlParser parses the statement. A check (ConstructCheck) allows onlyconstructs with an exact table-model equivalent, and a writer (
PgToTableSql) prints table-modelSQL. Among the rules:
count,?column?, ...);::casts and typed literals;ILIKE;LIKE/IN;NULLIF, and||/concatwith PostgreSQL's NULL semantics;date_trunc/extract/date_partandtimestamp ± intervalin the session time zone(
DateTimeRules; day truncation is exact across DST changes, and units that would be wrong inthe session zone are refused);
format_type,pg_get_expr,obj_description,current_schemas,regclass/regtypecasts, ...).Catalog (
catalog): thepg_catalogtables clients read (pg_namespace,pg_class,pg_attribute,pg_type,pg_proc,pg_description,pg_database,pg_settings, ...) andPostgreSQL's
information_schemaviews. They are implemented asinformation_schema.pg_*/pg_is_*tables through the existingInformationSchemaExtensionandAdditionalInformationSchemaProviderSPIs, so catalog queries, joins included, run in the normalengine. They show only the databases and tables the user may read. OIDs are stable (CRC32 of the
name, at least 16384), and column categories (TAG/ATTRIBUTE/FIELD) appear as column comments.
Tests
value encoding, catalog rows, translation rules, and
TranslatorGoldenTest. The golden testtranslates every recorded client statement and compares the result with
corpus/expected/<client>/<n>.sql(or the expected0A000).org.apache.iotdb.postgresql.it,TableSimpleIT):IoTDBPostgreSqlEngineITruns every translated corpus statement on the engine and checkspsql's catalog results, Npgsql's type-loading queries, date rules around both 2026 DST changes
in Europe/Berlin, pgJDBC
DatabaseMetaData, SQL cursors, cancel,statement_timeoutandprepared statements with every parameter type;
IoTDBPostgreSqlClientsIT: pgJDBC simple/extended mode, metadata;IoTDBPostgreSqlImportIT: 1,000,000 rows of every type through pgJDBC, compared with thesession API, plus the write timeout;
IoTDBPostgreSqlSecurityIT: login, refusals, connection limit, read-only;IoTDBPostgreSqlTlsIT: TLS-only off loopback, TLS 1.2/1.3, TLS 1.1 refused;IoTDBPostgreSqlNoKeyStoreITandIoTDBPostgreSqlPortInUseIT: the service fails cleanly andthe DataNode keeps running.
Differences from the fork (each available as a separate PR if wanted)
The fork has a few things this PR deliberately leaves out, to keep the core changes small. If the
project is interested in any of them, I'm happy to send it as a separate PR:
SCRAM-SHA-256 login. In the fork, built-in users log in with SCRAM-SHA-256, so the
password never crosses the network, even inside TLS. That requires storing passwords as salted
SCRAM verifiers (
$scram-sha-256$<iterations>:<salt>$<StoredKey>:<ServerKey>, PBKDF2-SHA256)instead of the current unsalted SHA-256 hash. Existing hashes are converted when the ConfigNode
starts or at the next login. That is an on-disk format change with no way back, so it needs its
own discussion. Without it, this PR uses the password inside TLS.
Plugin discovery instead of an enum entry. In the fork, optional services are found
through
ServiceLoader(META-INF/services/...IExternalService), and each plugin reads its ownkeys, so the core has no
enable_postgresql_serviceand noPOSTGRESQLentry. The fork alsoguards the start of discovered services against undeclared checked exceptions: Netty rethrows
BindExceptionundeclared, and the plugin catches it itself here too.Service status, settings and endpoints SPI. In the fork, external services report their
status and endpoints to an embedding host (the fork has an embedded in-JVM mode). That isn't
needed in IoTDB.
Host user backends. The fork lets an embedding host plug in its own user management.
PostgreSQL logins for such users use the password inside TLS as here, unless the backend
supplies SCRAM verifiers.
Configuration keys. The fork uses
postgresql_service.*keys with ISO-8601 durations,because its plugins declare their settings through the SPI in item 3. This PR uses IoTDB's style
(
enable_postgresql_service,postgresql_service_*_in_ms).Uniform login refusals. In the fork, a wrong password and an unknown user get the same
message (and a minimum response time), so logins don't reveal which users exist. This PR passes
IoTDB's own login messages through ("The user ... does not exist"), with SQLSTATE
28P01inboth cases.
Open question: catalog tables while the service is off
The catalog tables (
pg_class,pg_namespace, ... and thepg_is_*views, 29 tables) areregistered through
InformationSchemaExtensionwhen the jar is loaded. Because the jar ships inlib/, they appear in everyinformation_schemalisting (SHOW TABLES FROM information_schema,information_schema.tables/columns), even withenable_postgresql_service=false. That iswhy this PR updates
IoTDBDatabaseITandIoTDBServicesIT.Hiding them while the service is off would mean either making the registration depend on the
DataNode setting (
InformationSchemalives innode-commonsand is also loaded by theConfigNode, which doesn't know that setting), or filtering them out of the DataNode's
listings. Do you want them hidden while the service is disabled, or is listing them always
acceptable?
Help wanted: reviewing the golden files
external-service-impl/postgresql/src/test/resources/corpus/contains SQL recorded from realclients against PostgreSQL 14:
\l,\dn,\dt,\d table,\d+);DatabaseMetaDatacalls as DBeaver makes them);(
information_schema.tables/columns, keys), an Import of a table and three DirectQueryvisuals. All of them translate and run on the engine except
pg_total_relation_size()(tablesize), which is refused; Power BI carries on without it.
corpus/expected/<client>/<n>.sqlholds the expected translation of each statement, or theexpected
0A000refusal. The translator itself generated these files(
-Dgolden.update=true), so for now they only prove that its output doesn't change, not that itis right.
IoTDBPostgreSqlEngineITalready proves that every translated statement runs on theengine, and checks the rows for a few of them (psql's
\dnand\d, pgJDBC metadata, the daterules).
What we'd like reviewers to check, file by file: is the translation semantically the same as
the PostgreSQL statement, and is each refusal acceptable? A typical case is
expected/psql/5.sql(psql's
\d table):format_type(oid, NULL)becomes aCASEover the type OIDs;pg_get_expr(...)becomesCAST(NULL AS STRING)(IoTDB has no column defaults);relhasoidsbecomesFALSE.Comments on individual files in this PR are very welcome. The Power BI files matter most,
especially the DirectQuery ones (
expected/powerbi/50.sqlto54.sql): that is the SQL Power BIsends for every visual. A recording of more DirectQuery visuals (date hierarchy, slicer, relative
date and Top N filters) will follow; if you use Power BI, recordings of your own reports are very
welcome too (
corpus/README.mddescribes how they're made).This PR has:
(
external-service-impl/postgresql/README.md, configuration template).LICENSE-binary).obvious for an unfamiliar reader.