Skip to content

IGNITE-29211 Hot-reload TLS certificates automatically before expiry - #13786

Open
anton-vinogradov wants to merge 388 commits into
apache:masterfrom
anton-vinogradov:ignite-29211
Open

anton-vinogradov wants to merge 388 commits into
apache:masterfrom
anton-vinogradov:ignite-29211

Conversation

@anton-vinogradov

@anton-vinogradov anton-vinogradov commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

IGNITE-29211 adds optional automatic TLS certificate hot reload. Nodes attempt to read replacements from the configured source before expiry. Ignite does not issue certificates.

Behavior

  • Hot reload is disabled by default. SslContextFactory calculates hot reload dates from the key store actually loaded, including custom sources. Attempts start with 15% of the selected chain's lifetime remaining.
  • Manual and automatic replacements use the same init() path. The factory does not reject its own certificates because they have expired or are not yet valid, including at startup. Replacements may have equal, earlier or later expiry, including unchanged certificates and trust-store-only changes. Peers apply their configured trust checks during TLS connections.
  • Loading or context creation failures preserve the current context and hot reload date. A successful load installs the context for new connections; established connections remain open. The manual command reports reloaded, including when the material has not changed.
  • Each node runs automatic attempts and manual reloads of enabled factories on one executor. Before the hot reload registry starts, manual reloads of enabled factories fail without loading material. If an automatic load leaves hot reload due, the attempt counts as a failure and retries with exponential backoff and jitter. The installed context remains in use. A successful manual reload clears that factory's failures and schedules its next attempt, immediately if still due.
  • ssl.ConsecutiveHotReloadFailures sums consecutive automatic failures on the node. ssl.ManualReloadAttempts and ssl.AutomaticHotReloadAttempts count per-factory calls, including errors and automatic skips. They exclude startup, direct application calls and queued calls cancelled before execution.
  • Shutdown cancels queued operations, interrupts the running executor task and waits for it to finish. A task already waiting for a shared factory may still load and install a replacement after shutdown starts.

The existing final void init() contract is preserved. The factory publishes the hot reload date only after successful context creation. During loading, readers get the previous date without waiting. hotReload() checks whether loading is due; the registry evaluates the resulting hot reload date. Startup initializes missing hot reload dates. Nodes sharing a factory keep separate schedules and may load it sequentially. A startup load already waiting for the factory may proceed after shutdown starts.

Hot reload covers TCP transports. HTTPS remains in IGNITE-29212. Factories extending only AbstractSslContextFactory retain their own manual loading behavior.

Tests cover certificate dates, loading expired and not yet valid certificates, peer trust checks, equal or earlier expiry, trust-only changes, failure recovery, scheduling, shutdown and metrics. They also cover the reload command and replacement on a live TLS connection.

Depends on IGNITE-29210, merged into Apache master.

# Conflicts:
#	modules/core/src/test/java/org/apache/ignite/testframework/TestCertificateAuthority.java
# Conflicts:
#	docs/_docs/security/ssl-tls.adoc
#	modules/core/src/main/java/org/apache/ignite/internal/ssl/SslContextProvider.java
# Conflicts:
#	docs/_docs/security/ssl-tls.adoc
# Conflicts:
#	docs/_docs/security/ssl-tls.adoc
#	docs/_docs/tools/control-script.adoc
#	modules/core/src/main/java/org/apache/ignite/internal/ssl/SslContextProvider.java
# Conflicts:
#	modules/core/src/test/java/org/apache/ignite/ssl/SslTestUtils.java
anton-vinogradov and others added 5 commits November 13, 2026 16:40
The reload command test no longer changes the TLS protocol: every
successful load reports "reloaded", so the test reloads the same
material. The registry test replaces the renewal executor for any thread
name. The renewal node test connects with the default thin client
settings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Test names state the scenario and the expected result. The registry test
defines its terms: a manual reload is SslFactoryRegistry#reload, and an
attempt is an automatic renewal attempt that the registry schedules.
The manual reload before start also checks the reason for the rejection.
The metric docs state that such rejected reloads are not counted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each registry test enables renewal and sets the renewal time itself:
due() for a time that has come, later() for one two hours ahead.
setUp only wires the executor, metrics and log. Each factory test turns
renewal on or off itself and checks the renewal time its scenario relies
on. The attempt that leaves renewal due checks the logged reason, and
the failed startup load checks that start() rethrows the load error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A node starts with certificates whose renewal is due and its key store
keeps them. The automatic attempt loads the same certificates, so the
node counts a failure and logs the transport and the reason. This
replaces the registry test that stubbed the factory result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anton-vinogradov anton-vinogradov changed the title IGNITE-29211 Renew TLS certificates automatically before expiry IGNITE-29211 Hot-reload TLS certificates automatically before expiry Nov 13, 2026
anton-vinogradov and others added 21 commits November 13, 2026 18:24
The registry increments ConsecutiveHotReloadFailures before it logs the
warning, so the test could check the log before the line was written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Registry tests that depend on the factory outcome now use a real
SslContextFactory. The test writes certificates to a key store file
or deletes the file, as an operator would. Only time, concurrency and
node infrastructure stay replaced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The window test has a key store where the chain that expires first is
  not the chain whose window started first.
- The retry test checks the random addition to the retry delay.
- A factory without a loaded hot reload date does not load on hotReload().
- Registry start no longer clears the caller context: schedule() already
  clears it for every attempt, and the start task only schedules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…efore

- The reload command test now enables hot reload, so the command reload
  goes through the hot reload thread.
- The hot reload docs ask to put only certificates whose validity period
  has started: Ignite installs replacements without checking their dates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The tests for a disabled factory, a reload before start and a stopping
node now take the hot reload flag and dates from a real SslContextFactory
over a key store file instead of stubbing them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- hotReload() on a factory whose flag was turned off after loading now
  has a test; no test failed when the flag check was removed.
- Manual reload keeping the caller context, the single key store read,
  the uncounted startup load and the counted failed attempt each get a
  focused test instead of a side check in an unrelated one.
- Registry tests wait up to 30 seconds for events instead of 5, so a
  slow machine cannot fail them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each test states the guarantee it checks and how it creates the
situation. Checks that the descriptions do not need are removed. The
expired first load now runs with hot reload enabled, because that is
the path this change adds. The factory and node tests wait up to
30 seconds for events, like the registry tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first sentence of each hot reload test description states the
guarantee. The next sentences explain how the test checks it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each test description names the state the test creates, the action,
and the result it checks. Setup methods and helpers that prepare the
test state say what they give the tests and how.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A node cannot be stopping when the registry starts: an outside stop
waits until the start finishes, and a failed start stops the node only
after it throws. The automatic counter test again checks that the first
attempt fails and the second one succeeds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anton-vinogradov

Copy link
Copy Markdown
Contributor Author

/runall

@ignite-pr-checker

Copy link
Copy Markdown

@anton-vinogradov Ignite PR Checker verdict · RunAll build 9499819 · tested a5cfeb9 · 149 suites ran, 0 reused

⚠️ This run doesn't cover the PR fully:

  • 2 suites have no reliable result (non-zero exit code; execution timeout · non-zero exit code)

Everything below is what it did manage to say.

⚠️ Broken suites (no reliable result):

  • [Check Code Style Ducktests]: non-zero exit code
  • Thin client: Node.js: execution timeout · non-zero exit code

🔎 No blockers found — but the run above can't prove the PR is clean. 113 failing tests are not blamed on this PR. Re-run after you handle what is listed above.

What each label means and what to do: verdict glossary.

♻️ Settled after 2 auto re-run waves: #1 — 5 suites that failed mid-run; #2 — 3 broken suites.


/** {@inheritDoc} */
@Override SSLContext createSslContext() throws SSLException {
SSLContext ctx = super.createSslContext();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's get rid of temporary instance variable:

long prevHotReloadTime = hotReloadTime;

try {
    return super.createSslContext();
}
catch (Throwable ignored) {
    hotReloadTime = prevHotReloadTime;
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants