Skip to content

perf(entity-manager): preload a page's single references in one query - #19

Open
Spamercz wants to merge 4 commits into
v2.0from
perf/batch-single-references
Open

Spamercz wants to merge 4 commits into
v2.0from
perf/batch-single-references

Conversation

@Spamercz

@Spamercz Spamercz commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Problem

EntityFactory resolves every single-entity reference (a property typed as an ElasticEntityInterface whose document stores only the id, or an STIElasticEntity) with EntityManager::find(). That is one ElasticSearch search per distinct referenced document, run one after another.

On colectio.com, the seller order list hydrates 495 CollectionItemElastic documents, each referencing a CollectibleElastic. That came to 462 serial searches. Profiled in production, this one findBy() took 8.9–9.6 s, and the whole request took 16–49 s.

Change

  • EntityFactory::collectReferences() walks a hit the way resolveProperties() does and returns the referenced ids per class. Lazy collections (Collection, ElasticCollection) are left out; they resolve their references when first read, and preloadRelations() already covers ElasticCollection.
  • EntityManager::createCollection() collects the references of all hits before hydrating. It fetches each class with one terms _id query, in chunks of 1000. Hydration then finds them in the identity map. Because the preload goes through findBy(), preloaded pages preload their own references too: a chain of references costs one query per level.
  • The preload skips ids that are already in the identity map, being created, or being preloaded further up (IdentityMap::$preloadingList). This ends cycles such as Title → Image → Title.
  • A failed preload is not fatal. Anything it did not load is still fetched with find() during hydration, exactly as before. A missing referenced document still drops the hit that references it.

Measured

On a local colectio.com snapshot: findBy of 500 collection items referencing 363 collectibles, comparing v2.0 with this branch, two runs each.

collectible searches findBy + hydrate
v2.0 363 0.61–0.76 s
this branch 1 0.12–0.14 s

The hydrated result is identical (same hash of itemId, collectibleId and name).

Tests

New EntityManager/PreloadReferencesTest.phpt counts real searches on the referenced index through _stats:

  • A page of 5 titles loads its 5 images in 1 search. On v2.0 this fails with 5 should be 1.
  • A reference shared by several hits resolves to one instance.
  • A Title ↔ Image cycle hydrates both sides.
  • A reference already in the identity map is not queried again.
  • A deleted referenced document still drops the referencing hit.

Full suite run locally against ES 9.2.2: 684 tests. The 32 failures are exactly the same set as on v2.0 (ClearTest, the MatchAll class missing from the installed elastic-query, InsertMultiple, Scroll, …). None are new. PHPStan level 6 is clean, and phpcs is clean on the changed files.

🤖 Generated with Claude Code

Spamercz and others added 4 commits November 4, 2026 14:59
A page of hits resolves every single-entity reference with its own find(),
one ElasticSearch search per distinct referenced document. Hydrating 495
collection items on colectio.com ran 462 such searches, about nine seconds
in production.

collectReferences() walks a hit the way resolveProperties() does and
returns the referenced ids per class, so a caller can fetch them all in
one query before hydrating. Lazy collections are left out; they resolve
their references only when read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A preloaded page preloads its own references in turn. Remembering which
ids an enclosing page is already fetching stops a nested page from
fetching them again, and ends reference cycles (Title -> Image -> Title).
clear() resets it with the rest of the map.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
findBy() hydrated each hit's single-entity references with find(), one
search per distinct referenced document. Before hydrating, it now
collects the page's referenced ids and fetches each class in one terms
_id query, in chunks of 1000. Hydration then finds them in the identity
map.

On a local snapshot, 500 collection items referencing 363 collectibles
went from 363 searches and 0.61-0.76 s to 1 search and 0.12-0.14 s. The
hydrated result is identical.

Ids already in the identity map, being created or being preloaded further
up are skipped. A failed preload is not fatal: whatever it missed is
still fetched with find() during hydration, as before. A missing document
still drops the hit that references it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Counts real searches on the referenced index through _stats. A page of
five titles loads its five images in one search (five before the change).
Also covers a shared reference resolving to one instance, a Title <-> Image
cycle, references already in the identity map, and a deleted referenced
document still dropping its referencing hit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant