Skip to content

Move ESM loaders off-thread#44710

Merged
aduh95 merged 126 commits intonodejs:mainfrom
JakobJingleheimer:feat/esm_off-thread-loaders
Apr 13, 2023
Merged

Move ESM loaders off-thread#44710
aduh95 merged 126 commits intonodejs:mainfrom
JakobJingleheimer:feat/esm_off-thread-loaders

Conversation

@JakobJingleheimer
Copy link
Copy Markdown
Member

@JakobJingleheimer JakobJingleheimer commented Sep 18, 2022

Resolves #43658

To-dos:

  • Address Module serialisation issue (Proxies aren’t serialisable)
  • Spawn hooks worker only when custom hooks are supplied
  • Convert ESM defaultResolve to synchronous (no real change there)
  • Convert import.meta.resolve() to synchronous
    • When no custom hooks are supplied, just regular sync
    • When any custom hooks are supplied, leverage worker + atomics to simulate sync
  • Handle very large request/response payload (main ↔︎ worker)

Notable changes:

Custom ESM loader hooks run on dedicated thread

ESM hooks supplied via loaders (--experimental-loader=./foo.mjs) now run in a dedicated thread, isolated from the main thread. This provides a separate scope for loaders and ensures no cross-contamination between loaders and application code. A few things to know:

  • Communication between application code and loaders code must now be done via the globalPreload hook’s port. Global variables are not shared between scopes.
  • The dedicated worker thread is spun up only when at least one custom loader is provided.
  • There is an overhead cost to spinning up this thread, increasing both startup time and memory consumption. These are non-trivial, but likely not burdensome.

Synchronous import.meta.resolve()

In alignment with browser behavior, this function now returns synchronously. Despite this, user loader resolve hooks can still be defined as async functions (or as sync functions, if the author prefers). Even when there are async resolve hooks loaded, import.meta.resolve will still return synchronously for application code.

Contributed by Anna Henningsen, Antoine du Hamel, Geoffrey Booth, Guy Bedford, Jacob Smith, and Michaël Zasso in #44710

Loading
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

author ready PRs that have at least one approval, no pending requests for changes, and a CI started. esm Issues and PRs related to the ECMAScript Modules implementation. lib / src Issues and PRs related to general changes in the lib or src directory. loaders Issues and PRs related to ES module loaders needs-ci PRs that need a full CI run. notable-change PRs with changes that should be highlighted in changelogs.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

esm, loader: move to own thread