Overview
Every once in a while, you run into a bug that that sends you down a rabbit hole, questioning everything you thought you knew about how your system works. This is the story of one such bug - a performance anomaly that took us deep into the internals of Node.js worker threads, WebAssembly execution, and V8’s compiler optimizations.
At Attio, we have developed a custom JavaScript runtime that allows us to run untrusted third-party code in a secure way. The runtime is based on QuickJS and runs entirely in WebAssembly (WASM). On every execution of third-party code, we spin up a WASM module, run the code inside it, and then discard the module. This helps to ensure safe isolation between different executions.
While deciding on the best way to deploy this service, we experimented with using Node.js worker threads. However, once we deployed the thread based solution into production, we found extremely surprising results.
The experiment
We set up an experiment to measure the performance impact of parallelization. Each worker would spawn its own runtime context and run a simple loop to assess whether we could benefit from multi-threaded execution.
But when we ran the script, the results were surprising:
→ Running a single worker took a quite reasonable 330ms.
→ Running four workers made each worker's individual execution time balloon to 4053ms.
→ Cutting the worker count to two reduced execution time to 1200ms per worker.
The parallelization we expected to help was making things worse. Adding more workers increased per-worker execution time instead of dividing the load. We needed to understand why.
V8's Liftoff compiler
We dug into V8's documentation. The unpredictable performance and the link between worker count and execution time pointed to how V8 compiles WebAssembly.
The goal of Liftoff is to reduce startup time for WebAssembly-based apps by generating code as fast as possible. Code quality is secondary.
Liftoff is V8's baseline compiler for WebAssembly. It prioritizes compilation speed over the quality of the generated code. In our scenario, with multiple runtimes and repeated WASM execution, that trade-off worked against us.
Key takeaways
Liftoff can be a bottleneck when running many WebAssembly instances in parallel. If you see execution time grow with the number of workers, V8's compilation strategy is a likely cause. Disabling Liftoff for hot WASM paths can restore predictable, faster performance.
Disabling Liftoff
We ran an experiment: we disabled Liftoff using a V8 flag in Node.js. The inconsistent execution times disappeared. Workers scaled as we had originally expected.
If you hit similar behaviour with Node.js, WebAssembly, and worker threads, try turning off Liftoff for your critical code paths. In our case, it resolved the anomaly and gave us the performance profile we needed.
