gh-146073: Add fitness/exit quality mechanism for JIT trace frontend by cocolato · Pull Request #148089 · python/cpython

cocolato · 2026-04-04T12:54:16Z

Issue: Improving trace quality by tracking "fitness" and "exit quality" #146073

cocolato · 2026-04-06T16:38:30Z

It appears that the current parameters do not yet guarantee runtime safety; I will continue to work on fixes and optimizations.

markshannon · 2026-04-07T08:28:12Z

I've commented on the issue #146073 (comment)

cocolato · 2026-04-11T14:03:24Z

@markshannon @Fidget-Spinner gentle ping, I’d like to hear your suggestions about the current parameters

markshannon

I've a few comments, mostly broad ideas and suggestions, rather than anything that needs fixing.

I think we should merge this soon. We can tweak the parameters later as we refine our ideas.

markshannon · 2026-04-13T16:26:52Z

Include/internal/pycore_optimizer.h

+
+/* Exit quality thresholds: trace stops when fitness < exit_quality.
+ * Higher = trace is more willing to stop here. */
+#define EXIT_QUALITY_CLOSE_LOOP      (FITNESS_INITIAL / 2)


I think we want this higher, close to FITNESS_INITIAL*0.9. It is only super-short loops that we want to unroll.

markshannon · 2026-04-13T16:37:57Z

Include/internal/pycore_optimizer.h

+ * Higher = trace is more willing to stop here. */
+#define EXIT_QUALITY_CLOSE_LOOP      (FITNESS_INITIAL / 2)
+#define EXIT_QUALITY_ENTER_EXECUTOR  (FITNESS_INITIAL * 3 / 8)
+#define EXIT_QUALITY_DEFAULT         (FITNESS_INITIAL / 8)


I'd increase this to make sure that the fitness cannot drop from above EXIT_QUALITY_DEFAULT to below EXIT_QUALITY_SPECIALIZABLE in a single uop.

markshannon · 2026-04-13T16:38:00Z

Include/internal/pycore_optimizer.h

+ * N_BACKWARD_SLACK more bytecodes before reaching EXIT_QUALITY_CLOSE_LOOP,
+ * based on AVG_SLOTS_PER_INSTRUCTION. */
+#define N_BACKWARD_SLACK           50
+#define FITNESS_BACKWARD_EDGE      (FITNESS_INITIAL - EXIT_QUALITY_CLOSE_LOOP \


A backwards edge isn't necessarily bad, although too many suggests a poor trace.

Maybe don't penalize the first backwards edge much, but penalize subsequent ones a lot.
What we really want is to avoid unrolling a loop that doesn't include the trace start.

It would be easier to reason about, if we used a simpler calculation for the value.

markshannon · 2026-04-13T16:41:05Z

Include/internal/pycore_optimizer.h

+
+/* Backward edge penalty for JUMP_BACKWARD_NO_INTERRUPT (coroutines/yield-from).
+ * Smaller than FITNESS_BACKWARD_EDGE since these loops are very short. */
+#define FITNESS_BACKWARD_EDGE_COROUTINE  (FITNESS_BACKWARD_EDGE / 4)


It is not the length of the loop that matter. We want to include yields in generators in the traces of the loop that calls them, whether that is a yield from loop or a for loop doesn't much matter.

Python/bytecodes.c

Python/optimizer.c

markshannon · 2026-04-13T16:52:12Z

Python/optimizer.c

    assert(curr_instr->op.code == JUMP_BACKWARD_JIT || curr_instr->op.code == RESUME_CHECK_JIT || (exit != NULL));
    tracer->initial_state.jump_backward_instr = curr_instr;

+    // Reduce side-trace fitness as chain depth grows, but clamp the reduction


This is probably fine for now, but reducing the fitness can skew some of our assumptions about back edges and such.

One other thing to note for the future, is that the fitness at an exit might help us pick a better warmup value for that exit.

Mark is right, we should remove this feature. Side exits are important and we shouldn't penalize them.

cocolato · 2026-04-14T08:18:15Z

@markshannon Thanks for review! I'm holding off on changing the fitness parameters for now, but I can run some benchmarks if you think it's necessary.

Fidget-Spinner · 2026-04-14T08:34:16Z

Still seeing a big slowdown in richards on https://github.com/colesbury/fastmark:

Main of this branch:

Benchmark                     Time      Useful Work
richards                      115.9 ms      (100%)
richards_super                103.5 ms      (100%)

This branch:

Benchmark                     Time      Useful Work
richards                      119.2 ms      (100%)
richards_super                118.5 ms      (100%)

I'm going to check if this is affecting the optimizer somehow.

…ER_EXECUTOR for RESUME

Treat back edges as an exit, not a penalty, this way they are more likely to end at a backedge instead of ending at random spots

cocolato · 2026-04-14T14:06:44Z

Python/optimizer.c

-        OPT_STAT_INC(trace_too_long);
-        goto done;
-    }
+    assert(uop_buffer_remaining_space(trace) > space_needed);


MAX_TARGET_LENGTH must be less than UOP_MAX_TRACE_LENGTH / OPTIMIZER_EFFECTIVENESS; otherwise, assert(uop_buffer_remaining_space(trace) > space_needed) will fail. Or should we revert the assert to a check?

We want to keep it as an assert.
Otherwise we need to handle an awkward failure mode.

markshannon · 2026-04-14T14:18:01Z

Increasing the max trace length is only going to help if the trace is stopping too early.
How big are the traces on main for richards?

cocolato · 2026-04-14T14:29:50Z

Increasing the max trace length is only going to help if the trace is stopping too early. How big are the traces on main for richards?

The main branch executors dump by sys._dump_tracelets('richards.gvz'):

cocolato · 2026-04-14T14:32:17Z

After this pr:

Fidget-Spinner · 2026-04-14T14:33:31Z

I think we can safely reduce the max trace length, let me do that.

There were two problems with the older code:

Branch penalty was treated as before instruction count, when it should be treated as the sum over the expected trace length.
Treating non-closing JUMP_BACKWARD as penalty rather than exit quality seems to make it such that we stop traces at a certain offset after seeing a JUMP_BACKWARD. This isn't what we want. instead, we want to treat it as an exit that is worth stopping at (to increase the likelihood of linking to another trace).

New code has almost no slowdown on Richards, and a huge speedup on telco benchmark.

Benchmark                     Time      Useful Work
richards                      113.4 ms      (100%)
deltablue                     196.4 ms      (100%)
raytrace                      267.3 ms      (100%)
nbody                         150.6 ms      (100%)
go                            111.5 ms      (100%)
telco                        3517.8 ms      (100%)

Main:

Benchmark                     Time      Useful Work
richards                      111.6 ms      (100%)
deltablue                     192.2 ms      (100%)
raytrace                      270.4 ms      (100%)
nbody                         151.1 ms      (100%)
go                            112.6 ms      (100%)
telco                        3809.7 ms      (100%)

cocolato and others added 13 commits April 1, 2026 00:57

add fitness && exit quality mechanism

1bfa176

Rewrite the code structure

2f9438a

address review

709c0a1

address many reviews

ef6ac24

Merge branch 'main' into jit-tracer-fitness

21f7122

optimize some constants

b99fe61

fix comment

d09afb5

fix constent

c9957c3

reduce frame penalty

9447546

add debug log

7d3e4c4

address review

2c1b5e0

address review

2409b2f

Merge branch 'python:main' into jit-tracer-fitness

88a91dc

cocolato requested review from FFY00, Fidget-Spinner, ZeroIntensity, ericsnowcurrently and markshannon as code owners April 4, 2026 12:54

bedevere-app bot added the awaiting review label Apr 4, 2026

bedevere-app bot mentioned this pull request Apr 4, 2026

Improving trace quality by tracking "fitness" and "exit quality" #146073

Open

cocolato added the skip news label Apr 4, 2026

cocolato added 2 commits April 6, 2026 13:51

Merge branch 'main' into jit-tracer-fitness

4e12f04

fine tune parameters

4bd251e

This comment was marked as outdated.

Sign in to view

remove some special cases

1d93208

cocolato added 3 commits April 10, 2026 17:50

Merge branch 'main' into jit-tracer-fitness

386c23a

rewrite fitness mechanism

83fd8ab

remove static assert

c900563

This comment was marked as outdated.

Sign in to view

Merge branch 'main' into jit-tracer-fitness

97d8be4

markshannon reviewed Apr 13, 2026

View reviewed changes

cocolato and others added 4 commits April 14, 2026 10:49

Merge branch 'main' into jit-tracer-fitness

559b164

address partial review

7a5e1fe

restore slots_rev

9324df0

address review

e69443b

Fidget-Spinner added 6 commits April 14, 2026 19:05

Race MAX_TARGET_LENGTH to 800, compute branch after slots, ignore ENT…

751a1d9

…ER_EXECUTOR for RESUME

reduce MAX_TARGET_LENGTH

896e4fe

fix tests

1364159

fix a bug

9fbec75

magic numbers

76b9c9e

Treat back edges as an exit, not a penalty, this way they are more likely to end at a backedge instead of ending at random spots

lint

d565f41

cocolato commented Apr 14, 2026

View reviewed changes

reduce the trace length to less than half

598d332

Uh oh!

Conversation

cocolato commented Apr 4, 2026 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

cocolato commented Apr 6, 2026

Uh oh!

markshannon commented Apr 7, 2026

Uh oh!

This comment was marked as outdated.

cocolato commented Apr 11, 2026

Uh oh!

markshannon left a comment

Choose a reason for hiding this comment

Uh oh!

markshannon Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

markshannon Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

markshannon Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

markshannon Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

markshannon Apr 13, 2026

Choose a reason for hiding this comment

Uh oh!

Fidget-Spinner Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

cocolato commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Fidget-Spinner commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cocolato Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markshannon Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

markshannon commented Apr 14, 2026

Uh oh!

cocolato commented Apr 14, 2026

Uh oh!

cocolato commented Apr 14, 2026

Uh oh!

Fidget-Spinner commented Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cocolato commented Apr 4, 2026 •

edited by bedevere-app bot

Loading

cocolato commented Apr 14, 2026 •

edited

Loading

Fidget-Spinner commented Apr 14, 2026 •

edited

Loading

cocolato Apr 14, 2026 •

edited

Loading

Fidget-Spinner commented Apr 14, 2026 •

edited

Loading