Restore 304 performance after fixing `FileResponse` replace race #9002

Job	Run time
Linter	1m 17s
Generate llhttp sources	6s
Benchmark	5m 0s
Test (3.9, ubuntu, false)	3m 47s
Test (3.9, macos, false)	3m 56s
Test (3.9, windows, false)	4m 46s
Test (3.9, Y, ubuntu, false)	4m 4s
Test (3.10, ubuntu, false)	3m 54s
Test (3.10, macos, false)	4m 0s
Test (3.10, windows, false)	4m 42s
Test (3.10, Y, ubuntu, false)	4m 14s
Test (3.11, ubuntu, false)	3m 31s
Test (3.11, macos, false)	6m 34s
Test (3.11, windows, false)	5m 12s
Test (3.11, Y, ubuntu, false)	4m 1s
Test (3.12, ubuntu, false)	4m 11s
Test (3.12, macos, false)	4m 12s
Test (3.12, windows, false)	4m 51s
Test (3.12, Y, ubuntu, false)	5m 10s
Test (3.13, ubuntu, false)	4m 14s
Test (3.13, macos, false)	3m 39s
Test (3.13, windows, false)	5m 17s
Test (3.13, Y, ubuntu, false)	4m 27s
Test (pypy-3.9, Y, ubuntu, false)	7m 27s
check	31s
Pre-Deploy	1s
Tarball	1s
Build wheels on ${{ matrix.os }} ${{ matrix.qemu }}	1s
Deploy	1s
	1h 43m 7s

Provide feedback