Speeding up emerge depgraph calculation using PyPy3

If you used Gentoo for some time, you’ve probably noticed that emerge is getting slower and slower. Before I switched to SSD, my emerge could take even 10 minutes before it figured out what to do! Even now it’s pretty normal for the dependency calculation to take 2 minutes. Georgy Yakovlev recently tested PyPy3 on PPC64, and noticed a great speedup, apparently due to very poor optimization of CPython on that platform. I’ve attempted the same on amd64, and measured a 35% speedup nevertheless.

PyPy is an alternative implementation of Python that uses a JIT compiler to run Python code. JIT can achieve greater performance on computation-intensive tasks, at the cost of slower program startup. This means that it could be slower for some programs, and faster for others. In case of emerge dependency calculation, it’s definitely faster. A quick benchmark done using dev-perl/Dumbbench (great tool, by the way) shows, for today’s @world upgrade:

• Python 3.9.0: 111.42 s ± 0.87 s (0.8%)
• PyPy3.7 7.3.2: 72.30 s ± 0.23 s (0.3%)

dev-python/pypy3 is supported on Gentoo, on amd64, arm64, ppc64 and x86 targets. The interpreter itself takes quite a while to build (35­­–45 minutes on a modern Ryzen), so you may want to suggest emerge to grab dev-python/pypy3-exe-bin:

$ emerge -nv dev-python/pypy3 dev-python/pypy3-exe-bin

If you want to build it from source, it is recommended to grab dev-python/pypy first (possibly with dev-python/pypy-exe-bin for faster bootstrap), as building with PyPy itself is much faster:

# use prebuilt compiler for fast bootstrap
$ emerge -1v dev-python/pypy dev-python/pypy-exe-bin
# rebuild the interpreter
$ emerge -nv dev-python/pypy dev-python/pypy-exe
# build pypy3
$ emerge -nv dev-python/pypy3

Update 2020-10-07: Afterwards, you need to rebuild Portage and its dependencies with PyPy3 support enabled. The easiest way of doing it is to enable the PyPy3 target globally, and rebuilding relevant packages:

$ echo '*/* PYTHON_TARGETS: pypy3' >> /etc/portage/package.use
$ emerge -1vUD sys-apps/portage

Finally, you can use python-exec’s per-program configuration to use PyPy3 for emerge while continuing to use CPython for other programs:

$ echo pypy3 >> /etc/python-exec/emerge.conf
# yep, that's pypy3.7
$ emerge --info | head -1
Portage 3.0.7 (python 3.7.4-final-0, default/linux/amd64/17.1/desktop, gcc-9.3.0, glibc-2.32-r2, 5.8.12 x86_64)