We rewrote cross-encoder inference in Rust, benchmarked it against Python, and on a small reranking model it came out only 1.1x faster at the median. That is not a sign of a slow Rust implementation. The Python libraries we compared against, fastembed and sentence-transformers, hand the same work to the same C++ engine, onnxruntime, so all three spend almost all of their time in the same compiled code.

In the Rust community, rewriting code in Rust is called “oxidizing” it, since rust is also what iron turns into when it oxidizes. This post covers how we oxidized cross-encoder inference into a library, cross-encode-rs, how it works, where it does pull ahead (about 1.4x on a larger model), and where it does not (model load time).