MUID miner rewrite in Rust

When I started playing with Microprediction, I wrote a simple MUID miner in Python. I used multiprocessing to parallelize the work, and while it wasn’t super fast, it was fast enough to get me started.

If you’re not familiar with them, I wrote a blog post about MUIDs last year.

Rewriting this in a faster language like Rust or C was on my todo list for a while, and I finally got around to it.

The crates I used were sha256 v1.4.0 and crossbeam v0.8.2. SHA256 is because the keys we mine need to have specific prefixes when hashed with SHA256, and crossbeam is because I like its channel API.

The Python version used very simple data structures, I stored the prefixes of valid keys as strings in a set. Determining the “difficulty” of a key was done by getting the length of the prefix. This worked fine, but it was overall not the most efficient way to do it.

In the Rust version, I converted the prefixes to 64-bit integers, and stored the difficulty in the least significant 4 bits. After that, all the prefixes were stored in a hash set.

Another speed gain was in generating the keys. In the Python version, I was getting random bytes from /dev/urandom and converting them to hex. This resulted in a system call, and we really didn’t need cryptographically secure randomness.

In the Rust version; I read some bytes from /dev/urandom for each miner thread, and then incremented the bytes by 1 for each key. This was much faster.

Something else that made the Rust version more convenient to use was the include_str! macro. This allowed me to bundle the data file that contained the animal prefixes with the binary, and resulted in a single executable that could be copied easily.

Writing the initial version in Python was definitely the right choice. And coming back for a version 2 in Rust meant that I had an idea of what I needed, and a reference implementation to compare against.