Update Saturday Oct 13th, 2018
We don’t usually have a month going by where someone doesn’t ask this question ‘Why did you implement Grin in Rust instead of C’ in one of our various communication options. Sometimes this is accompanied by a little diatribe in which the poster implies that if only we had implemented Grin in a language that ‘everybody knows like C++’ they would have been able to lend their considerable skills to the project.
Alas, we must forge ahead without these people (I don’t think we’ve heard from any of these people twice). But my answer to anyone who asks either online or in real life is simply ‘When you write code in Rust, it does exactly what you think it’s going to do, every time.’
A lump of C/C++ code of any meaningful size rarely gets itself into a state where it does what’s intended without a considerable amount of time spent in front of a debugger tracing down odd threading issues, investigating segfaults, and dealing with general weirdness. Of course it’s possible to include tools or implement standards to lessen these issues, but it’s time consuming, and every team will implement all of this differently for every project. Then of course different compilers will produce different code and there are approximately 4000 different C++ standards.
Because Grin uses rust, we rarely have to deal with this sort of oddness, which frees up time to focus on the actual issues rather than spending weeks in LLDB. Now there are plenty of article about the merits of Rust on the Internet, and I’m not trying to create another one. This is just for context when I talk about this week’s work.
So Cuckoo-miner, which provides a Rust bridge to slightly-modified versions of @tromp’s cuckoo/cuckatoo miners, is a Rust-C hybrid that I put together well over a year ago when I knew far less about Rust than I do now. The Rust-side is little more than an interface meant to be called by other rust projects, while the C side is an FFI shim to call the C code in the C solvers, however it also contains quite a bit of C code for queue management and to enable the multi threading needed to run multiple devices at once. When I was originally putting this together, I decided to do the queue management on the C side because I reckoned it had to work as fast as possible and doing it in C is the way to do this.
Nowadays, I realize that is entirely the wrong approach to take. When you’re using Rust, any decision to use C should be similar to when you’re primarily using C and are considering doing an algorithm in assembly. You might be able to make it faster, but an optimizing C compiler is generally going to produce far better assembly than you can hand-write, unless it’s something very specific and you really know what you’re doing. Similarly, you should never consider doing something in C when working primarily in Rust for ‘speed issues’ unless there’s some demonstrable need. The cuck(at)oo solvers themselves are good candidates for being written in C as they’re very hardware specific and need to interface into other C frameworks. The queue management portion of cuckoo-miner is not. It should have been kept in Rust as much as possible and the interface into the plugins themselves kept far thinner.
Reason I’m saying this is because over the last two weeks I integrated 3 solvers into cuckoo-miner, (mean CPU, lean CPU and cuda GPU) and each and every time I spent hours tracking down obscure issues that would either have been caught at compile time if those portions of the code had been using Rust, or simply not have been able to happen in the first place. The process of tracking down these issues are extremely frustrating and complicated exercises, usually hours spent hopelessly starting into a debugger with your branch of the code littered with print statements. This kind of session does not happen when working on Grin itself, and I don’t think I’d ever advocate C for a new project unless it’s absolutely required. Rust would simply a better choice in 99% of cases.
So with that little glorified tweet out of the way, here’s what I actually did this week:
-
Integrated the Mean CPU miner into Grin-miner, which took a bit longer than expected as explained above. I’d love to re-do parts of cuckoo miner to move the queue management into Rust, but it works now, isn’t mission-critical and can be rewritten at any time. Probably more important things to focus on for the time being.
-
Updated libsecp with the very latest and hopefully final version of element’s bulletproofs A few critical (but T3 consensus breaking) fixes in there, so glad to get it in before T4. Also a few upstream changes to how commits are represented meant a bit more headache integrating this than anticipated (seems to have been the theme of the week)
-
Creation of the testnet4 branch and integration of all of the wallet, aggsig, bulletproof PRs, etc. I’ll go into far more detail on T4 next week (with an exact list of what’s changed), as there are still a few bits and pieces on it being done, suffice it to say this is where all the focus is now.
-
Finally a bit of a breather between large coding efforts to catch up and better understand some recent concepts, particularly Header MMRs, the newly proposed
prev_rootfield and flyclient, our dual PoW difficulty adjustments and a couple of other things. I can get a bit of myopia sometimes when i’m deep in my own things, so find it helpful to come up for air once in a while and get back up to speed with everything else that’s going on (at the rate things are moving at the moment that could be a full-time job in itself).
That’s it for now, and the next week will most likely be about pre-release T4 testing and addressing whatever bits and pieces come up there. Everything is almost in place, and looking very forward to getting T4 out there.