Yeastplume - Progress update thread - Feb - April 2018

Announcements
1

Hello Grinners and MimbleWimblers,

Given that I’ll be working full-time for the community very soon, I was going to start up a separate blog to keep everyone informed of my own progress updates and personal happenings. However, since this shiny new forum has come into being, I think a thread here is a much better way of doing it. Rather than sending everyone off to a separate site, I’ll keep this thread updated week-to-week with a description of what I’ve been working on, as well as my intentions for the coming week.

Anything in this thread is my own, and doesn’t necessarily reflect the views of the project as a whole.

Personal Status

I’ll be 100% Grin from Feb 1st onwards, perhaps briefly pausing in the evenings to tend to the deep pathos of a screaming Yeastling. I’ll be a little bit quieter than usual activity-wise until then as I need to wrap up loose ends in the place I’m leaving, but I’m very much looking forward to being able to think about nothing but Grin.

Funding Update

The funding campaign a few weeks back was a huge success, and my thanks again to everyone who donated. Unfortunately and annoyingly, Bitcoin also decided to begin a long, slow (in crypto-terms) decline since the campaign, which is bad news for everyone involved in the space professionally and for many people on a personal level. In my case, watching it decline removed any small sense of security I had for the funding period, watching €50k become €40k and then €30 and then wondering whether there’d even be enough left to cover the intended funding period when it came time to actually withdraw the funds.

The intention was to use Bitcoin’s openness to ensure transparency, however, in retrospect relying on something so volatile for short-term funding needs was probably a mistake we’ll need to hedge somehow for future fund raising efforts. In the meantime, after speaking with Igno I’ve decided that the best course of action right now is to convert the entire amount into cash while there is still definitely enough remaining to cover the original intended 3 month funding period. Nobody knows whether BTC will recover tomorrow or whether it’s on a steady decline back to €1000, and keeping the funds in the BTC account basically amounts to placing a short-term bet that the price will hold steady or increase. The funds aren’t intended for this kind of speculation, so taking them out now ensures they’re available for their intended purpose, and removes a source of distraction from me (having to be worried about the price of Bitcoin). I apologise for switching course slightly here, and especially apologise to those who donated, as the amounts they’d give turned out to be less than they intended. However, I really felt this was the best course of action given the situation.

Development Work

This is just an overview of what I’ve done since the campaign, to give some context for future updates. Please note that every time I outline my work in this thread it’s in the name on openness and transparency for my community funding. This is not for bragging rights, and many other people have contributed just as much or more in the same timeframe.

  • Written a single-signer implementation of an Aggregated Signature (AKA Schnorr) library adapted from Andrew Poelstra’s work. This is for inclusion in our fork of libSecp256k1 to support Grin’s Aggregated Signature transactions. What this does is allow wallet-to-wallet transactions to happen without either party revealing their private keys, and in such a way that neither party can pretend they didn’t receive it (both features lacking in Testnet1). Details can be found via the PR

  • Integrated such into Grin by modifying the wallet workflow… Grin master now uses AggSig transactions.

  • Updates to cuckoo-miner to support @tromp’s latest CUDA Miner, bug fixes for asynchronous mining mode, and some tweaks to mining parameters in Grin to better tune for Cuckoo 30 on Testnet2. Basically, the master version of Grin should now allow mining at full Cuckoo 30, and on multiple devices in parallel… see PR for further details.

Upcoming

I’m not 100% decided what I’ll be working on next, but here are the main things currently on my radar. This will change over time:

  • More fixes to mining parameters (there’s some floating point arithmetic in the difficulty adjustment I’ve been talking with @tromp about how to get rid of) and tweaking.
  • There is some work on AggSig remaining to handle concurrent transactions and handle errors better… I put this aside for a bit to update cuckoo-miner, but this still needs to be done
  • Bullet Proofs … for our purposes, smaller range proofs (under 1k each instead of 4-4.5k). I need to check on the status of Andrew’s work, but I need to familiarise self with their internal workings before thinking about integrating them

This is already far longer than intended, so will sign off now. Will update this thread with future updates after I’ve settled in to my new job.

8 Likes
3

So without any ado, a brief update for Friday Feb 2nd, 2018

I’ve been a full time employee of Grin for nearly 48 hours now! But the work I’ve been doing over the past couple of days has mostly been finishing up work I’ve been doing on mining. Since the last posting, these are the major tasks I’ve undertaken:

The changes here are to clean up the difficulty adjustment code to simplify it and make it easier to follow (important for consensus-critical code), and to allow for earlier adjustment of difficulty. In the previous version, the code would not adjust from genesis difficulty until a number of blocks had been mined. After these changes, it will start coming down immediately after the first block is mined if it’s way off. This means that if the initial difficulty is set too high on launch (which we should be aiming for,) then the difficulty will immediately come down, a good thing lest miners start getting dismayed by hopelessly long blocks.

  • Integration of lastest CUDA miner and bug fixes

The latest MEAN CUDA miner has been fully merged into cuckoo-miner and grin, and you can now use it to test full-production style mining in within Grin. To assist with this and to get us started collecting stats, I’ve put together a Cuckoo 30 Mining Guide which can be found here:

There were quite a few parameters needed to be integrated within the final version, which meant more changes to the supporting cuckoo-miner code than I’d expected. However, it’s all there any working now, and hopefully most changes from now on will be bug fixes and incremental improvements as the community starts testing.

Hopefully this last bit of work draws a (very faint) line under mining changes for Testnet2, and as of next week I’ll be moving on to other things that need work. My current plan for early next week is:

That’s just a start… and I’m sure plenty of other things will come up

1 Like
4

Update for Friday, Feb 9th

It’s been my first full week working on Grin full-time! I managed to get a few decent chunks of work done, with highlights here:

  • Further aggsig functionality, mostly to ensure that a listening wallet can handle multiple concurrent incoming requests without failing https://github.com/mimblewimble/grin/pull/685

  • Added some support code to allow for automated checking of wallet contents during automated testing, something that was missing before. I also added some basic transaction integration tests (also something that was lacking) so future updates to wallet code should have a reference to test against.

  • Some consideration, thought and discussion towards the entire transaction workflow, notably around fees and which party in a transaction should be responsible for posting the transaction to the chain. Nothing was resolved, but I did take an action to better document the current transaction workflow (with pretty UML diagrams) so that we’ll have a reference to point to and modify during any future discussions around transactions and fees. I’ve started on that and hope to have a draft of that out early next week.

  • Bulletproofs - I made a fairly large commit into our libsecp256k1 fork that integrates @apoelstra’s work to date on bulletproofs… as well as create Rust wrappers for the prove and verify functions. The good news is that it’s integrated now and I can create a 674 byte bullet proof in Grin (massive storage savings). The bad news is there’s still some functionality missing from bulletproofs that we need, mainly to hide an amount within to allow for wallet reconstruction. So even though we basically have a test version ready for integration into Grin, I don’t actually want to integrate it until we have it. I’ve added an update on the current status into the bulletproof issue here

  • More cuckoo miner updates, mostly integrating the very-latest version of the solver, playing around with default values and updating the build to make it a bit more robust based on feedback from the field. It seems quite a few people have manged to build the CUDA plugin and exercise their GPUs against the master version of Grin, which is great to see. I hope to start collating some of the stats from the mining result threads soon.

  • This one was exciting: recorded a Podcast with Adam B. Levine and Andreas Antonopoulos for the Let’s Talk Bitcoin The topic was mostly MimbleWimble itself… an attempt to explain how it works for a general audience, which is a very difficult thing to do at the best of times. Andreas I think did a good job of taking my dev-speak and making it a bit more accessible, so hopefully the final product will be at least somewhat informative and it won’t be just me ranting incoherently about Pedersen commits. It should be out over the next couple of weeks.

  • Did some reading and catching up on the issues and work @igno.peverell and @antioch have been addressing, as I want to get back up to date with the status of the core code in advance of taking another large chunk of work. I look very forward to the next Dev meeting on Feb 13th so we can all sync and I can figure out what to address next without stepping over anyone else’s current work.

3 Likes
5

Update Friday, Feb 16th

Starting to settle properly into a nice Grin-related daily routine, head stuck into terminal windows for most of the day, any needed gitter communication in the evenings, and a little break once in a while to pull a yeastling down from the roof. Two major things this week:

  • Bulletproofs (or at least, the most current version) are fully integrated into Grin via https://github.com/mimblewimble/grin/pull/711, (or at least they will be once I merge the PR). I’ve elected to insert a flag to allow a chain to use either bulletproofs or ‘rangeproof classic’ for the time being, defaulting to bulletproofs. This does break wallet reconstruction as we currently don’t have the ability to hide a message within bulletproofs as we do the original rangeproofs, but this can be added/fixed later on. If @apoelstra doesn’t get to it before we need it I’ll take a pass at making the need changes within the bulletproof API of our secp256k1 fork.

  • Get the sums out of sumtrees … This was something I thought would take a day or so to but turned out to be a much larger job than I’d thought, mostly due to my needing to fill in gaps in my understanding of the low-level UTXO set storage functionality within Grin. Basically, we’ve decided that what we’ve been calling ‘Sumtrees’, in which we’ve been storing merkle-hashes and commitment sums, don’t really need to sum. Therefore instead of storing a complex ‘HashSum’ object containing the Hash and Sum and UTXO Data it’ll be far more efficient to just store the simple hash in the tree instead and move the data itself (output, rangeproofs, tx kernels) into much simpler flat files. This actually simplifies a lot of the code (which was a bit of a chain of templatized templates to handle a few different types of elements, which also go away). However, I’ve had to put together a new flat-file storage mechanism and then try to ensure I haven’t broken anything. Work is ongoing in the associated PR: https://github.com/mimblewimble/grin/pull/702

In addition to the above, I also put together a diagram of our entire transaction workflow, to document and to make it simpler to have future discussions here: https://github.com/mimblewimble/grin/blob/master/doc/transaction/grin-transaction.png

A good chunk of the coming week will most probably be taken up with #702… big long chunks of work where I educate myself on core-components don’t make for exciting reading or very good live-streaming material, but it’s all necessary activity.

1 Like
7

Update for Friday, Feb 23rd:

Plenty of solid progress this week, mostly on the same issues from last week:

  • I managed to get all of the ‘sum out of sumtrees’ work done and merged in the PR -https://github.com/mimblewimble/grin/pull/702. As mentioned last week, this turns what were formerly SumTrees into simpler hashtrees, and splits the storage of hashes and their related elements into separate files, which should hopefully make them easier to manage. I’m very glad I took on these changes because they’ve given me a solid understanding of (and appreciation for) PMMRs as well as a much better handle on Grin’s data storage than I had before.

  • Large progress on the Bulletproof front in that I managed to implement the ability within our libsecp256k1 fork to embed 64 bytes worth of message into a bulletproof (as @apoelstra suggested here), as well as create and write tests for the necessary Rust bindings. This means that we should now be able to embed amounts, as well as an Output’s switch commitment hash right into the rangeproof (which we require to ensure it remains non-malleable).

And if you haven’t heard it yet, the (Let’s Talk Bitcoin Podcast)[https://soundcloud.com/mindtomatter/ltb-356-privacy-on-the] I recorded a couple of weeks ago was released and seems to have been generally well-received. (Other than a terrible error around the origin of the Invisibility cloak, hopefully I can one day be forgiven).

That’s really about it for this week… I’ll be moving along to integrating the bulletproof messages into Grin properly, as well as some remaining issues around storage, before picking my next major task. Have a good weekend all and looking forward to the dev meeting next Tuesday!

1 Like
8

Update for Friday March 2nd:

Snowed in most of the week, which is something that never, ever happens in Ireland. What would be a run-of-the-mill snowfall in places more equipped to deal with it becomes a national event shutting down schools and shops and forcing parents to spend an uncomfortably long and continuous amount of time exposed to their children. Despite all this, I think I managed to get some good additions in:

  • Bulletproofs are done and fully integrated into Grin (for now), and I manged to get over a couple of problems I was having with them last week to the point where I think they’re mostly good for Testnet2 (review and updates to the underlying bulletproof implementation aside). In short, message passing works properly (which I’m using to embed the amount only so that wallet reconstruction will continue to work) and the switch commit, which we use for a layer of quantum protection as well as identifying outputs in the UTXO set, is being properly signed in the bullet proof in a manner that bulletproof validation will fail if the switch commit isn’t the original used to sign the proof. If you’d like a more detailed explanation of what’s going on with bulletproofs and the switch commit, I’ve done a bit of a write-up in an issue here that goes into further detail: https://github.com/mimblewimble/grin/issues/734

  • In an effort to make the PMMR data set more corruption-resistant, I’ve been doing some work on https://github.com/mimblewimble/grin/pull/739, which basically stores an index of file positions in the PMMR files each time a block’s data is successfully added to the dataset. Further, on start-up the system will us this stored index for the last successful block to set its read position in the data files, whereas before it was simply using the size of the file. This on its own isn’t going to provide all of the corruption resistance we’ll need for testnet2, but by having file positions available for the last few blocks we’ll have more options to deal with minor corruption in the dataset. Rather than declaring the entire dataset invalid, we should be able to roll back a few blocks in the dataset until we get valid data, and then continue from there. Storing this and retrieving on startup is just the first step, so this PR is a foundation for further robustification efforts. Also, between this PR and last week’s work on the PMMR cleanup, I’ve got a much better handle on the nitty-gritty details on what’s going on in the grin/chain portion of the code, one of the places I’d spent the least amount of time on previous to now.

  • A few minor miner-related things, updating a few parameters that I’ll probably release shortly. Apparently, however, someone is working on GPU speedup cuckoo-miner bounty, so I’ll probably wait to see the results of that before pushing an update to the miner.

There are a couple of minor issues outstanding issues I want to clear off my plate early next week (I’ve assigned them to myself in the issues list,) but it would seem work towards testnet2 is really chugging along at a good pace to the point where there isn’t really a large specific chunk of work to grab in the planned Testnet2 feature set. I’m not sure what I’ll take once my self-assigned issues are finished, but I have plenty of ideas, particularly around infrastructural work in advance of testnet2. Will have a clearer idea very soon.

Until then, I think one of the lumps of snow in the garden might be the missing Yeastling. Will definitely have to check at some point.

1 Like
9

Status update Friday, March 9th:

The current phase of mad development and feature additions is drawing to a close, and a phase of integration, testing and debugging all of the new testnet2 features is imminent. Development on a project usually goes through distinct phases, and the past couple of months have been a bit of a ‘Silo’ phase where all developers are more or less focused on their own parts of the project and coding mostly to unit tests… That is going to change very soon as the features are marked complete and we actually try and use what we’ve just created.

Some of this week’s work has been preparing for this coming phase, but I’ve also embarked on another small project that I think will be invaluable as we get into proper testing:

multi-grin
multi-grin.png2387×1389 98.5 KB

That’s 4 instances of Grin running in 4 tmux panes, each proudly displaying a new basic status screen instead of the familiar log scrolling. Up until now, the only way to tell what was going on with your running Grin instance was by sifting through the log output. The new status display should hopefully make it far easier to see the state of the server at a glance.

When the first TUI PR (Text User Interface) is merged, there will be a new option in grin.toml called ‘use_tui’… disabled by default, but it you set this to true then you’ll get this basic status screen instead logs scrolling on stdout. This is highly experimental at the moment (and the logo is nowhere near final, I just selected one that ASCIIfied well enough), but already I think it’s far more useful for quickly detect if the server is having an issue than staring at logs.

This is mostly placeholder at the moment, but in addition to the basic info here, I’d like to add an ‘advanced’ screen where you can tab through details of the various subsystems (peers, chain state, mining, etc), and then perhaps (some day in the future) allow for configuration through this UI. This will be an ongoing effort, but hopefully over the coming months with user feedback we can refine this into something a lot more approachable for users.

Also worked on this week:

  • Resurrecting the docker test framework… with testnet2 upcoming and lessons learned from Testnet1, I think it will be important to have a decent automated testing framework to allow us to exercise the various scenarios that come up in a wild-blockchain. While I’ve just focused on getting the framework working with master again this week, I want to extend this to allow for automatic detection of more cases. Fortunately, there is quite a bit of crossover here with the UI project, as the ability to collect certain stats from the code that aren’t currently being collected needs to be added for both cases.

  • A large renaming of some elements in the code, the last remnants of ‘sumtrees’ are gone, and the transaction data set is simply referred to everywhere as the ‘txhashset’

  • General ad-hoc testing… I’ve been coding to unit tests for a good while now while working on the various features, and it’s time to step back a bit once in a while and actually try running the whole thing. Have definitely flushed out a couple of issues that I intend to investigate further.

  • A bit of PMMR related work, ensuring PMMR elements are hashed with an index

And a few other minor bits and pieces. That’s it for this status update, and looking forward to moving into a mad test/debug/fix cycle over the coming weeks.

5 Likes
10

Update for Friday, March 16th,

Testnet2 is becoming more viable by the day, and the PRs are becoming smaller and more frequent, meaning there’s plenty of bug-fixing and cleanup of smaller but crucial issues going on. We’re starting to think about deploying this beast in some way or another, most likely in a limited manner for developer testing before we release it into the wild.

I spent the first couple of days of the week mostly on continuing TUI work, trying to get it to a state where it’s basically usable, that is to say its presence is more useful than its absence would be (even though it’s still very rough). I’ve tried to make the basic layout a bit more sensible and something we can add easily over time, and tried to structure the code to make it easy enough to follow for anyone wanting to make changes. I’ve also added in a first pass at Peer and Mining status screens, as well as some work on generic table views to support them. I think the TUI is useful at a basic level now, but this is nowhere near its final state and I look forward to updating it as we go based on user feedback.

Screenshot%20from%202018-03-16%2015-29-42
Screenshot from 2018-03-16 15-29-42.png1325×831 34.8 KB

Also thanks to @gyscos who develops the Cursive framework, who’s been very responsive so far fixing minor issues in the TUI library and has done an excellent job making a usable callback-based UI framework in Rust, an extremely challenging task thanks to Rust’s strict ownership requirements.

Beyond that, a flurry of fixes and updates to issues I’d been wanting to tackle for a while, which I hope will be of great value in testnet2 and beyond:

  • A Fix to the Bulletproof API to address an issue where the nonce and blinding factor had to be the same in order to unwind values. Not strictly needed for grin at present, but this may help anyone else who picks up the library for use or ourselves in the future.
  • Fixes to the server/wallet startup code The grin startup code was actually taking slightly different paths depending on the combination of config file and switches that was provided, so unified this as well as a simple fix for automatically running the wallet.
  • Send all panics and backtraces to the log file This one has been annoying us for a while; Previously, whenever the server paniced the results would only be sent to stderr and not to the log file, making it awkward to collect information from the field… further, if you weren’t running with RUST_TEST_THREADS=1 you could forget about seeing the backtrace from whatever the error was and had to run again to try to reproduce the error. This ensures a full backtrace is written to the log on each and every panic, which means logs should be much more self-contained for the purposes of troubleshooting.
  • PMMR data compaction… there was an issue on PMMR compaction whereby the hash file was being compacted, but the data file wasn’t working. Fixed this, which should greatly reduce the size of TXO datasets in the wild.
  • Restructure the Blockheader to make mining code a little easier and change the block hash to the pow hash based on @tromp’s suggestions

And a few other small bits I’m sure I’m forgetting. We should be looking to do an internal deployment amongst core developers any day now, and the yeastmonster is itching to start mining some pre-testnet 2 cuckoo-30 blocks.

Speaking of which, as an added bonus here’s what the Yeastmonster looks like today with it’s sad assortment of cards. I hope to get him kitted out with a few more 1080tis if I can ever find any. On the left is the first card in world history to find a 2^30 cuckoo cycle in less than a second, as well as a few other world firsts I’m sure we can make up:

IMG_3688
IMG_3688.jpg2016×1512 573 KB

Oh and it’s Paddy’s day weekend, and one on which Ireland is set to defeat England for a Six Nations grand slam. All of this means I’ll likely be drunk for most of it, so apologies to the gitter channel in advance.

8 Likes
11

Update Friday, March 23rd:

Loads of activity going on this week, with a greater than usual amount of changes and PR activity. Testnet2 will be upon us any day now, and we’ve been preparing, testing, and fixing large and obvious issues in our small pre-testnet2 devnet (in which the yeastmonster is currently running away with most of the blocks).

A lot of time this week was spent doing ad-hoc testing on several pre-testnet2 nodes and reporting or fixing issues as a result. In addition to that, I got a couple of things I personally wanted to get in before Testnet2 begins in earnest:

  • After one too many times having to deal with special cases around SwitchCommits and an impromptu developer meeting, Switch commits are gone. I won’t go into the details around why (see the PR for more details,) but they were proving to be a wart on the design and more trouble than they’re worth… however, the one thing I did really like them for was to give us an obvious means of identifying outputs for wallet restore. However, after some discussion with @apoelstra, we managed to come up with a much lighter scheme using message hiding within bullet proofs I won’t go into detail here (plenty of detail in the issue,) but suffice it to say this scheme works much faster than the previous switch-commit based version, and has further obvious room for optimisation… ultimately wallet restore should just have to look through each bullet-proof once to determine whether it’s a match. At any rate, I’m glad we could save this feature even after ditching switch-commits as I found it very useful in testnet1.

  • More TUI work adding a difficulty screen showing the times and difficulties of the past few mined blocks. This should give us an easy way to see the past few blocks as well as check that difficulty adjustment is acting as it should.

  • I also added Versioning and build information to the build, to give testnet2 users a simple way to check their version and build info. Not a huge amount of work to get included, but getting this sorted sooner rather than later should hopefully spare us a few headaches when we’re troubleshooting.

Looking very forward to the official deployment of testnet2!

Continuing on the theme of hardware that can be found in YeastHQ, (and hoping this is the sort of community who can understand these odd compulsions), I thought I’d celebrate the imminent release of testnet2 by purchasing this:

IMG_3693
IMG_3693.jpg480×640 57.2 KB

That’s a PAL Commodore 1702 Monitor… absolutely impossible to find anywhere in Europe, yet someone was selling a pristine and beautifully functioning example locally.

Unlike Mrs. Yeastplume, I’m sure you understand…

6 Likes
13

Update Monday, 9th April, 2018

Just realised it’s been far too long since the last post, and I apologise and ensure you this won’t become a habit, as these things are supposed to be weekly. My attention’s been diverted slightly over the past couple of weeks due to intense school holidays followed by a short illness. In any case it’s probably no harm, because with a long ongoing project like this, it’s vital to disengage for a bit days once in a while or you end up driving yourself quite mad. But all Yeastlings are back at school, most disease microbes are vanquished and, and the… mental particles, I guess… have been recharged so I’m back in full force.

I’ll give a fuller update at the end of the week, but I have it in my head to now to start looking into (in no particular order):

  • Wallet restore fixes, mostly enabling such on fast synced nodes, which first require a sensible method of chunking the UTXO set in some manner other than by height
  • Integration of @apoelstra’s latest bulletproof API, which should make wallet restore easier, hopefully contain some speed improvements and pave the way for future enhancements
  • Mining improvements, going to have a good review and play around with a new stratum server implementat here: https://github.com/mimblewimble/grin/pull/937, as well as a good look exciting new GPU miner developments from @photon and @urza here: https://github.com/mozkomor/GrinGoldMiner
  • Ongoing wallet design discussion

Fuller update at the end of the week… but just one more shameless plug for my ongoing funding campaign here: Funding Campaign - Yeastplume - May to Aug 2018 which I think is going well… we’re about 1/3 of the way there now, which is great. So very big thanks to everyone who’s donated so far, and I hope to be able to keep serving the project full-time for as long as is possible!

1 Like
14

Update Friday 13th April, 2018

So, as expected, the major tasks this week have been:

  • Fixes to wallet restore, and the underlying API calls to retrieve the entire UTXO set from a running grin node. Previously this had relied on entire blocks being present on the node, which don’t exist when the node has fast synced. https://github.com/mimblewimble/grin/pull/950 added the api calls, as well as a fix to the wallet code that should allow restore to work for the most part.

  • With respect to bullet proofs, the good news is that while integrating the latest code, I decided the easiest way to clean up our fork was just to take the latest upstream elements code + @apoelstra’s bullet proof changes, then merge our aggsig library on top. The end result is that I’ve managed to create a fairly clean libsecp branch that should be easy to keep up-to-date with upstream changes. The newly cleaned-up and rebased branch is here: https://github.com/mimblewimble/secp256k1-zkp/tree/vendor-rebase, and contains everything Grin should need.

  • Now the bad news… there appears to be a problem in that the latest bullet proof code has increased the size of a single bullet proof from 674 bytes to 675… given that we store PMMR elements in flat files where the sizes are assumed, I have no idea how we’re going to integrate the latest code… even if via a hard fork it still leaves the question of how to handle pre vs post fork data. This may be pointing to something we’ve overlooked in our data storage design, and will have to discuss this with the other devs. Also, due to format changes I’m not sure if the new code will validate proofs created with the older code. At any rate, while the secp lib is ready there are still some considerations before we can integrate the latest bulletproof code into grin.

  • and of course, other minor changes and fixes… one in particular as refactoring the code a bit to allow for the new stratum implementation: https://github.com/mimblewimble/grin/pull/937, which I’ll be looking at in far more detail once the bulletproof/wallet restore situation is resolved.

That’s it for this week, have a good weekend, and if you’re feeling generous, the funding campaign is still live and underway! Funding Campaign - Yeastplume - May to Aug 2018

1 Like
15

Update Friday 20th April, 2018

A couple of large mining-related items on list this week:

  • First, the creation of grin-miner, which will be a completely standalone miner for grin that interfaces with the new stratum mining server embedded into Grin. There are a few reasons why I think this is a good thing. Firstly, mining and plugins are the most crash-prone parts of the system (due to them usually being on the bleeding edge, tweaked with all sorts of parameters and tied into system libraries like CUDA,) and moving mining out of the grin process keeps the core node relatively rust-pure. This means that if a mining plugin crashes somehow, it won’t take down the whole node, making it easier to isolate both mining and code node issues. Grin-Miner mostly works at the moment, but needs one or two more fixes before it can be deployed.

  • Second, and obviously related the removal of all mining related code from Grin in favour of making grin-miner the sole way of mining Grin. This removes a lot of cruft from the code and simplifies configuration options and the build itself. The only mining code left in Grin is an internal Rust-implemented test miner, which is only intended to run tests for Automated testing mode. Again, a few more tweaks are needed here to ensure developers can still build and test Cuckoo16 chains in a reasonable fashion as well as some other things around genesis block creation.

I’m hoping to have this all deployed on the master branch some time next week, but even without the remaining fixes I think grin-miner is already more flexible and useful than the current in-process mining, (where you have absolutely no control over the miners once they’ve started and can’t tweak them without taking the node down). Also, you can mine as many instances of grin-miner as you like into a single grin wallet/node, which I think is very desirable and paves the way to facilitate pooled mining!

And in other good news the funding campaign target has been reached, which should ensure my continued focus on the project until at least September! Thanks to everyone who made this happen, and I’ll be officially closing the campaign shortly and sending out proper thanks and acknowledgements via the Mailing list shortly.

Now I think I’m going to go celebrate

5 Likes
16

Update Friday 27th April, 2018

Testnet2 seems to be ticking along nicely with a relatively small number of minor known issues, but nowhere near the mad forking behaviour we had with testnet1. It would be nice to see a bit more mining power on the network, but I guess most miners won’t want to be pointing their expensive and noisy farms at our worthless test coins! At any rate, tell your friends about Grin!

What I looked at this week:

  • grin-miner is released and in the wild… a couple of fixes were made here and there post-release but in general it seems to be working. There are may optimisations and tweaks to be done, but those will come with time and testing.

  • A couple of day spent fixing up wallet restore to work with the latest iteration of the code, mostly to ensure that restore works properly against a non-archival (i.e. fast-synced) node. The wallet restore now scans the UTXO set (which all nodes have) instead of block history (which only archival nodes have), and recreates a merkle proof for each coinbase output. The only downside is that coinbase outputs need to be re-locked for 1000 blocks (as if they’ve just been mined,) from the time of wallet restoration. There are a few ways we can optimise this, but they get a little bit complicated so I thought it best to just get things working and we can come back and tweak later… waiting a day or so to spend coinbase outputs after restoring the wallet shouldn’t be the end of the world.

  • And then a little while spent looking at issues and fixes to both of the above… I think everyone who wants to mine with grin-miner is able to do so, and the people who had issues with wallet restore (including one wallet with about 9000 coinbase outputs) are able to restore properly.

Coming up I think we’re going to have to have a little discussion about plans for testnet3, as I think there are a few issues that need to be addressed that would be very difficult to manage with a hard fork… namely I think we’ve backed ourselves into a corner a little bit with the PMMR data format and we need to be able to accommodate hard-forked changes in Output, kernal and proof data sizes without making a mess. Also, the latest version of @apoelstra’s bullet proof implementation is incompatible with our testnet2 version (which was a very early integration effort,) and there are one or two other consensus breaking issues around. On the plus side, if we did want to do a testnet3 I think there are only a few very specific development tasks we’d want in it, so it would be quite well defined. On the downside, we really should be able to hard-fork rather than restart the chain from scratch on consensus breaking changes, and we need to practice hard-forks.

Also… we really need to fixate on the wallet at some point… though this might be easier once all of the above has been more or less dealt with and the chain itself is solidified and in pure testing mode.

Right, back to the coal face for a bit then I think it’s vodka night.

4 Likes
17

How many testnets are there going to be?

18

As many as needed.

1 Like
19

I can confirm that the wallet is extremely unstable at the moment. Wallet crashes pretty quickly when stress tested. Definitely needs some love.