Yeastplume - Progress update thread - May - Sept 2018

[Yeastplume]

Update Friday, May 18th, 2018

Theme is the same as last week, plenty of work continuing on the wallet refactoring and library creation:

• https://github.com/mimblewimble/grin/pull/1061 was integrated earlier in the week, and went a long way towards putting all of the bits of functionality that comprise a wallet into the right places. There’s a full description in the issue of what changed, so I won’t repeat it here.

• With that in place, I’ve moved on to part 3, and spending time really refining what the transaction workflow should look like. This phase is about chiselling away at the API to try to get it into the most flexible and re-usable state possible, and I’m starting to like the results… the form it’s moving toward should make it far easier to support all of the transaction workflows we’ve talked about (including carrier pigeon,) as well as support future models such as multi-party, multi-sig, etc. Again, there’s a lot more detail in the issue, so I invite you to look there if you want to know further details.

Once thing I’ve done is started using the term transaction ‘Slate’ to describe something that holds all of the semi-public data needed to complete a transaction. The idea here is the slate is passed around between all parties involved, each one filling in their information until there’s enough info to finalise the transaction. How it’s passed around doesn’t matter (http or avian carrier), just that everyone adds their inputs, outputs and information correctly during the appropriate round. Once all parties have completed their rounds, anyone can finalise the transaction and post it. The slate doesn’t care who the sender or receiver is, or indeed how many parties are involved. In any case, I think using a term other than ‘Transaction’ or ‘PartialTransaction’ to describe this chunk of data makes for less ambiguity and will hopefully make future discussions clearer.

Still a lot of work to go, but it’s starting to go faster and easier as the pieces move into place and the remaining tasks become more obvious. Once I’m happy with how the aggsig workflow looks, I’ll move on to tightening up the other pieces and hopefully we’ll have a decent wallet library foundation to build on.

One last thing, if you’re in the London area and haven’t signed up for this yet, what’s wrong with you?

Grin & MimbleWimble Meetup in London with Yeastplume

Look forward to seeing everyone there next week!

[Yeastplume]

Update for Friday May 25th, 2018

More screw-tightening on the wallet for most of this week, and with the lastest PR I think I can say that the wallet refactoring is mostly done for now, or at least to the point where I think other bits of work can now start. The transaction workflow I think is is as optimised as it can get for the time being, and any changes or refinements will most likely be smaller changes coming out of the next phases of wallet work.

There’s a been a bit of chat on Gitter at what to do next, which will probably include moving the wallet storage from a flat file into lmdb, possibly some related work on the keychain, and creation of all the other bits up to and including a much more usable wallet. I still don’t think we have quite a complete picture of what should eventually be released with Grin, so I’ve started to put some thought into it and got down at least one UML diagram to start the discussion. I’ve just pasted what I have so far into the Gitter dev channel for the time being for discussion, and I’ll put it somewhere more permanent once we have some agreement on what we’re doing. In the meantime, just take a look on the dev channel for the ongoing discussions.

And while you’re there, I’d also encourage you to look though the recent discussions on dandelion and aggregation… I’m not directly involved it at the moment but @antioch and @quentinlesceller are, and it’s interesting stuff… I have no doubt they’ll arrive at the best solution.

The meet in London was a success, I think… I enjoyed myself anyhow and it was great meeting so many people interested in Grin! I look forward to getting out to a few more meetups and conferences eventually, (as and when it makes sense, of course, given the amount of work there’s still left to do!) Thanks to everyone who came out!

[Yeastplume]

Update for Friday, June 1st 2018

At the risk of getting boring here, it was mostly even more wallet refactoring this week, via https://github.com/mimblewimble/grin/pull/1113 and https://github.com/mimblewimble/grin/pull/1096… In short:

• the wallet storage was abstracted away, meaning that it should be very easy now to drop in another backend (as opposed to a simple file as exists now). @igno.peverell is working away now on LMDB storage for consensus breaking branch, and the plan is to add an LMDB based wallet, which should hopefully operate more quickly and allow for more flexibility.
• There are 2 wallet libraries now, libtx which handles transaction building, and libwallet, which calls into libtx and adds functionality like selecting, querying and updating wallet outputs and keys (with no pre-knowledge of the underyling storage format).
• Most of the common/reusable functionality has been factored into libwallet, leaving the main wallet implementation (called FileWallet now) with manageable amount of code.
• The other periphery bits around the wallet, running a server, invocation etc have been reduced to a near trivial amount of code.
• Error handling reworked a bit to ensure we’re capturing errors from all the right places

Now, the current state of things isn’t perfect… we may still want to fiddle around with the expected ownership of trait objects and at some point factor our traits from the Keychain, but I think it’s in a state now where we can start attempting to build a few more things on top of it, and continue to refine and refactor based on progress there.

As I mentioned, Igno is planning on LMDBifying the backend storage. I think for my next trick, I’m going to try to start building out the skeleton of the structure planned here, and get some wallet API calls working end-to-end for both the HTTP and command line cases:

It’s been a lot of internal refactoring and foundation work on the wallet thus far, but hopefully soon we’ll start getting to the user-visible enhancements and improvements.

Have a good weekend all!

[Yeastplume]

Update Sunday, June 10th, 2018

Bit late on this one as I was too deep into nasty web UI-type code on Friday, but here’s a quick update on last week and plans for this week.

As mentioned last week, I’ve factored out communication from the wallet code as well as created separate listeners for what I’m calling the “owner” and “foreign” api interfaces, which are intended to provide rest apis for the wallet owner and external senders respectively. @igno.peverell has jumped into the wallet fray by factoring out the Keychain into a trait and is now working on an LMDB version of the wallet backend, which should hopefully be faster and more flexible than the current file-based implementation.

Up this week, first I’m going to spend a bit of time integrating Tromp’s latest ‘meaner’ miner into grin-miner, to try and get the core Grin mining experience up to the speed of the GrinGoldMiner. Then, continuing work on a bare-bones web wallet that uses the wallet owner interface. There’s a bit of work already done but it’s nowhere near ready to show. I’ll be creating a new repository soon to keep it all in, but shan’t be providing instructions to build and run it until it’s in a ready-ish state. So there.

And just as an aside, if there are any web frontend experts looking to get involved in Grin’s development (I’m using Angular for the time being), now would be a great time to jump in!

Will provide a lengthier update and hopefully a thing or two to demonstrate at the end of the week!

[cilphex]

I’d potentially be interested in contributing to the web wallet. How can I take a look and get started?

[Yeastplume]

@cilphex I’ll have a repository up within a few days containing what I have so far. Will give more detail when it’s there.

[Yeastplume]

Update Friday June 5th, 2018

So without further ado, some more tangible and user-friendly results from all of this wallet work:

That’s a rough fist pass at Grin’s web-wallet, which can be viewed, cloned, downloaded, played with and PRed here: https://github.com/mimblewimble/grin-web-wallet. It’s done using Angular 6 and Bootstrap, and I’m intentionally trying to keep the number of dependencies to just that. I’m a fan of using Angular for any client side browser development, and the really nice thing about recent versions is that it contains decent tools to compile a massive project full of typescript files to a few minimised and uglified .js files that can easily be embedded and served from within Grin itself, if so desired.

For now, no secret information is passed to the client. All secrets remain on the ‘server’ (which is your local machine running your wallet,) and processing happens there. The architecture should stay this way for the forseeable future, but I may make a case for finding a solution for sending passwords back and forth as this will give us the ability to manage multiple wallets through the single application, which I think would be a nice usability feature and make it feel like other wallets.

Server side, I’ve had quite a few PRs this week to support web-wallet functions, the most major being the addition of a wallet detail file which contains information such as the last assigned key and the height at which the wallet was updated. Previously this was being inferred by the wallet output contents, however that turned out to be very irksome for a client interface where you’d like to keep a consistent view of the data, and want to be able to retrieve your wallet without necessarily wanting to update all of your outputs against a server every time.

As it stands, the web wallet does more or less exactly what the command line wallet does, and a few recent fixes and improvements I’ve done automatically apply to both, as they’re both using the exact same interface and sharing most of their code (one of the many reasons behind the past few weeks of wallet library refactoring and creation). It’s still very rough and doesn’t necessarily do all of the error checking and user-friendly hand-holding that it should eventually do, but it works and can even be used to send at the moment:

So, now this is in place, all I can think of is the massive amount of work still remaining to make the wallet as robust and usable as possible… Off the top of my head, the order of events should be:

In the immediate future:

• Finishing the LMDB backend for the wallet, which should make operations faster and more robust than the current simple file version. @igno.peverell is working away on that
• Replacing the web-server code… Grin is currently using Iron to serve files for both the wallet and the API, which is basically abandonware at this point. @hashmap is looking at replacing this with Hyper+an http routing solution… I hope to be serving up the wallet as well when grin is run with defaults. (localhost only for now)
• Adding a transaction log, which will be completely supplemental to the wallet data but give the user the option to record what has been sent and received (like a traditional wallet). I think this is really necessary to make the wallet feel familiar and more usable.
• Figure out how the compiled application is going to be served.
• Continued development on and improvements to the web wallet app based on all of the above.
• Wallet restoration based on the committed elements version of bulletproofs

And importantly:

• Defining the wallet security model and implementing it. A very large topic in and of itself. We’ve had some thoughts on this and will most likely be addressing this after T3 is launched and underway.

And longer term adding features that have been talked about but not really addressed, multi-party, multi-sig, vaults and all of those good things we think we can build on top of MW/Grin.

And all of this is on top of a load of other consensus-breaking work planned before T3, so a lovely fire-hose of work for the foreseeable future.

By the way, to try the web-wallet (recommended for technical-types only ATM):

• Run grin node as usual
• Run grin wallet owner_api in whatever directory you want the web wallet to be acting upon.
• Follow the build and run instructions for the web application here. You’ll need npm installed, but other than that the build process should get all of the dependencies.
• After running ng serve you should be able to get to the app in a browser at http://127.0.0.1:13420

I’ll put more detailed instructions up when it’s further along and we’ve figured out how it’ll be packaged with Grin.

Have a good weekend all!

[Yeastplume]

Update Friday June 22nd, 2018

Very quick update this week… since we’ve decided to branch away testnet3, we’re undergoing a pile of merge-pain with plenty still ahead of us as we try and merge all of the consensus-breaking PRs into the branch that will become testnet3. These contain all sorts of good stuff, many fixes as well as a switch to LMDB for the chain and wallet backends, but are going to take some time for everyone to merge properly and test.

I won’t go into too much detail on particulars I’ve done because it’s just PR merging (which can get very tedious,) but it doesn’t make a lot of sense to start anything new or continue a lot of of the things I have planned until all of this is done (so we have a stable branch to work against,) so I’ll most likely be focused on that until we have a stable testnet3 branch. You can see the branch and progress here (which I don’t recommend trying to run just yet): https://github.com/mimblewimble/grin/tree/milestone/testnet3

Hopefully within a week or so we’ll have that branch more or less together and then everyone can start bombarding us with questions like ‘when t3’?

[Yeastplume]

Update Friday 29th June, 2018

Testnet3 is looking closer than expected, with only a few consensus-breaking items outstanding (and of course plenty of developer testing before we release it into the wild). Most of the work this week was on various pre-T3 items, in no particular order:

• https://github.com/mimblewimble/grin/pull/1208 Update to error handling, particularly in the chain module… we’re using a particular error strategy within the wallet crate which should work well for error handling needs, however it needs to be implemented throughout the rest of the code. This is a start anyhow.

• Testing the t3 wallet and transaction code, and a few fixes based on that: https://github.com/mimblewimble/grin/pull/1204

• Latest (reviewed and hopefully near final) bulletproof code implemented within T3: https://github.com/mimblewimble/grin/pull/1194

• Some direct commits to the T3 branch to get it stable and compiling earlier in the week (Merging some earlier messy PRs

As well as some maintenance on our issues list which I think had gotten a bit flabby. There were quite a few out-of-date or otherwise resolved issues in there, but as of now, all of the issues in there are relevant and current (as far as I can tell)

I’d say at some point mid-next week or so we’ll be able to start developer testing on T3 to ensure it’s stable enough for general consumption. More to come on that, probably in our Dev meeting on Tuesday evening.

Have a good weekend, and remember to use sunscreen, kids!

[Yeastplume]

Update Friday July 6th, 2018

For what is hopefully the last time, we’ve just launched our traditional pre-testnet developer network through which we’ll try to convince ourselves that nothing is horribly broken in advance of Testnet3. At a guess, T3 represents about a 30% rewrite from Testnet2, which is less code changed than the T1 to T2 jump, but still means it contains tons of new and changed code that needs to be tested in the wild again.

Off the top of my head and in no particular order, the changes and enhancements between T2 in T3 are (some of these are implemented or partially implemented in T2 as well:

• PMMR improvements and enhancements (storing PMMR markers in headers, using a bitmap for pruning and rewinding)
• Merkle Proofs no longer needed to spend coinbase outputs
• Kernel history verification on fast-sync (to address a security issue)
• Newer version of bulletproofs, with features included to facilitate wallet recovery
• Chain storage migrated to LMDB from RocksDB
• Variable cuckoo size POW, with miners able to submit cuckoo solutions from size 30 upwards as opposed to just 30.
• Error handling improvements
• Improvements to difficulty adjustment, remove ‘double hash’ from cuckoo cycle input
• Changes to report sync state to the TUI
• Many improvements and enhancements to the wallet library to pave the way for future wallet enhancements

And I’m sure a few others I’m forgetting (perhaps I can update this list as people remind me)

The past week has been mostly about testing the various parts of the system (particularly consensus breaking issues) and fixing issues as and when they arise. Everyone has done bits of work on various issues as they’ve come up, but I’ve mostly focused on making sure mining works properly and ensuring the difficulty adjustment algorithm works well enough though some scenario modelling tests. Quite a few fixes to mining and a few tweaks made to difficulty adjustment as a result.

In any case, I’m now fairly comfortable saying that T3 is imminent. Hopefully this is the last distinct testnet, and (hopefully again) any consensus-breaking issues that come up from now should be addressed via forking rather than a completely new chain. At some point, we might even be able to call this chain a Beta.

Enjoy weekend, and Grin hard.

[Yeastplume]

Update Friday July 13th, 2018

Testnet 1 was described as ‘bumpy’ by @igno.peverell in one article or another, a nice media-friendly way of presenting its tendency to fall apart at the seams. Testnet2 held up much better due to all of the fixes and learning from Testnet1, and allowed us to focus on a whole load of vital (but consensus-breaking) improvements that became evident after observing a functioning blockchain in the wild.

I have to say that Testnet 3, launched last week, feels very good so far. There doesn’t appear to be any massively odd strangeness in the chain itself or any blocks being accepted that shouldn’t be (yet,) and most of the issues we’ve seen are either user-errors (understandable due to the fast-moving nature of the code) or due to the immaturity of the supporting bits (miner, wallet, etc which themselves are holding up quite well but still need more time for feature-completeness and robustness). Signs are looking good for this chain lasting a while and possibly becoming the official development/testing chain instead of retired on the launch of Mainnet.

So, with the chain underway and issue investigations ongoing as required, there’s time to start focusing on the periphery bits again… so this week’s work mostly consisted of:

• Wallet API Client trait refactor … which just means abstracting some code to allow for different communication implementations between wallets. The default http wallet-to-wallet code can now easily be replaced with other more robust implementations that use different means of communication, and paved the way for the …

• Wallet API Test Client … by creating an implementation of the above WalletClient that communicates with a local instance of a chain as well as in-process test wallets, I’ve created a framework that allows for more robust and accurate wallet testing, which then allowed me to confidently build tests to ensure that the transaction API works and is properly covered by automated testing, which then made it easy to finish and verify igno’s work on:

• The LMDB Wallet Backend, which implements alternate storage for the wallet in an LMDB database instead of flat files. Right now it’s just at parity with the File Backend functionality-wise, but more is planned and it should become the default. (instructions on how to start using it are found in that issue.

So, with all of that done, the wallet code is in a good place to begin the next trick, storing transaction metadata within the wallet, which should allow all sorts of good things like showing activity of funds going in and out of the wallet, as well as finally allowing you to cancel transactions and un-lock previously locked outputs if it’s become evident that a particular transaction isn’t going to hit the chain. There’s still plenty more to do beyond this around security, deployment, and the web wallet, (along with more tests and generally ironing out all the bugs,) I think this is the last missing piece with respect to the wallet’s usability. Looking very forward to getting it in there.

Have a good weekend, and kudos to the rest of the team for their really impressive work so far!

[Yeastplume]

Update Friday July 20th, 2018

This week was about adding more feature completeness to the wallet, and this roughly went as follows:

• I forcefully shoved everyone onto using the LMDB Backend for their wallets via a small and automatic migration feature. It seems to have gone smoothly enough and I haven’t heard too many complaints, so let’s just assume all is going well there and everyone is now using a more robust and easier-to-develop-for LMDB backend for their wallets.

• The Transaction Log has been added and appears to be more or less working in the initial instance.

With all of these changes in place and all of the previous refactoring work, it’s past time to update all of the wallet-related documentation, which I intend to address at some point next week. In the meantime however, I’ll give a bit of a tour of the new Transaction Log and why I thought it was very important to get this in place.

First, every wallet operation that adds or spends outputs (i.e. all of them) now logs the event in the transaction log. You can get a dump of the entire transaction log with the command:

grin wallet txn

The output of this command looks something like this:

As you can see here, each entry logs the time, amounts debited, amounts credited, number of associated outputs, and other information which makes it much easier to see what’s actually been happening with your wallet.

The above entries are the result of mining a new chain for a few blocks, then sending a transaction for 60 grins into the same wallet. You can see there that there are 2 resulting log entries, 6 and 7, that logged the creation of the transaction (regardless of whether it was actually posted to the chain, hence the status ‘unconfirmed’). If I want to dig a bit deeper into what’s going on with a particular transaction, I can now go, for instance:

grin wallet txn -i 7

Which shows a single transaction along with all of the related outputs:

In this case, you can see that outputs were locked and added to the transaction and a change output was created. They’re all at unconfirmed, as you can see, as the transaction hasn’t been posted to the chain and mined yet. Let’s wait a little bit and run the same command again:

Whoops, something went wrong here and the transaction wasn’t posted to the chain for some reason or another. Maybe the sender didn’t send it, there was a communication error or it’s just igno’s fault. In the pre-tx log days, this meant either going through your wallet data manually reverting the status of each locked output, or restoring the wallet, however, with the TX log I can now reverse the transaction out of my wallet by running:

grin wallet cancel -i 7

and check again with

grin wallet txn -i 7

Lo and behold, the unconfirmed change output is gone, and my previously locked inputs are now available to the wallet for spending again.

Now of course, this is the first pass and we have some eventualities to think about, particularly around not making a mess of someone’s wallet if someone does go and post the transaction after the receiver has cancelled the associated output. (On the sender side, a transaction won’t be valid if it’s posted after they’ve gone and spent associated outputs, but the receiver will have the output missing). We’ll deal with that in time, but at least going foward everyone should now have the transaction data accumulating in their wallets to deal with these sticky situations and reduce the need for wallet restores on every send failure.

Also, it’s much easier now to produce a usable web wallet with this in place, as it should be able to show a usable transaction log of what when in and out and when. You might point out that every other crypto wallet can already do this, but with MimbleWimble this doesn’t come for free; most other (BTC and clones) chain formats out there allow wallets to deduce this info from the chain, but since MW transactions are a pile of anonymous outputs, wallets need to keep this (very private) information separately. It will be a bit of a challenge ensuring this information is kept consistent on all sides, but very doable.

Right, that’s it for now. I’ll get fuller documentation up next week and hopefully make some progress on other smaller issues starting to pile up as well as another round on the default web client itself.

Enjoy weekend.

[Yeastplume]

Update Monday July 30th, 2018

Apologies for the late update here, but was dealing with some family matters.

A couple of things done last week:

• updates to cuckoo-miner / grin-miner to incorporate @tromp’s latest solvers, (which now allow cards with much less RAM to be used effectively for mining Cuckoo)
• Another round of changes and additions to the web-wallet to refactor, make it work with the latest T3 as well as incorporate the transaction log. A Picture or two is worth 1000 (or 2) words here, so without further ado, this:

is how the main summary screen looks now and this:

Is what the transaction detail looks like, also showing associated inputs/outputs. I don’t yet have the ability to cancel/rollback transactions in there, but it’s trivial to implement.

As always you can take a look and try to get it going via the web wallet project page itself, but truth of the matter is this thing isn’t going to see any useful exposure until we figure out how to package it within grin so it’s actually available to anyone running grin without having to install a load of unrelated webby-type packages.

Before that can be addressed though, the large and messy effort to replace the no-longer-maintained web service framework Iron with something more up to date has to be integrated, and is being done by @hashmap and co here: https://github.com/mimblewimble/grin/pull/1241. Once this is in place and verified, hopefully including the wallet client by default should just be a matter of serving up some static files from grin locally. I’ll be looking at that PR next, reviewing and testing locally before we integrate it.

Also, as a bonus addition there’s a large round of documentation for the design and usage of the wallet, complete with pretty pictures going in any second now. Once that’s merged (basically now). The docs/wallet directory in grin should contain up to date design info and a bit of a user guide on the current commands available in the wallet.

That’s it for now. I’ll be on holidays with Yeastlings from about the middle of this week for a couple of weeks (approx Aug 5th-19th,) however I’m not intending to be completely absent… will be (mostly) available on Gitter as usual, and might pick up the odd bug here and there for annoying things I haven’t got around to fixing, (definitely won’t be undertaking large chunks of work until I get back). Hopefully this will allow everyone else to catch up with their PRs without me continually rewriting the wallet code base.

Next update will be around Aug 20th, hopefully (mostly) refreshed and ready for another fast-paced round of Grin.

[Yeastplume]

Update Saturday August 18, 2018

Back from herding a bunch of recalcitrant Yeastlings around North Rhine-Westphalia and back into full Grin swing from about Wednesday onwards. The project had been a bit quiet (in summer holiday mode) up until about 10 minutes after I got on a plane, at which point activity exploded to a point that we’ve never seen before. The number of people actively working on or somehow involved with Grin at the moment is at a high and continually growing, which is great to see.

For my own part, the past week’s work included:

• Updates, reviews and re-jigging the rust-secp256k fork to faciliate @garyyu’s work on finishing the integration of batch bullet proof verification: https://github.com/mimblewimble/grin/pull/1363. Fast sync should be greatly sped-up as a result.

• Integrating latest solver work into grin-miner and a few updates and changes there to build Cuckoo31 solvers and allow grin-miner to run them. I haven’t manged to get any of them working yet, but I didn’t spend a lot of time on them other than get them building and runnable… hopefully the community will give them a try and find out what needs to be tweaked in order to get some Cuckoo31 blocks onto T3.

• Getting the web wallet application intergrated into grin … There’s now a half-decent build process that puts tagged versions of the grin web wallet into grin, and another bit of glue within grin to serve up the application… to get it working in the latest grin code, change into the desired wallet directory and run:

grin wallet web

This should start up the embedded application server and the wallet API needed to service it. Note that I haven’t looked at the actual application itself for a bit (and it’s very alpha), but hopefully we’ll be able to give it some more direction now that a major barrier to people actually trying it has been removed.

That’s it for now… the next chunks of work should be about bug fixing, refining, improving. In the absence of anything else more pressing coming up, I’m probably going to start next week with another round on the wallet restore, and possibly start thinking about wallet-to-wallet communication and security issues (e.g, right now it’s trivial to DOS a wallet with dummy transactions). I have a few other things in mind as well, but will talk about them as and when I get to them.

Enjoy rest of the weekend!

[Yeastplume]

Update for Friday August 24th, 2018

The burst of activity continues and loads of smaller things tackled this week (seems to be the mode all developers are in at the moment). The core chain code is looking more robust by the week, lots of speed and other improvements being done and quite a few smaller quality of life things are being addressed. There’s a Work-in-Progress (and still under revision) list of things that need to be in place for MainNet: https://github.com/mimblewimble/docs/wiki/Functionality-for-Mainnet, and the next few weeks/months will most definitely be focused on ensuring all of the items on this list are addressed.

As for my own work this week (in no particular order):

Two major(ish) issues with the wallet fixed; the fact that the reported height of outputs in the wallet was not always reliable, and that wallet restore hadn’t been updated. These were addressed via:

• Adding functionality to retrieve a block header by an unspent output commit … this builds on work done earlier to include a PMMR index in each block, which makes it possible to search through the header chain to find the block an arbitrary unspent output is contained in. This in turn makes it possible to get the height, which means the height of an output in the wallet can be refreshed with the correct values after dandelion propagation or a wallet restore (both lacking earlier). I’ve tried to do this as efficiently and simply as possible, though there is a chance this may need to be revisited later depending on the performance of large wallets.

• Ensure wallet heights are always correct - Just using the above functionality to ensure the heights in a wallet are refreshed correctly once they appear in the UTXO set.

• Wallet Restore fixes + updates In addition to including the above, a few changes and updates needed to be made to the entire process to support recent developments (the code was a little bit out of date). Most importantly, I added some automated tests to test the process at a basic level and ensure the contents of both pre and post-restore wallets match exactly (which should help ensure that bit of code doesn’t fall out of date again)

And a few other non-wallet issues

• Automated test for 1325 There’s some particular oddness with larger transactions that everyone’s been looking at nobody’s yet been able to track down… I’ve jumped in a bit as well by creating a test that tries to replicate the issues… (and to nobody’s surprise, the test seems to work fine) Whether or not this ends up being useful to anyone remains to be seen, but that’s the nature of the business.

• Batch Verify proofs during validation a small update but also did a bit of benchmarking to ensure batch verification is indeed faster in all cases.

And finally, a bit of a wallet thinkabout, particularly on the subject of how wallets can communicate easily and safely (as opposed to the NAT-negotiation hell the process currently entails) and catching up on details of some technologies that could provide solutions. This is still being thought about and under active discussion, so I won’t go into too much detail at the moment (but you can see most of the discussion to date on the dev gitter channel).

Of course, you didn’t think you were going to get away this week without any mentiong of my current funding campaign and thanks to the few kind souls who’ve started it moving already! Takeup hasn’t been as definite or as quick as the previous ones, but I think we can probably put that down to the fact that it’s the tail end of the summer season. I hope the community will see fit to re-hire me for the coming few months once everyone is back at their desks!

That’s it for now, enjoy the weekend dreaming of a rack stuffed with new 2080ti’s.

[Yeastplume]

Quite a lot done this week (between endless PoW discussions), but all very easy to describe, so won’t be a long one. Overall, very much trying to stick to the issues labelled ‘must have’…

• Tons of ‘must haves’ that aren’t very entertaining or glamorous, but somebody’s gotta do em. In that spirit, a good chunk of the week was taken up with Config + Default directories by yeastplume · Pull Request #1433 · mimblewimble/grin · GitHub, which just deals with grin’s configuration and default path setup. I changed around paths to a more final looking layout, and added the ability for grin to spit out its configuration files, which make them much easier to manage, create, and split up as necessary (there are now separate wallet and server configuration files as a result).

• I did some digging into handling the case of a full transaction pool talked about in Evict transactions from the pool by order of fee/weight · Issue #1417 · mimblewimble/grin · GitHub… This was mostly for my own understanding and I’ll be going back to it later

• And moved straight onto a more interesting ‘must-have’, BIP-32 support for the wallet which basically allows a wallet to have several child derivation chains as opposed to the single one currently found in the wallet (which will make wallet restore that much more interesting ) Still in the middle of this one

That’s all for now, plans for the coming weeks are to pick through the launch ‘must_haves’, and hopefully the rate of their resolutions will soon outweigh the rate of their creation!

Enjoy weekend, and don’t do anything I wouldn’t do. (Which leaves you with a very wide range of questionable things you can do.)

[Yeastplume]

Update Friday, Sept 7th 2018

Well, this has been a strange 24 hours. About this time yesterday I wasn’t quite sure I was going to be ‘employed’ beyond the end of the month, and now we’re sitting here with the funding goal pretty much achieved! I can now comfortably say I’ll be able to continue full-time on Grin through the end of February 2019. By that time, I will have been community-funded for 13 months (and we might even be somewhere near a mainnet launch).

I’ve been really humbled by the reaction of the community to these past few funding drives, and hope I can live up to everyone’s expectations. I posted my thoughts about the greater Grin community (for unrelated reasons), but much of what is in there directly applies to my current sentiments.

I’ll shout out to everyone who’s donated when I close this campaign, but right now I need to again thank:

@catnic For leading the charge, for a generous donation, and (via BlockCypher’s newly announced Grinmint Mining Pool) backing up her belief in Grin by spending time and resources adding to its infrastructure.

@awrelllRo for an incredible $40,000 donation to meet the funding goal in one massive swoop. This is definitely the single most dramatic gesture we’ve seen in our fundraising adventures so far, and we’re all very, very grateful for this.

Will probably close the campaign successfully shortly, but if anyone is still itching to donate to Grin right this moment, please feel free to do so directly into the fund. I intend to transfer anything above the current funding goal into the the general Grin fund we’ve been talking about as soon as the logistics of it are set up. (Or you can wait until then, it’s all good)

By the way, I’ve seen some kind words in relation to my work, but I’m neither a wizard nor a rock star. There are better developers on the Grin team than I, some part-time and some who have found other ways of funding their Grin development addictions. I’m just the one who’s been visible and in people’s faces for funding, and I really need to thank everyone else on the team for allowing me to continue these efforts.

---

But also: Right. Enough Oscar speeches. Been distracted by fundraising and other issues the past couple of days, but this isn’t what I’m being paid to do. As a special bonus this week, let’s talk development.

So we decided last week that there will be a Testnet4, with a very short list of consensus breaking isses this time. Mostly aggsig changes (one bug, several enhancements to support higher level constructs), mainly dual proof-of-work changes (which will mean Grin-miner and cuckoo-miner work in the coming weeks) and I’d also like to squeeze in BIP-32 Compatibility in the wallet. I’m addressing the BIP32 work first, so that entails:

• integrating the BIP-32 library from rust-bitcoin into Grin, a necessary first step completed earlier this week.

• Changes to the underlying libsecp256k library to support this. There are quite a few things that need to go into our libsecpetc fork, with a lot of work done by @jaspervdm and @gary to fix and refine our aggsig library. In addition, there are some modifications that both @jaspervdm and I want to do to essentially embed values within bulletproofs (in his case, allow for multi-party bullet proofs, and in my case embed BIP32 derivation indices to allow BIP32 wallets to be quickly and easily restored). So I’ve done a rebase of our fork from the latest version of Andrew’s code, know more or less what I need to do, and will combine it with the results of @jaspervdm’s changes. I’m not finished with my part yet, but it’s next on my list.

I’d like to get all of our combined changes to libsecp256k put to bed for the time being, and once that’s done, the focus will be to update the wallet code to actually use BIP32 keys (thinking of the number of annoying tiny changes and broken tests that will entail is already giving me a headache,) but once that’s done our wallet will be more-or-less compliant with existing software tooling, as well as be a lot more flexible about how keys are derived, hopefully giving us some more tools to think of creative wallet solutions.

Also on my radar is the dual-PoW, but @tromp still needs to do some work there before that can really begin to be integrated (which is fine as the BIP32 work will still take a while).

So that’s it for now. Everyone enjoy your weekend (I definitely know I’ll be a bit more relaxed,) and don’t forget to ring your mother.

[Yeastplume]

Update for 14th Sept 2018

So we had our first practice hard-fork on T3, which was simply a test that changed a few header fields around. Other than a small mining hiccup, it all seems to have gone well.

All the required work for the upcoming Testnet4 is next, which involves updates, fixes and changes for aggsig, bulletproofs, wallet BIP32 support and dual POW. Because of the way libsecp is structured and tagged, we’re likely going to end up with a couple of massive Franken-PRs that we’ll merge into a test T4 branch shortly before we launch it. Most of my work this week has been on just such a PR at https://github.com/mimblewimble/grin/pull/1501, which I’ll explain in a moment.

As I’ve mentioned before My main focus right now is getting BIP32 support into the wallet, which from an end-user perspective means that instead of having a single wallet with a single key derivation path, a wallet will be able to support multiple ‘accounts’ in a branching tree structure. So that involved:

• libsecp256k work, mostly to support the migration of wallet key derivation from our existing setup to BIP 32. Most notably, I’ve inserted an optional 16 bytes worth of message into to a bullet proof. This will allow wallet restore from a seed by inserting and restoring BIP32 derivation paths into the bullet proof, (an impossiblity otherwise, as there would have been too many key paths to grind through to try to find the right key derivation index during a wallet restore.) This will also solve a long-standing annoyance in wallet restore. Since the key index is stored the wallet restore no longer has to try and find the right derivation for each output it identifies in the UTXO set, which will greatly reduce the amount of time it takes to restore a wallet.

• The main PR integrating this into Grin, building on last week’s work merging in the BIP32 code from rust-bitcoin. This is all mostly working now with tests exercising multiple accounts in a wallet, and I’ll soon be adding a ‘–account’ flag to the wallet command line as well as an ‘account create’ command of some description (which just stores a map of labels to BIP32 paths). This is going to end up including all of the latest libsecp changes as well as I need to update the linked libsecp version to support bulletproof messages as described above. But that’s also going to include the aggsig changes to support atomic swaps, and other bulletproof changes to support multi-party proofs. So we’ll probably end up with 2 large PRs to merge for T4, this one and one that includes the planned dual-PoW work.

Next couple of days are going to be about, (roughly):

• updating grin to work with the newest version of our libsecp256k library (the interfaces will have changed, and the aggsig signature message needs to change)
• integrating BIP32 identifiers into the bulletproof message in grin
• fixing/updating wallet restore to use these identifiers
• Updating the command-line wallet commands to handle multiple accounts
• merging some preliminary BIP39 work @mcdallas (though will probably wait until after T4 to integrate this into the user experience)

And then I’ll pile onto the needed updates to grin/cuckoo-miner for the dual PoW changes, depending on where we are when I’m done with BIP32 changes.

And that’s just me. The amount of work being done on Grin seems to be accelerating, and there’s plenty on everyone’s plate for the foreseeable future.

Have a good weekend, and to everyone working on Grin: Remember to have a weekend.

[Yeastplume]

Update Friday Sept 21st, 2018

Going to be a very short one today, as all of the detailed information on what’s been happening is contained within the links.

• The bulk of the week was taken up with finishing The Mega-PR for T4; basically finishing off the list of task I outlined in the post above. It’s ready to go, but is fairly consensus-smashing so needs to wait until we’re just about ready for T4. In the meantime, I need to keep it updated with the ongoing updates to master to keep it merge-able.

• And the rest of it was moving straight into the other major bit of work needed for T4, Cuckatoo PoW changes. We need a few things here, namely validation, a way to mine it, and a test miner (to replace the plain-cuckoo one we have for development. So to that end I’ve done some work on a simple rust cuckoo miner based on @tromp’s original C code. It’s mostly there, but still a few issues to iron out. Once that’s working and the changes to validation is in place, I can probably a second large PR to change the PoW to cuckatoo based on the groundwork @igno.peverell has already done in https://github.com/mimblewimble/grin/pull/1534.

So that will be the focus for the next few days of work, as well as extending cuckoo-miner and grin-miner to incorporate cuckatoo solvers. Fortunately there should be no changes to any interfaces as a result (inputs to and solution outputs from cuckoo are the same,) so it should really be a matter of turning the cuckatoo solvers into grin-miner compatible plugins.

So with some luck, we might be within reach of T4 over the next week or so (even if we’ll only be able to CPU mine it in the beginning).

That’s it for now. Time for a weekend.

[Yeastplume]

Update for Friday Sept 28th 2018 -

And going to be the last update for this thread, cause I’ll be starting a new one for the October-February “Contract”).

So aside from the odd segue to keep the T4 Mega-PR up to date, most of this week has been about trying to get a validator and test miner in place to support the upcoming cuckatoo algorithm. I’ll leave any discussions about mining and PoW plans in general for another (more heated) thread, but all I’ve been doing at the moment is trying to put supporting code in place to cover all cuckoo/cuckatoo possibilities.

So, without any ado:

• The Cuckatoo PR puts support for Cuckatoo in place and refactors the Cuckoo verifier and test miner somewhat to allow for easier switching of algorithms as needed. I’ve keep the actual choice of miner compatible with T3 to avoid creating more merging headaches, so am able to merge it directly. I’ve redone the interface into them to be consistent with the C solvers, and, as an added bonus, I’ve written/rewritten the simple Rust solvers to support either u32s or u64s for edgepoints (the older version just hardcoded u64s). In theory it should allow us to run test-solvers with slighly less ram, but in practice I’m not sure this will have much benefit, (it make make it slightly easier to run test miners,) but at least it’s more consistent with the C solvers and should be easily upgraded when humanity’s transcendent core-mind can solve cuckoo-sizes that require 128-bit edge points.

And that is basically it. With this done, we should be able to start discussing when and how we’re going to put a T4 branch together and when we expect to release it. We generally want to minimise the window where we have distinct T3 and T4 branches, but hopefully this time we can do it without creating too many development annoyances.

There’s still one piece that needs to be in place before we can seriously think about T4 though, and that’s of course a way to actually mine Cuckatoo blocks via Grin-miner. I think my next move should be to start getting support for cuckatoo plugins in place, (which should mostly be config infrastructure anyhow, since the interface into them is the same,) and start getting some test cuckatoo miners in place based on @tromp’s ongoing work. It’s been a while since I’ve updated cuckoo/grin-miner anyhow, so it’s long overdue some attention, will hopefully get some QoL updates into it while I’m there.

That’s it for now, keep Grinning.