Going to be a very quick update from my end this week, since it’s mostly been a deep-dive into coding mode:
Mainly, a large refactor/reworking of the ‘lifecycle management’ for a wallet in anticipation of the corresponding RFC. The goal of this PR is to ensure all of the traits and structs are properly re-worked to support full lifecycle management via the API, without actually changing anything from the user’s or API consumer’s perspective. Once this is in place, the new API functions in the RFC should be fairly easy to implement. Much struggling with the Rust compiler with respect to the borrow and lifetime checks, but I think all of these changes will make things easier to work with, and the entire code will be very well modularised.
That’s pretty much it, and I’d expect at least a couple more weeks on this PR in particular and refining the RFC before it’s ready to merge.
On the wallet-subgroup front we don’t have an exact structure for the wallet working group, we’re basically just running the channel now to see what happens. I can say that there seems to be more relevant conversation and feedback about wallet changes than there was before, so from that perspective (though it’s early days,) it seems to be having a positive benefit. A good start, and I hope to see a lot more activity in the group over the next few weeks.
Well, that turned out to be possibly the most uneventful hard-fork in the history of hard forks. I think the worst issue we had was that wallet listeners need to be restarted if they were running over the course of the fork (and I’ll take 100% responsibility for that one). Like the whole Y2K thing, all of the work that gets put into ensuring these events run smoothly goes completely unnoticed by most people if it’s done correctly, and everyone involved in Grin has done a great job making sure this one was completely unremarkable. Well done everyone!
Also, everyone should be on the same page with respect to running the 2.0.0 node, V2 Slate, V2 APIs etc and as far as I can tell everyone is able to transact again without these compatibility issues. We’ll all be being extra careful from now on that this remains the case, and will be highly unlikely to be making slate changes without a) very, very good reasons or b) at a scheduled hardfork opportunity
So on we go. For my part this week:
The large aforementioned wallet refactoring is mostly done, or at least to the point where everything works again and all tests now compile (and I took the opportunity to refactor a few tests to make this a bit easier in future). I’ve paused development there a bit to have a think, because one of the aspects of this refactor is to handle passwords and seeds over API calls in a relatively secure way. Currently researching and discussing to come up with a scheme here, and will be incorporating thinking there into a revised version of the RFC next week.
The London Grin Meetup went well, was good to see a few familiar faces and meet a few new ones. Also took the opportunity to have some decent technical brainstorming with @lehnberg, @jaspervdm and @ravidio (on keybase) about possible improvements and directions for Grin. I won’t go into too much detail about any of those, but I expect we’ll be relating some of these ideas in both the wallet and node subteam channels very soon.
That’s it for now. Enjoy post-hardfork bliss everyone! (for a little while before we all start worrying about the next one)
Apologies for missing a week there, but when I go into deep code mode I lose the ability to be social, and need to step away and decompress for a while before I can talk to people in a civil fashion again. Naturally, Mrs. Yeastplume loves this.
So that’s pretty much the story of the past couple of weeks, deep-code mode with a couple of very large PRs. The first one was intentionally large, the second one ended up larger than I thought due to the sheer number of places arguments had to be changed. The end result of these changes to the end user of the wallet or a user of the JSON API should be absolutely nothing, which is exactly what I’m aiming for here. However, with these changes in place the wallet lifecycle RFC implementation should consist of a few relatively small and much more incremental PRs on top of the groundwork that’s been done here.
In a bit more detail:
(The wallet lifecycle PR)[https://github.com/mimblewimble/grin-wallet/pull/184] was finished over the previous week, and contains a fairly large refactor that better defines a wallet instance, in such a way that functions such as create_wallet and open_wallet can exist. This adds a new WalletLifeCycle trait to provide these kinds of functions which can be implemented however it’s needed (and a default implementation is also provided for the reference wallet). Much code around creating/recoving/restoring wallets etc was moved from disparate places around the CLI implementation into the much better contained DefaultLCProvider. This cleans up the code considerably as well as paves the way for exposing this functionality via the OwnerAPI.
Once that was done, I focused on adding a more detailed security model to the wallet lifecycle RFC. The proposed solution XORs the stored wallet seed with a random mask in memory, and clients must provide this seed as a token on each API call. There is also a layer that encrypts JSON-RPC traffic using an ECDH derived key. I hope this gives sufficient security for passing sensitive data via JSON-RPC calls (even if it is just over localhost in many instances).
And I have just finished up the second large preparory-PR related to the above, implementing XORed wallet seeds. There’s not a huge amount to say about this, other than API changes are implemented in a backwards compatible way, and yet again there should be no changes whatsoever from a user perspective. Those interested in the details can have a look through the PR, but the vast majority of the changes there are just tests, doc fixes, and ensuring the XORed wallet seed mask is propagated when it’s needed.
Less visibly, also spent quite a bit of time getting the Grin donation address back up and working, and ensuring the key is split among council members via the Shamir share splitting lib I did a few months ago (which I hope to get into the wallet as a feature add one of these days, but far too many more important things to focus on first.
That’s the story of my past two weeks. Just as a warning, it’s coming up on ‘running out of runway’ time in the yeast coffers, and the council has been discussing whether it’s worth running another donation campaign. Some of the reasoning for doing this, despite having “a lot” (air quotes, will get to that in a second) of money in the general fund is thus:
The amount we have sitting there (600-700k) is a lot for an individual, but absolutely nothing for a real-world development team. The amount there wouldn’t run an average-sized development team for a year in the corporate world. This is why we haven’t gone mad with spending it and artificially inflating the team, it would disappear very quickly if we did this.
Yes, we can easily cover my current needs from the general fund, but Grin is always in a state of needing money and we’ve taken the foot off the accelerator with respect to funding, partly because of the large 50BTC donation. We should always be doing our ABCs (Always Be Campaigning). This might be a good excuse for a bit of a campaign drive, cause even if a sliver of money is raised from a separate campaign effort, it’s a sliver that might not have been there before.
And a couple of seemingly good reasons for not doing a campaign and begging directly from the general fund:
It’s less confusing to those who want to donate to donate to a single council address. Most who’ve done so have done so on the assumption the funds will be put to good use. There was definitely some confusion on my last campaign where those who thought they’d donated had donated to the general fund.
There may be reluctance to donate more when we have an (apparently, see above) large amount of funds still untapped.
Running a campaign in light of the above may seem cheeky, and I’ll have to go to great lengths to explain why it’s being done and why we don’t actually have all that much money. (On the plus side, we’d get some amusing twitter-trolling as a result).
So basically, still undecided what I’m going to do here, but need make a decision over the next week or so. Would appreciate any thoughts from people who have an opinion on this one way or another.
That’s it for now, long weekend here (i.e. everyone else gets a holiday and I get stuck with hellspawn Yeastlings for an uncomfortable number of subsequent days).
Short this week due to earlier bank holiday and I’m also taking a few days off here and there this month as opposed to a single long holiday (even yeast needs to recharge itself sometimes).
Major point this week was completion and integration of the The wallet lifecycle PR above, which also included a minor change to ensure the APIs are returning VersionedSlates as opposed to Slate objects themselves (needed for an upstream PR)
Currently working on the next phases of implementing the wallet lifecycle RFC (which should be ready to merge anyday now). The next changes should be coming in smaller stages than the previous, and the next PR will focus on getting the security elements outlined in the PR in place, namely the init_secure_api function (all of the new functions will likely go into a V3 API with the older insecure functions remaining in V2 for the time being).
There are a few other long-outstanding issues I’d like to target for the 2.1.0 release, so currently getting together a proposal of what should go into 2.1.0 to share with the wallet team shortly.
That’s all for now, enjoy weekend and cherish your family time with any ungrateful offspring you may have.
Short again this week, but this time due to the fact it was all spent on a single significant PR:
Addition of the init_secure_api functionality into a V3 API… as outlined in the wallet lifecycle rfc, this adds the ECDH exchange, api method call encryption (and loads of tests) to support the upcoming lifecycle functions. I won’t go into detail on it here as it’s covered in the comments on the PR and RFC documentation, but I definitely urge everyone involved with the grin wallet to review and comment.
That will get merged once I figure out why tests are randomly failing on the windows portion of the CI build (fun).
And with this in place, all of the supporting work for the lifecycle functions is done, so hopefully their implementation should be relatively straightforward. Will find out soon enough, as the next task will be to work on the create_wallet and open_wallet functions.
Onwards and upwards, enjoy the weekend and what remains of kids-in-your-face-all-day season.
The culmination of the past few months of work and refactoring is now coming to fruition… or in more down-to-earth terms I’ve been working on adding the first batch of lifecycle functions to the Owner API. Once this is done, wallet authors will be able to create, open and close wallets entirely via the owner API, which should reduce the need for custom jigging to perform these fundamental tasks.
The previous work on securing the API has meant that the implementations of the functions so far has been fairly straightforward, so I’ve also been taking time to enhance the tests and testing tools to allow for better and more realistic test coverage (the results of which can be seen in the main ‘tests’ directory. A bit more testing and documentation remains on the first batch of functions (create/open/close) type functions (hopefully this week), with the rest of the lifecycle functions as defined in the API to come in future PRs.
Plenty of future work to come beyond this, of course. In addition to the lifecycle API, the upcoming 2.1.0 release will also aim to address a few longstanding bugs/issues, with the exact scope to be defined over the next couple of days. Then I have about 3-4 potential RFCs in my head that will need to be written up and discussed. I’d hope the next 6 months sees us getting the wallet and APIs into a very robust and usable state, entirely ready for the community to pile layers on top of it.
Enjoy what’s left of the summer, and particularly kids returning to school.
Took some time off this week, mostly to attend to personal matters but also to recharge a bit. The previous few weeks had been fairly intense in a lines-of-code-added type way and the next few are likely to be full-on ‘getting ready for a release’. No real holidays taken over the summer, so I hope you can forgive yeast needing to ferment for a bit.
Previous to that though, the aforementioned batch of lifecycle functions was done, tests written and merged: https://github.com/mimblewimble/grin-wallet/pull/211, so I very much urge anyone who uses the wallet API to get the latest from master and start using the V3 API. It’s not quite final (I’ll be adding a couple more functions to deal with passwords and BIP passphrases before calling that API complete for now,) but what’s there now is pretty much what the 2.1.0 release will look like.
Back in full swing next week tackling those new functions and the rest of the outstanding issues for 2.1.0.
All very straightforward this week, mostly getting through whatever issues remain for the upcoming 2.1.0 release.
Firstly, the lifecycle API functions and all the work needed to support that is now officially done! Very happy to have these in place as it should now be relatively easy for anyone to come along and create a Grin wallet, and not have to worry about writing special code to create and instantiate wallets, etc. This is the major feature of 2.1.0, so again I encourage all Grin wallet developers everywhere to start playing with it now (the older v2 API isn’t going away in this release, but the lifecycle functions aren’t available in it).
And moving along to looking at work started by @jaspervdm on transaction confirmation via kernel lookup](https://github.com/mimblewimble/grin-wallet/pull/216) (as opposed to outputs), which should be a more robust approach overall to confirming transactions as well as fix a nasty issue where a transaction won’t confirm on the receiver end if there are no change outputs (doh).
Also been thinking a little bit about a helper library (discussed in the wallet dev channel) that includes both the node and wallet libraries, spins up both and exposes the APIs directly into Node.js as an NPM package. Not sure if this is getting a little deep into ‘community’ territory and whether it’s something we want to maintain long term, but on the plus side it would be good to have at least a sample starter project that anyone versed in js/css can just pick up and run with. If you have opinions on this, happy to hear about them in the wallet dev channel.
With a large chunk of work out of the way, I’m now thinking about the next issues I’d like to pay attention to, in no particular order:
The one true method of transaction exchange, much discussion has been going on around this, which I’d like to catch up on and weigh in on very soon.
Implementing a proper interactive CLI mode to the default wallet (in addition to the existing ‘one invocation per operation’) method it currently uses. (All of the work for the lifecycle functions needed to happen before this)
Chain scanning as part of normal operation (trying to eliminate the need for grin-wallet check)
Multiple wallets per installation
The node helper library, as described above
Long-outstanding bugs/issues within the current list (some of them may not be issues, but need investigating)
And 2.1.0 release work, 3.0.0 release planning etc all as a matter of course on top of this.
First off, all of the issues for the 2.1.0 release on the wallet have been closed off. I’ve done a first pass at a beta 1, but several issues have come up between the node and the wallet, so will likely be a couple of days before we urge everyone to start trying it out.
Secondly, the subject of transaction exchange has been at the forefront in recent discussions, and if you’re at all interested you can see the entire colourful history in the keybase tx_exchange channel. To summarize my current position on this (which I think is basically the majority consensus most participants in the channel reached independently:)
Mixnets look great on paper, but most of them are purely academic, untested, and don’t actually exist
i2p looks good in theory as well, however:
it uses a lot of outdated cryptography
The project doesn’t appear to be particularly well maintained or have a lot of direction
Stopping and starting i2p nodes in the way a wallet needs adds to instability in the network
Tor isn’t perfect either, but as outlined in @david’s RFC, onion services give us a logical manner of creating verifiable addresses unlinked to IP addresses or other identifying information, alleviates firewall issues and gives us much better privacy then we have now (pretty much none over http).
There are still a lot of potential issues with Tor as the only method of transaction just yet, namely the fact it could be very difficult to get it working on mobile (iOS in particular,) and it’s not a great choice for node p2p communication. There are also installation and bundling issues, which could be annoying to maintain but not insurmountable from a technical perspective. However, of all of the ideas presented this one give some very tangible benefits, and has the big advantage that the Tor network exists and is usable today. So I think it’s therefore very much worth exploring to the fullest to see where we can get with it. I absolutely cannot promise right now this will become the only method of exchange, but it’s definitely a promising approach.
So that’s brings us to the Experimental TOR PR
I’ve been working on over the past few days. It’s at the point where it works with a lot of manual setup, but it looks promising, and I look forward to seeing how we can deal with the outstanding issues over the next couple of weeks. If you’re at all interested in this method of exchange, we’d very much appreciate if you could give it a try. I’d hope to merge the work there into master once 2.1.0 is out (as it still remains very much optional,) but for now, the instructions are all in the PR
Will have more to say on the particulars of the Tor integration over the next few weeks. Time for weekend, I think.
