Yeastplume - Progress update thread - April to June 2020

Another funding period another thread. Same everything else as yesterday.

It’s RFC season, which means a lot of discussion trying to finalize the exact contents of the Hard Fork 3 / v4.0.0 releases. The exact list of changes and priorities were suitably firmed up in the last development meeting, and a pile of RFCs have been created as a result. My own focus has obviously been the reductio-ad-absurdum of the JSON slate, along with keeping a healthy eye and engaging in much healthy thought and discussion on @joltz’s Slate serialization draft. Also trying to at least cursorily keep up with @antioch’s hugely important NeRD kernel RFC, so feels a bit like drinking from a firehose at the moment with all of the information and conversations to keep up with.

From a pure-code perspective the PR I’ve been maintaining while putting the RFC together is more or less ready to go, provided there’s no immediate pushback on the contents of the RFC. A small upstream change is required to libsecp256k1 before that can be reviewed and merged, but once that’s in my tech. focus will be on implementing the rest of the changes to the Slate outlined in the RFC, most likely followed by more exploratory work focused on whatever the state of @joltz’s PR is at that time.

Another initiative that’s come up (with some help and feedback from others) is the ‘Community Approved Wallet Initiative’, which I’m actually quite excited about but haven’t actually had much time to follow up on. I hope to get moving on this over the next week or so, with the first step being a proposed set of approval guidelines to eventually be fed into an RFC. I also really want to start the community actively working with those upstream wallets as much as they do the Grin project itself to help them continually improve over time.

That’s all for now. Hope everyone is adequately distancing, listening to professional advice from people who actually know what they’re talking about, and sneering and commenting to your loved ones about any congregations of teenagers you may spy on your way to the shop.

4 Likes

Update Friday, April 24th 2020

Deep code mode for the past couple of weeks, which necessarily means I’m far less socially inclined, apologies. Very much trying to get the compact slate RFC + related work done and dusted, and I think it’s about 99% there definition-wise and about 85% there code-wise. It’s taken a few iterations, but I think the current state of the V4 Slate is about as minimal as it can be. And given it has become so minimal, it seemed a logical step to go ahead and make it even more minimal.

Here are the highlights:

  • Rather than continuing with the mega-PR, which was holding back development, I just decided to create a separate branch altogether then rapid-fire PR the changes into there. We can merge the whole thing once everything is ready and the RFC is accepted, and hopefully everything is a bit easier to review now that the changes are all broken up

  • As the minimized slate turned out quite flat, as seeing as we already have plenty of serialization code in the core, i decided it was an easy-enough step to allow slates to be serialized to a very-compact binary format This works well enough and maintenance/new versions should only require a single translation from the current slate version to binary and back.

For reference, here is a size comparison of a few slate files between json/binary, a collection of standard, standard with pament proof, and invoice transactions:

107 standard_S1.txbin 
211 standard_S1.tx 

942 standard_S2.txbin 
1350 standard_S2.tx

1969 standard_S3.txbin 
2821 standard_S3.tx

99 invoice_I1.txbin 
193 invoice_I1.tx

1125 invoice_I2.txbin 
1655 invoice_I2.tx

1899 invoice_I3.txbin 
2697 invoice_I3.tx

172 standard_pp_S1.txbin 
330 standard_pp_S1.tx

1071 standard_pp_S2.txbin 
1658 standard_pp_S2.tx

2028 standard_pp_S3.txbin 
2821 standard_pp_S3.tx

For reference the max that can fit in a standard QR code is 3k. Though we have yet to define standards for armoring or QR codes to go on top of this, the reasonable size of the S1/S2/I1/I2 parts of the journey gives us options we didn’t previously have.

  • And all of this is of course defined (along with any confusing terminology I might have just used) in the compact slates RFC, which now includes the binary spec.

Still a bit more work to go, mostly updating the doctests with all of the slate changes as well as ensuring backwards compatibility with version 3 wallets. But should have this done over the next week and then I think we’ll have time to turn some attention to armoring possibilities.

That’s it for now. Enjoy what at one time was perceived as the weekend.

8 Likes

Update Friday, May 1st 2020

And with one mighty pull I can (kind of) confidently say that the work to support compact slates, with the added bonus of binary output, is complete. There may be a few additions to the RFC, but I very much hope to have any changes done + the RFC into final comment period for the Governance meeting next Tuesday.

It’s taken more time than initially anticipated, particularly because all of the reductions mean that wallets need to keep full track of their parts of the transaction as opposed to being able to read them back from returned slates. Further, everything needs to remain compatible with 3.x.x wallets for the period between the release of v4.0.0 and HF3, meaning that the wallet still needs to be able to output and handle V3 slates. Then between http/tor sending, receiving, invoicing workflows and file workflows between both versions of wallets, then updating tests and fixing issues found during tests then manually testing wallet compatibility (and then maintaining the RFC in parallel,) it’s been quite the chunk of work.

But it’s more or less done now, and we now have a compact slate branch that should be ready for merge into master while remaining compatible with all of the wallets out there.

Next up, one or two possible changes and refinements, then hopefully next week should finally be the week of slate-serialization/armoring experiments.

I think I’m going to celebrate by staying at home this weekend. Please remember not to do anything interesting.

6 Likes

Update, Friday May 15th, 2020

Very much continuing in deep-code mode for the run-up to 4.0.0, here are the major highlights.

  • As mentioned at great length before, both the compact slate work and related RFC are done and dusted.

  • You may have had a look at the potential for lock-free transactions, an idea so exciting that I thought we needed to introduce some changes into 4.0.0 to prepare for it, namely an enhanced transaction offset selection process.

  • Plenty of experimentation with Slatepack, to the point where it more or less works. There’s still plenty of experimentation and iteration to go, but now that all of the other issues are out of the way slatepack should be my sole focus for the next two weeks, including hopefully getting encryption working for in time for its first outing.

Think it’s time to step away from the machine. Have a good weekend!

4 Likes

Update Friday, May 29th 2020,

Last few weeks have been fairly brutal work-wise, trying to finish defining and implementing all of the changes required for slatepack. Just lost about 2 days worth of work to a bug in a dependency that prevented TOR sends from working altogether, but there’s hopefully some light at the end of the tunnel. And given we’re supposed to be releasing a beta next tuesday, there isn’t much tunnel left.

Numerous PRs created and merged over the past couple of weeks, which I won’t go over in too much detail here. I’ll leave perusing through them as an exercise for the interested reader. The end result, of course is that we should have the Slatepack formats + encryption + workflows ready for the 4.0.0 beta release, with the only real major thing outstanding on my side being the actual changes to the CLI to integrate them into the workflow

I see all of the other issues and conversations and threads going on, but I’m famously bad at concentrating on more than a particular issue at a time so bear with me. Trying to get the previous batch of developments and ideas implemented is a bit like being stuck cleaning up the mess from the last firehose while someone tries to attract your attention by pointing another firehose full of chunky ideas for your immediate consideration at you. But I should hopefully be able to breathe a bit after the 4.0.0 beta is released and start thinking about some of the other things going on.

4 Likes