Yeastplume - Progress update thread - Oct 18 - Feb 19

Announcements
#16

Update Friday November 23rd, 2018

There appears to be some sort of mandatory thanking-type holiday going on in the US, so the community’s been a bit more quiet than usual over the past couple of days. I’ve had a solid week smashing away at the keyboard in front of several vim/tmux windows. The small matter of a release date being announced (on or around Jan 15th) has helped focus the mind on what absolutely, positively needs to be done by then, so I’ve altered course slightly. Here’s what happened:

  • The ‘plugin’ refactoring PR referenced last week is done, tested, and merged. There should still be a few things left to work out on that functionality, such as how to pass parameters into a plugin and whether/how to separate ‘plugins’ builds from the main code base. I’m hoping someone from the community interested in seeing wallet exchanges happen over their medium of choice will have a look at it, but otherwise I’m quite interested in integrating a keybase exchange when it can be appropriately prioritised.

  • Of more pressing concern now is BIP39 support and better handling of the wallet seed BIP 39 simply means “wallet recovery word lists”, but some changes to how we use the password and store seeds were also necessary to support this. This, I think, was the most obvious piece of user functionality missing from the command-line wallet, and will be in place once a cursory review or two is done and I push the big green merge button. Full details on what’s changing exactly are in the comments section of the PR, and the ever-growing wallet usage doc has been updated to reflect the changes.

  • I’ve also undertaken to start properly documenting all of the wallet API layers as an ongoing project. Putting together documentation, especially good documentation with code examples is quite painful (rust even tests documentation code examples to make sure they compile,) and much better done over time in short, focused sessions as opposed to marathons where you start taking shortcuts cause you’re sick of looking at it after 10 minutes. So more to come there over time, particularly on the more ‘inner’ layers of the wallet libraries, which I think are less likely to change now.

Just quickly on that, while the lower layer apis (libtx and libwallet) are of decent enough quality now (or so I think), the wrapper portion that currently comprises the HTTP owner api listener has more or less been thrown together over time and there’s no coherent JSON api into the various functions. This will need to be addressed at some stage, but it’ll probably be on lower priority than ensuring the command line wallet is ready for launch. I know quite a few people have been using it for their own projects, so if you have any suggestions for making the HTTP wrapper more coherent, let me know. Also keep in mind that when this happens everyone will likely need to update their applications to use a newer API.

Right, that’s it for now. Please return to your thankful bird slaughter or further fraying your relationships with your family or whatever it is you do over there.

3 Likes
#17

Update Friday Nov 30th 2018.

Nothing like a release date to sharpen the senses, especially one with tons of Yuletide disruption between now and it.

The wallet march continues:

  • Testing and bugfixes based on user findings in the field. With release date looming, and most of the functionality complete, I’ve started to pay more a bit more attention to these.

  • Endless documentation of the wallet APIs. Rustdoc is great, but it’s very time consuming to put together, especially given that it goes as far as compiling your code examples to make sure they’re current. I have about half of the OwnerAPI documented, with the rest coming very soon (I think I have most of the awkward ones done. Painful but worth taking time on.

  • Various API refinements springing from above. As I’m documenting, I come across things in the API and say to myself “this is pants” and remove or rework the offending API call rather than document it. (Actually takes less time in most cases).

  • Addition of an optional Slate message, basically provide a space for a user to include a signed message in the slate (should hopefully enhance UX somewhere down the line.

  • Also starting to do some thinking about what the HTTP/JSON api should actually look like. Thoughts thus far are in the issue, so won’t repeat them here.

  • Also noteworthy: @mcdallas is implementing a keybase plugin in accordance with the plugin work last week, and it looks really promising, very excited to see this.

Coming up: more documentation, perhaps a bit of a rework of the wallet command-line code (Igno pointed out that file was getting large, and he’s correct,) some grin-miner modifications to assist plugin developers (@hashmap is in the middle of one), more thought about how to build out the V2 plugin API, and bug fixes, performance testing,… and that doesn’t touch the backlog of ‘must-have’ issues that we have marked for release.

…and all of this with holiday disruption season coming up. Fun times ahead! (But don’t think I’d rather be anywhere else.)

6 Likes
#18

Update Monday Dec 10th, 2018

Completely forgot to write anything here with all of the excitement going on, the definition of ‘excitement’ being ‘broken the build during the time I should have been writing this’.

Last week’s work was mostly about re-organising the wallet command line code properly to make it more consistent and manageable, as well as changes to support end-to-end automated testing of the command line client (to the point where tests are actually parsing a command line and executing the result). Want to get moving on this weeks task’s (more wallet fixes, wallet performance, look into these mining issues everyone seems to be having to see if there’s anything in it,) so will keep it short and save a longer update for Friday.

So off to Vim I go…

#19

Update Friday Dec14th, 2018

Bug fixes, testing, stability, issue resolution the past couple of weeks, likely to continue for the next few weeks as we approach launch. Nothing earth shattering here, just many smaller items and fixes as befits the weeks before a release. The kind of development that makes you feel a bit over-excitable and scatter-brained. As a sampling of the kinds of things I’m addressing:

And of course, the large refactor of the wallet command line code and test organisation is complete: https://github.com/mimblewimble/grin/pull/2114

Also a few mining fixes, starting to look at wallet performance, and @jaspervdm is doing some work on converting wallet outputs to switch commitments that I need to be reviewing as well: https://github.com/mimblewimble/grin/pull/2157.

All in all, things are looking relatively stable still quite a few issues left to address as ‘must-haves’ before launch, but nothing too massive. And of course, mainnet release is just another development milestone. Once this round of consolidation and main-net launch excitement is over, we’re going to keep right on with additions and improvements.

Right, expect more of the same next week from everyone, and enjoy weekend. Only 376 mod 365 shopping days until Christmas!

3 Likes
#20

Update Friday December 21st, 2018

Santa’s on his way, so we’ll all be getting together with our families to roast chestnuts on our new 9-GPU rigs, the kids unable to sleep with excitement for Grin’s mainnet.

I expect I probably won’t be getting around to an update next week (or have enough to report to make one worthwhile,) so I’ll leave the next update until consumer-stimulus season is over and the Year of the Grin rolls around. I might even spend a bit of time with the Yeastlings as well, mostly just to make sure they don’t wreck anything else.

The past week has been about Floonet, which (a couple of hiccups aside), launched happily enough. For my own part, it was mostly bug fixes and knocking one or two things off the ‘must-have for mainnet’ list.

But I think I want to write about something slightly different today. After the launch of Floonet, I’m in as close to a sentimental mood as I ever get, meaning the empty space in my rib cage where my heart should be is a degree less cold than it should be.

So I’ve taken off the developer Santa hat for a few moments, cause when that’s on all I can do is focus on the massive list of things that either could be better or still need to be done. Instead, I’m just sitting back a bit and playing with all of the things we’ve all built over the past couple of years.

I recently set up a new rig (this one’s the Yeast Tsunami,) which I’ll be using to add my own paltry hashpower to the fray on mainnet, and today I went to install from scratch. I pulled and installed Grin (while carefully following the build guide, of course,) and it worked. I type grin at the command prompt, and up comes a fairly-informative TUI telling you what’s going on with your node and the chain. Sync kicks in (not enough blocks yet for fast sync), and I’m up to date and synced and in consensus with 18 other nodes. Stats on all of my peers and what’s going on mining-wise is all available right there if I need it. Although it’s early days for floonet, nobody’s reporting being out of sync, and the difficulty adjustment that has to account for AR and AT POWs appears to be behaving well

In another tmux pane, I type grin wallet init -h to create a new wallet. It gives me a BIP-39 recovery phrase which it tells me to write down (which of course I do, diligently,). I create a new account for mining, and start the wallet listening on that account.

So, let’s try mining. I take down the server, then turn on the stratum server in grin-server.toml and restart the server, which is configured by default to mine into the listening wallet on the default port. Because of the embedded stratum-server approach in Grin, now I can mine into my node and listening wallet from however many rigs I care to set up.

So then grin-miner… I check it out, change a config file to have it build the CUDA libs, and build. Grin-miner builds with all available plugins, and I have a range of CPU and GPU options to mine with right out of the box. I configure it to concurrently run cuckaroo29 on 2 2080s currently in the rig, and start grin-miner. Both cards hum into action and start mining away into my listening stratum server, confirmed by another TUI with useful stats about each mining device. Back on the server side, the ‘mining’ tab shows me the connected worker, and gives me more useful stats about how many shares each connected worker is contributing, and how many blocks have been found.

I find a few blocks and funds go into the wallet, which I can check with grin wallet info. Once they’ve matured, I should be able to exchange them with other users via files, http, or even directly via keybase.

Of course, Grin is larger than its github projects. Invariably, I’d want to check the status of the chain on a block explorer… I can go to Grinscan or Grinmint’s explorer.
Speaking of Grinmint, I decided to point a yeastmonster at Grinmint for a bit, everything works, and it looks excellent… very well designed and as easy to use as it possibly can be. I also try Grinpool, which gives me a similarly successful mining experience.

And that’s just a small sampling of community projects. On the wallet side we have a few upcoming projects,such as Grinbox, as well as quite a few mobile wallets being built. Open CL miners are being worked on (which I hope will be integrated into Grin-miner) and I’m sure there are many other bits of infrastructure that have yet to be announced. If I’ve missed anyone I apologies, but think it’s safe to say that a completely open-source coin with this kind of quality community infrastructure in place is a rarity to behold.

Now, of course there are still things to do, the code isn’t perfect, and your particular machine might not be able to compile the Cuda plugins without some painful support issues. But we’ve come a long way since Voldy did his bitcoin-wizards drive-by thing and Igno decided to push a bit rust code to github (WTF is this rust thing?).

When I look at these things in front of me, I feel proud. And I mean for real, not like you’re at some corporate launch party with a few project managers pretending to give a shit. Everyone involved in Grin’s development should spend a few minutes sitting there bonding with this daftly-named Floonet, and feel some genuine pride, because there’s genuinely something here to be proud of. And that’s not something you get to feel much of in this life.

So, many congratulations to the entire Grin community. Take some time this holiday season to relish this moment, cause silly season is coming on Jan 15th, The Year of the Grin.

A very Yeasty holiday season from me and all Yeast-beings in this house. And don’t let Granny drink too much.

17 Likes
#21

Keep on building the hustle is real merry Xmas to all you guys and your families .I had one question though . When grin does come online how will the tokens be distributed ? I’ve read the entire forum but have not found any concrete answers .

#22

Thanks for a very lively and pride-inducing report, Yeastplume.

Your prose writing skills are right up there with your code writing skills!

3 Likes
#23

I’m proud of where we are and what we’ve accomplished as well. And thank you for all your work and contributions to Grin. Wishing all the best things to you and your family. Cheers!

4 Likes
#24

Wonderful work, build and hustle, & a great update thank you.

Real nice to read this pre-Xmas, wish the core team & contributors a good rest and enjoy the ride 2019 will be fun at the least :slight_smile:

3 Likes
#25

Update Friday, Jan 4th 2019

Happy new year (year of the Grin) all!. Been swamped with bored/rowdy kids off school/creche for the past couple of weeks so time has been limited. However, I did look to squeeze in whatever Grin work I can wherever I could, so here’s a quick rundown of what happened over the disruption season:

Next week I’ll be doing more wallet testing, putting together a ‘howto’ section in the wallet usage doc, hopefully provide a very simplified mining ‘howto’, (now that there’s a binary build,) and whatever else comes up in the mad run up to launch!

2 more points of interest:

Thanks it for now. Enjoy a nice relaxing post-holiday extended-family-free weekend.

6 Likes
#26

Update Friday, Jan 11th 2019

A bit sick of writing things at the moment, so this is going to be a fairly short one. Here’s the reason why I’m sick of writing things:

Basically a lot of work getting the wiki filled with some Grin noob guides and a large pass over the wallet user documentation, which I’ve moved out of the source and into the wiki page (where it will be far more useful). Also a bit of work getting the site into a state where it’s better leading newcomers to the information they need. I’m trying to get all eyes towards the Getting Started wiki page, which should contain all the relevant links people need to get up and running with Grin

Some dev work too, mostly quality-of-life things with one major fix:

  • Add the ability to init a wallet from a recovery phrase added after a few users correctly pointed out the previous process was needlessly cumbersome and you should be able to recover a wallet in a single step. (though it’s two steps at the moment, grin wallet init -r followed by grin wallet restore which I believe should stay distinct steps, reasons why some other time)

  • Only update wallet outputs from the node if they’re involved in an outstanding transaction, this should give a large perf. improvement to the wallet under most conditions, as it was previously validating each and every Unspent output in the wallet each and every time to see if it had become spent. Makes sense that a wallet shouldn’t have to check outputs it hasn’t inserted into a transaction.

  • Automate wallet seed recovery process automagically backs up/moves your existing wallet seed file if you’re trying to recover a seed into existing wallet data. Should be useful for those who forget their seed passwords

  • libSecp bug fix Important fix, and very glad this was caught in floonet, as it would have made some outputs under certain conditions hard to recover and cause problems with wallet check and restore.

Oh and renamed check_repair to check, since check_repair sounds like the wallet broke, which is not the case. I very much think the check process should become part of usual wallet operations instead of having to run it manually, just the scanning process is potentially too cumbersome to perform all the time at present (I think we need an incremental scan approach).

And thanks again to everyone who attended the Grin London Meetup! Was great seeing everyone and I hope to be coming out to a few more of these things once the mainnet launch dust settles down.

Speaking of that, Mainnet is launching in 4 days. Enjoy the weekend.

7 Likes
#27

Thanks for drafting this update and being consistent every week.

You may want to check you link to “Getting Started wiki page” it is not taking one to:

Getting Started With Grin: Links and Resources

#28

Update Monday, Jan 21st 2018,

We launched a blockchain. Hope you like it! This brings the number of blockchain launches I’ve been personally involved in from 0 to 1.

We’re about a week in, and we don’t seem to have had any major surprises. The initial difficulty adjustment worked exactly as planned to ensure everyone had a fair chance to mine, and nobody ran away with the first thousand blocks simply by virtue of being first. (Actually, perhaps the fact it worked counts as a surprise.) Network mining activity appears more or less healthy, plenty of transactions are happening, fast-sync works, and nobody appears to be panicking over anything.

So there’s a chain there now. What next?

Well, as we’ve repeated many times before, Grin’s not supposed to be get-rich quick scheme for early miners. It needs to become useful as a medium of exchange. Obviously we’re not shooting to replace Apple pay right now, but we can certainly try and become useful in areas that are already predisposed to working in cryptocurrency, i.e. in situations where Bitcoin or Monero might already be accepted. In order to do that, we still have some work to do.

Transactions are cumbersome, and the vast majority of potential users don’t (and shouldn’t need to) understand the details of how a transaction works; they should be made as seamless as possible. The approach Grin is taking is to try and make the core wallet APIs as useful as possible, and leave transaction exchange mechanisms for different needs up to the community. This appears to be bearing some fruit, but there is still much more work to do here to facilitate all of the varying needs out there. We’ve always known this portion would be a challenge, and we’re working on it.

All of the potential extras and additions we’ve been talking about on the Grin project to date, (many of which would be very distinguishing features): scriptless scripts, atomic swaps, vaults, Confidential Assets… none of these exist in V1. Hopefully now some of us will have the time to start thinking about them a bit harder. Many of these are not as straightforward as they might have first appeared, but we’ll get there.

There’s performance (the network is young, so there isn’t enough data yet for the bottlenecks to make themselves known). There’s scaling (both on-chain and via enabling secondary layers). And of course, there’s the continuing march toward more perfect privacy, acknowledging the work and thought that still needs to go into the potential weak points of the system.

And of course, anything that comes up in the course of trying to manage a new chain under continual widespread continual public scrutiny, (no pressure).

So, in short I’m very happy with where we are, especially given the resources to date. But there’s still an unfathomable amount of work to do over the years to come.

I’m just going to try to take it one week at a time, and I’ll let you know how this week went on Friday’s update.

9 Likes
#29

Update Friday Jan 25th, 2019

A week and a half in and nothing has exploded yet (touch wood). The run up to mainnet launch felt a bit frenzied, but that’s over now and we’re starting to settle in. In the absence of anything being totally on fire (touch wood,) it feels like we have a chance to catch our breath and continue with the business of developing Grin in a more relaxed manner.

The pace of development is going to be a bit different from now on. Since we’re actually live and need to support everything that’s out there now, development needs to be that much more thoughtful, measured, reviewed and tested. Sometimes it may seem that things are moving more slowly, but this is very much a good thing.

I think I’ll cut off what I was about to launch into there, as I’ll actually be discussing the future of Grin in SF at Grincon US next week.

So, this week, I:

  • Did much preparation of presentations for GrinconUS, results of which will be obvious next week
  • Did some fixing upping of error output messages, here and here to hopefully provide us with more clues when an issue is reported. It’s very difficult to sort out actual issues from user error at this point (not always users’ faults’, there’s a lot that’s non-intuitive to get wrong,) so adding more and better logging should help in many cases. Of course, these won’t be in widespread use until 1.0.1, so it may be a while before we see results (told you the pace had changed).
  • Save slate messages in wallets so they can be accessed later (at the moment they’re just thrown out). Should make it easier for all parties to identify their transactions in various settings.
  • Some triage, some reviewing of what’s there, some support, some reading about all of these future techs we want to put into Grin, all sorts of catch-up activities that kind of got put aside during the run-up to launch.

It’s GrinconUS for the first part of next week (11 hours there and 11 hours back). Once that’s done, my mental map of what to look at next involves:

  • Recipient-initiated transactions
  • Splitting the wallet and libs out of the main Grin executable
  • Wallet V2 API (important one)
  • Wallet Performance
  • Fixes as identified!

That’s it for now, have a good weekend and look forward to seeing/meeting everyone at the first ever GrinconUS!

3 Likes
#30

Update Monday, Feb 4th 2019

Another campaign brought to a close, and I’m once again delighted, privileged, and honoured to be able to continue in the service of the Grin community. This time, we’ve overshot the goal by just over a month’s worth, so happy to say I should be able to work on Grin worry-free until about the end of September 2019. All of this thanks to the community, which I still maintain is one of the best around in any open-source or crypto project currently going on.

Special thanks this time to Qtum, (with a massive 10BTC donation,) Sparkpool (With 100 ETH), f2pool, Prokapi, Bitfish, and several others who have either donated to my fund, or are continuing to donate directly into the general fund. Our friends of Grin list is growing longer by the day.

So on I go with the business of developing, improving and enhancing Grin!

Last week was mostly dominated by travel to Grincon in SF and back, as well as recovering from said travel. It all went extremely well, and thanks again to @catnic for organizing and running such a great event. Aside from that, I got a start on two important issues:

image
image1200×691 280 KB

That’s the wallet, node (in user testing mode), and test miner running on Windows 10, which I think demonstrates Windows support is not far away. Still quite a few issues to sort out, but I’ll start a meta-issue on Github to track exactly what needs to be done.

My short list for the coming week is as follows:

  • Create a windows build meta-issue, and start getting those elements crossed of
  • Work on splitting the wallet out of the main Grin package, to allow it to be enhanced and versioned separately from Grin
  • A grin miner update to include latest cuckoo 31 solvers

And of course, much more. Will let you know how I got on in my Friday update. Thank you again, Grin community.

4 Likes
#31

Update Friday Feb 8th, 2019

Going to be very short and simple this week. Basically, the ‘Proof-of-concept’ I managed last week getting grin to build and run on windows has resulted in a few github issues based on the findings:

Everything I know about to get Grin released on Windows is in the meta-issue here:

And I did quite a bit of (tedious and fiddly) work this week getting these issues resolved in a more platform-independent way than the proof of concept, as well as get it to the point where the node can fast-sync and run mainnet on Windows:

Even though we’re still quite a bit away from something we can call a release, once the above PR is merged, it should be possible for anyone to compile and run grin on Windows 10. Will still be a bit of a manual process until everything is resolved and proper binaries are released (just like running grin on linux used to be,) but those interested enough to do so can definitely start having a go with WinGrin.

Biggest issues still would be (big hint for anyone windows-leaning who’s looking for something to do):

  • TUI works, but freezes for some reason, needs investigation
  • Code needs to be introduced to get LMDB to allocate a small DB at first, then grow as needed. This is because windows allocates all of the requested DB space at once on creation (as opposed to on linux, where it grows as needed.
  • CI Build

Just on the wallet front, if you’re thinking of submitting a PR against the wallet, please talk to me first. Next task I have up is to remove it and wallet libs from grin into a separate repository, so anything you PR will likely need an annoying transplant once you’re done with it. There are a couple of outstanding wallet PRs up there which I’d like to see merged first, then I’ll start that work.

That’s it for now, enjoy weekend and be exemplary.

1 Like
#32

qr-code
qr-code.png1147×1147 28.7 KB

This QR art is for Gifting to the Dev fund

1 Like
#33

OMG i love it. Brilliant!

Hm, the QR code even seems easier to read/easier to spot in this yellow/black colorway. I wonder if that’s true for cameras and computers too, or if they prefer black/white.

Anyway, thanks!

#34

The yellow is a great iconic color and works well with camera phones or scanning. I will be able to make additional QR artwork look looks like raised ‘3D wet black-metal’ styling too and it will still scan. Plans to make a template for merging the Logo with QR inside the face is possible.

1 Like
#35

Update Friday Feb 15th 2019

Don’t have massive amounts of detail to relate this week, other than the creation of the split-off grin-wallet project here:

https://github.com/mimblewimble/grin-wallet.

I took the liberty of splitting it up into a few distinct crates, (still subject to some review,) and performed a bit of a clean up on it. That aside, it should work exactly the same as the current integrated grin wallet. The point of this exercise is to separate the wallet into a separate executable so we can release wallet fixes, updates and new features without being tied to grin point releases.

The grin-wallet repo is mostly up and running now with CI and automated builds. Next up should be to remove wallet code from the 1.1.0 branch and ensure anyone with wallet PRs knows to include them in the new repository. We’ll likely want to syncronise the first real release of the separate wallet alongside grin 1.1.0. Once that’s all in place, I’ll likely get back to the few remaining windows issues and see about getting an automated windows build and release in place, as I would very much like to see windows support for 1.1.0 as well.

Very simple this week, but happy with progress. Enjoy the weekend!

3 Likes