Hi everyone, I will be using this thread to give weekly updates of what I am working on.
The big feature I am currently working on again is parallel IBD. It will improve the sync times of new nodes and nodes that have been offline by downloading and verifying the initial data from multiple peers instead of a single one. This project is pretty large and will require a significant amount of changes to the sync process. Earlier in the week I opened this PR, which streamlines the p2p code and is a preparation for the sync changes. More of these kind of PRs will follow in the coming time.
I am also currently in the process of rewriting the parallel IBD RFC, based on discussions in the PR and some tests I performed. In short, to avoid edge case scenarios and possible attack vectors, the sync will start off with a download and validation of the full utxo bitmap. The rewrite of the RFC is underway and I am expecting to publish a new WIP draft of it in the coming week. I am considering to limit the scope of the RFC to purely the p2p messages, meaning we likely will have a separate one that describes the required changes on the sync flow.
Since the project is fairly large, me and Antioch will both be working on it in parallel from different angles. My current objective is to have a fairly general implementation of generation and validation of the PIBD data to be sent over the wire, which we can then integrate into the node.
All in all I am excited to get this back on the road and implement this! I think it is an important feature and a fun challenge.
It grows by one bit for every txo. It’ll probably be a very long time before we would ever get any practical benefit from downloading the utxo bitmap in parallel.
In the proposal the bitmap request will have a parameter to control the number of returned chunks (and an index). This would be the most flexible solution, allowing us to either download the full bitmap from a single peer or maybe in the future get it from multiple. But this is something we can discuss in the upcoming RFC PR.
More work on the p2p reader in grin#3433. The actual changes are quite technical and I recommend anyone who is interested in them to read the discussion in the PR. In my opinion the new code is conceptually cleaner, we now have better separation of the different layers and it makes it easier for us to build on top of it. The PR is in final review and should be ready to be merged soon.
Progress on the new p2p messages for parallel IBD. I wrote down some of the technical details in a new RFC. Note that this RFC is unfinished and subject to changes, but it reflects my current thinking about the subject. It turns out that the data we need for the different types of MMR is very similar, which allows us to generalize a lot of the code. The implementation of it is underway and I will have a first PR (of multiple) up in the coming week.
Thanks everyone and see you in the next update post
grin#3433 is finished, has been reviewed and is ready to be merged.
Progress on PIBD: I have put up a PR for the generation and validation of (P)MMR chunks. There are still some edge cases that need to be thought through and fixed but the base is there. Right now the chunk contains slightly more data than strictly necessary, but this is something we can iterate and improve on, and can change without altering the base structure of the chunks. Next up will be the (de)serialiation of these chunks to be sent over the network and the implementation of the full p2p messages.
I have been investigating an issue that occurs for some wallet users after a period of time: grin-wallet#509. While the exact cause of the error is still not clear, I am working on a fix that prevents it from occurring in the first place. As a bonus we will be able to remove some of our custom http client code and replace it with a more established one. A PR will be up this coming week.
I forgot to post an update yesterday so here it is.
Last week I’ve been continuing with the code to generate and validate segments of MMR data for the parallel IBD process. The idea is that in the future these can be downloaded and verified in parallel from multiple peers. At this point the PR is pretty much ready for review, which I expect to take a bit due to some of the subtleties involved. In the meantime I’ve been working on another preparation PR that will make our transition to parallel sync easier, which will be put up not long from now.
Secondly I have a PR up that fixes the wallet issue I mentioned last week. I need to do some more testing with it but it can go through review after that.
Hey everyone, here is a quick update from me. Things have been progressing on all fronts. I had been putting off the writing of the RFCs for a bit but last week i picked it up again. This will continue in the coming week and in the next post I will link them so anyone that is interested can check them out. Besides that there are a few things in review, nothing major to report for now. I will keep you all updated!
The review of grin#3453 is still in progress, but moving along well thanks to antioch. There are some subtleties in this PR, which is why review is being done carefully. I have made some adjustments based on his suggestions.
The RFC accompanying this PR has been updated and is nearing completion.
On the implementation of the new sync process that will be using the new p2p messages from the above RFC: I have been doing some experimenting with finding the most optimal method. Some technical details: right now all our incoming messages are handled by different “chain adapter” layers, which provide a good separation of concerns (e.g. keeping track of requested data and banning of peers is done by different layers). This does however mean that the chain is locked all over the place, which requires more coordination between each p2p reader and is less efficient. The alternative would be to create a channel that processes these messages in sequence, since writing to the chainstate is inherently sequential anyway. I am not entirely convinced either method is better, but am hoping to decide on it soon.
On the wallet side, me and yeastplume have been working together on a PR to provide a late locking mechanism. This will allow users to minimize the time their outputs stay locked, preventing the annoyance of locked outputs when the transacting counterparty is slow in responding with a slate.
Unfortunately last week was a bit slow due to me being sick in the middle of it for a few days. Thought I’d write an update regardless, just to explain the current focus. According to the priorities defined in the v5.0 release planning I am assigned to 2 items, which I have both been mentioning in the past weeks. I am spending my time (together with the other assignees) on these to make the first milestone: first beta in little less than a month.
Late locking: this is well underway, a part from a pesky bug we still have to fix. This requires some investigating, to be done in the coming week
Partial PIBD: this is assigned P3, because it is not crucial to be included during a Hard Fork. I think it would be best to limit the scope for this to purely the response and requests by the clients. It saves us some headache if we introduce it now as opposed to after the HF, as it would mean all nodes will have the feature. As a large part of the RFC and impl is done, I think we can realistically make this deadline. Then, in the future we can introduce the new sync procedure in a new node version whenever it is ready, as it requires no further consensus/p2p changes.