Jaspervdm - Progress update thread

Hi everyone, I will be using this thread to give weekly updates of what I am working on.

The big feature I am currently working on again is parallel IBD. It will improve the sync times of new nodes and nodes that have been offline by downloading and verifying the initial data from multiple peers instead of a single one. This project is pretty large and will require a significant amount of changes to the sync process. Earlier in the week I opened this PR, which streamlines the p2p code and is a preparation for the sync changes. More of these kind of PRs will follow in the coming time.

I am also currently in the process of rewriting the parallel IBD RFC, based on discussions in the PR and some tests I performed. In short, to avoid edge case scenarios and possible attack vectors, the sync will start off with a download and validation of the full utxo bitmap. The rewrite of the RFC is underway and I am expecting to publish a new WIP draft of it in the coming week. I am considering to limit the scope of the RFC to purely the p2p messages, meaning we likely will have a separate one that describes the required changes on the sync flow.

Since the project is fairly large, me and Antioch will both be working on it in parallel from different angles. My current objective is to have a fairly general implementation of generation and validation of the PIBD data to be sent over the wire, which we can then integrate into the node.

All in all I am excited to get this back on the road and implement this! I think it is an important feature and a fun challenge.

-Jasper

14 Likes

This will also be from many peers in parallel?

It grows by one bit for every txo. It’ll probably be a very long time before we would ever get any practical benefit from downloading the utxo bitmap in parallel.

In the proposal the bitmap request will have a parameter to control the number of returned chunks (and an index). This would be the most flexible solution, allowing us to either download the full bitmap from a single peer or maybe in the future get it from multiple. But this is something we can discuss in the upcoming RFC PR.

4 Likes

Updates from the past week:

  • More work on the p2p reader in grin#3433. The actual changes are quite technical and I recommend anyone who is interested in them to read the discussion in the PR. In my opinion the new code is conceptually cleaner, we now have better separation of the different layers and it makes it easier for us to build on top of it. The PR is in final review and should be ready to be merged soon.
  • Progress on the new p2p messages for parallel IBD. I wrote down some of the technical details in a new RFC. Note that this RFC is unfinished and subject to changes, but it reflects my current thinking about the subject. It turns out that the data we need for the different types of MMR is very similar, which allows us to generalize a lot of the code. The implementation of it is underway and I will have a first PR (of multiple) up in the coming week.

Thanks everyone and see you in the next update post

6 Likes