Antioch - Progress Update Q4 2020

Just a quick update to cover the past couple of weeks.

Spent a bunch of time investigating and tracking down an issue related to 4.1.0 when syncing from a 4.0.x node against a more recent 4.1.0 node.

This was fixed here - https://github.com/mimblewimble/grin/pull/3465

Bugfix release here - https://grin.mw/blog/2020/10/08/grin-node-411-released.html

I have also been working on reviewing @jaspervdm PRs related to PIBD.

Big one here - https://github.com/mimblewimble/grin/pull/3453

Also spent some time making progress toward improved “flexible peer filtering” - https://github.com/mimblewimble/grin/pull/3458
We are going to want to be more flexible with selecting peers to sync from once we start supporting PIBD (alongside existing txhashset archive) and this is a step toward supporting this.

7 Likes

Quick update on the past couple of weeks.

Spent time reviewing and design/discussion with @jaspervdm around PIBD changes for HF4 -

I think we are getting close to getting this initial work to a place where we can merge it.

“Flexible peer filtering” is in review and hoping to merge soon -

Spent time looking into what would be required to rebuild our “prune list” when receiving PIBD segments. The “prune list” tracks pruned/compacted sections of the output and rangeproof PMMR structures. This “prune list” is critical for correct reads in to the hash and data files on disk to correctly calculate byte offsets to handle pruned data.

This culminated in a rework of our append() implementation on our PruneList. We not have consistent (also simpler and more efficient) logic for building/rebuilding/reading from disk.

This is also up for review and should be hopefully merged soon.


Also took some time to investigate future support for FlyClient (https://eprint.iacr.org/2019/226.pdf) as we wanted to make sure our proposed DAA changes (https://github.com/mimblewimble/grin-rfcs/pull/61) were compatible with FlyClient.

There are currently some open questions regarding our header MMR and what data we commit to in intermediate nodes.

We are hoping to hear back from Benedikt Bunz regarding this. I would like to avoid finding ourselves in a position where FlyClient support would necessitate a subsequent hardfork post HF4.

6 Likes

Quick status update for the past couple of weeks -

Implementation of the “segmenter” for PIBD is ready to go -

This is dependent on the work by @jaspervdm to build “segments” based on current PMMR state here that we spent time testing/reviewing -

We believe have a solid approach now for extracting segments of chain state that can be requested and validated in parallel during PIBD.

One of the areas of my focus has been to ensure the provider of these segments can generate these efficiently with minimal cost. We take advantage of the append-only properties of our internal PMMR structures and cache a “snapshot” of the UTXO state for the current PIBD sync period (every 12 hours).

Also spent some time adjusting our default node capability flags to facilitate support of both the existing txhashset state sync and the new PIBD p2p messages. We will need to support both of these as we gradually transition across to PIBD post HF4.

Finally I have spent some time reviewing and testing the “late locking” changes to wallet. This is likely to remain “experimental” for now and not enabled by default (enabled via cmd line flags). But this should give us some much need flexibility during tx creation, particularly with slatepack interaction.

6 Likes

Does this mean the new PIBD p2p messages will be enforced (old ones removed) post HF4? My understanding was that if the new p2p messages are ready come the HF, then we don’t have to support the old type of protocol messages forever.

Sorry - that was probably unnecessarily confusing. “Post HF4” as in some time beyond HF4.
We are going to support both message types for a while after HF4, likely several months.
The transition to PIBD will be gradual and there is still much work involved.

We will not support the old txhashset messages forever, but they are not going away any time soon.

Thank you for clarifying.

My question now is: How can this be achieved without a HF5? Support for both types will always have to be there moving forward, until we enforce the new messages with a hard fork, no?

Current IBD and PIBD are doing the exact same validation in the end, they just do it in a different way. There are no changes to validation rules or anything, PIBD just chops the history into smaller chunks so that we can validate small parts instead of one big one which is what the current IBD does. Even if we switched directly from IBD to PIBD, it would just mean that the PIBD and IBD nodes would not be able to bootstrap each other (because they wouldn’t understand the messages of the other node), the network would be fine though

No hardfork required. This is simply support for varying p2p messages at the network level.

But if support for the old messages type is deprecated, then it old nodes are “kicked out” of the network. They can no longer communicate correctly. That sounds equivalent to a hard fork.

I’m not sure I follow.
No nodes are being “kicked out”. The plan is to have a transition period where both existing IBD (txhashset archive) and new PIBD are supported. Eventually we plan to migrate to pure PIBD once the majority of nodes support it.
In no case is this going to be equivalent to a hardfork.

Ok I think I understand now. Should have paid more attention to @oryhp’s comment earlier.
Thank you both for explaining.