Jaspervdm - Progress update thread - March to May 2020

To keep the community updated on my progress, I will be posting weekly in this thread. Here’s what I’ve been doing in week 1:

  • Node API: Don’t error on missing output. When the wallet queried a node to sync the status of its outputs, any output that aren’t confirmed (yet) would show up as an error in the logs. This alarmed users unnecessarily and had the side effect of suppressing real errors such as data corruption. This is now fixed.

  • Wallet: fix sending to https wallets. Unfortunately in v3.1.0 sending to any https listener would give errors. After some investigation it turned out that the underlying problem was caused by a dependency. A new version of the wallet, v3.1.1 will be released shortly which addresses this.

  • Wallet: detect and display reverted transactions. In short: any confirmed transactions that suddenly disappears due to a re-org will now show up as “reverted”. Running the “info” command will also show these funds. If the transaction is included in a block again it will also be updated in the wallet. For more information, see the pull request.

  • Grinscan: linking in/outputs. This work is not done on a public repostitory like the others but since it helps anyone in the Grin ecosystem I’m mentioning it anyway. Since this week Grinscan is performing a rudimentary analysis of mainnet block contents, by comparing the inputs/outputs/kernels with previously broadcasted transactions and grouping them accordingly. This will allow anyone to check if and how much pre-fluff aggregation occurred for their transaction. Here’s an example block without aggregation: https://grinscan.net/block/590958 and an example with aggregation: https://grinscan.net/block/591102

See you next week!

7 Likes

March 9-15
I have spent most of last week refactoring the node p2p communication layer. In v3.1 (and older) the node uses blocking code. It spawns 2 threads for every connection: a reader and a writer. These threads attempt to do locking reads and writes with short timeouts in a loop. This is not very resource efficient, as each thread uses a bunch of memory and possibly wastes cpu cycles when there is nothing to read or write.

I have been working on moving the p2p layer to asynchronous code, using the async/.await functionality that was stabilized in Rust a while ago. This means that there is a multi-threaded runtime that manages all connections and only tries to read/write when necessary. This should reduce resource usage and also opens up possibilities for future improvements, such as having a single thread that is responsible for processing headers/txs/blocks.

At the end of last week I managed to do an initial sync from scratch using async p2p code. However, there is still a bunch of work to be done before the PR can be reviewed and eventually merged. Since there is still a lot of blocking code in the node that interfaces with the p2p layer, it is kind of awkward at times having to juggle synchronous and asynchronous contexts, so I will be spending more time trying to refactor code to simplify this. There are also some stability issues that need to be addressed. I will provide another update next week.

Stay safe everyone

9 Likes

Hi! Can I use this repo safely? https://github.com/jaspervdm/grin/releases

There’s no longer a need to. https://github.com/mimblewimble/grin/releases/tag/v3.1.1-beta.1 now has the compatibility fixes that were originally in Jasper’s repo.

Progress since last update:

  • We have released version 3.1.1 of the node which should fix all “illegal instruction” errors on startup for the linux binaries. If you still get this problem for v3.1.1 (or higher), please open an issue for it on the grin repository
  • I have continued to make improvements to the async p2p code. Managed to fix some issues regarding resource usage, particularly around spawning threads. I’m still seeing some connection stability problems that I am trying to hunt down and fix. They are likely caused by contention on a lock, so I’m looking into possibly replacing some of it with message passing. Once these issues are fixed I will ask the community to help testing it and give me feedback on its stability!

-Jasper

1 Like