# \[Antioch\] Status Update Q1 2021 **URL:** \ **Category:** Uncategorized\ **Created:** [February 14, 2021, 5:08pm UTC](https://forum.grin.mw/t/antioch-status-update-q1-2021/8407 "2021-02-14T17:08:29Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![antioch](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.grin.mw/antioch/32/45_2.png) [@antioch](https://forum.grin.mw/u/antioch)\ **Post date:** [February 14, 2021, 5:08pm UTC](https://forum.grin.mw/t/antioch-status-update-q1-2021/8407/1 "2021-02-14T17:08:29Z")
Funding request for reference - > [@Request for funding @antioch, Feb-Mar 2021](https://forum.grin.mw/t/request-for-funding-antioch-feb-mar-2021/8334): > > [This is all with the understanding that I am walking straight into the middle of various active conversations around the funding of contribution to the grin project. I am only speaking for myself here.] January was an unpaid “sabbatical” and I was not actively involved or contributing in any meaningful way. I was originally going to request a shortened two month period of funding for Feb/Mar to sync back up with the quarterly schedule. Given the somewhat heated and ongoing discussion around d… I have spent the past couple of weeks getting back up to speed after taking some time off in January. I am focused on “block archival support” currently and have made progress with some ancilliary work necessary to support this. Tracking issue for “block archival support” - > > > This is a tracking issue for the sub-tasks and identified requirements. > Will b…e updated as investigation continues. > > Related https://github.com/mimblewimble/grin/issues/3092 > > \---- > > \* Introduce new \`BLOCK\_HIST\` capability > \* Enable \`BLOCK\_HIST\` if config \`archive\_mode=true\`. > \* This should flow through to the capabilities we advertise to other peers. > \* https://github.com/mimblewimble/grin/issues/3562 > \* Add ability to query connected peers by \`BLOCK\_HIST\` capability. > \* We do not want to waste network resources trying to sync historical blocks from non-archive peers. > \* \[tbd\] > \* Implement logic to suppress state sync and initiate full block sync from height 1 (if archive mode enabled) > \* \[tbd\] > \* Performance issues during sync > \* current implementation has issues when header height is \`1\_000\_000\` and block height is at \`1\` etc. > \* https://github.com/mimblewimble/grin/issues/3554 > \* Issue with "abusive" behavior when syncing from a small number of peers > \* we get banned for requesting lots of blocks or we ban the peer because \_we\_ asked for too many blocks > \* https://github.com/mimblewimble/grin/issues/3553 > \* Related https://github.com/mimblewimble/grin/pull/3564 > \* Current behavior is to trigger full chain compaction after syncing to prune the MMR structures. > \* This is not designed to handle compaction over full history and is very inefficient. > \* We should consider periodic chain compaction during sync (currently explicitly suppressed as part of fast sync). > \* https://github.com/mimblewimble/grin/issues/3567 * * * Investigation into block archival identified a performance issue related to block sync and how we identify “missing” blocks that need requesting from peers - > > > We identified a performance issue during sync, specifically when we are requesti…ng full blocks with a large difference between the height of the current header and the height of the full block. > Specifically this is exacerbated during an experimental "full archive sync" when header height is \`1\_073\_555\` and block height is \`1\`. We iterate back over the headers to identify which full blocks are missing, so we know which to request. This is \_slow\_ when we iterate over a million headers. > > Related https://github.com/mimblewimble/grin/issues/3552 > > \---- > > We know the height of the current chain head and we know all headers beyond this (full blocks not yet sync'd). > It should be fairly easy to rework this logic to iterate \_forwards\_ from the current chain head (last full block), taking advantage of \`get\_header\_by\_height()\` to do this. > > While this is particularly noticeable when syncing historical blocks I suspect the performance issue is still present during a regular "fast sync" for the most recent 48 hours of full blocks. There is no need to continually keep iterating back over all the headers in this "recent history" time period. > > Planning to fix this issue as a standalone task. This is resolved here - > > > Resolves #3554. > > When we transition from "header sync" to "body sync" we ident…ify missing blocks based on the delta between the chain of headers and the current chain of full blocks. i.e. We sync headers and then we go request the full blocks for each of those headers. > There is some additional complexity where we support a fork between the header chain and the full block chain as one is not always a subset of the other. And to do this we identify the "fork point" which is the last common header between header chain and full block chain before they begin to diverge. > > \---- > > Fork example - > > Headers: \`A -\> B -\> C' -\> D -\> E\` > > Full blocks: \`A -\> B -\> C\` > > Header sync gave us a chain of headers building on \`C'\` while our local node was previously aware of \`C\`. > We need to "rewind" to the fork point \`B\` immediately prior to \`C\` and \`C'\` and request missing blocks \`\[C', D, E\]\`. > > \---- > > The existing implementation rewinds back from the head of the header chain to identify both the fork point and a set of missing block hashes. This is fine if the delta between header chain and full block chain is small but is not suitable if the header chain must be rewound a significant number of blocks. > > It fails badly in the "full archval node sync" scenario where we must rewind from height ~1,500,00 back to 1. > > \---- > > The PR introduces a more efficient approach - > > 1. identify height of full block chain > 2. increase this height by the number of blocks we will request in parallel > 3. lookup this "max" header directly on header chain via \`get\_header\_by\_height()\` > 4. Iterate back from this "max" header to the fork point. > > This places an upper bound on the iteration required as we can lookup headers directly based on height. > This greatly reduces the amount of iteration required when identifying missing blocks to sync. > > \---- > > As part of this rework \`check\_txhashset\_needed()\` was refactored, splitting out \`get\_fork\_point()\` and \`check\_txhashset\_needed()\` into separate fns. > This allowed the implementation to be simplified significantly. * * * Also uncovered an issue with banning peers when requesting too many blocks (i.e. during archival sync) due to “abusive” peer behavior (p2p msg rate limit exceeded). > > > Exploration/investigation into full block archive sync identified an issue relat…ing to banning abusive peers. > > Related https://github.com/mimblewimble/grin/issues/3552 > > We ban a peer if we receive (or send) excessive numbers of messages to/from that peer within a 60s time period. > > The limit is 500 messages within 60s. If we see more than this then the peer is flagged as abusive and eventually banned. > > It makes sense to monitor this for inbound received messages but I'm not convinced this is the right thing to do when we initiate requests by sending messages to peers. > Specifically this is an issue if we request large numbers of blocks from a small number of peers (say those that support full archive mode) as we can easily hit the 500 msg limit when requesting historical blocks. > > Proposal is to limit the abusive peer handling to \_inbound\_ msgs only. If we engage in abusive behavior when sending messages to a peer then they will react accordingly. We do not need to ban them for something we are responsible for. > > We would need to monitor (and rate limit) our outbound msg rates to ensure we did not exceed peer limits. > We can do this by inspecting the \`Tracker\` (msg rates) associated with each peer and potentially rate limit by introducing a small delay across large numbers of outbound "get\_block" requests. As part of block archival support this has been reworked to introduce “rate limiting” to maintain acceptable p2p message rates without exceeding the limit and risking getting banned. Fix is here - > > > Resolves https://github.com/mimblewimble/grin/issues/3553 > > We treat peers as a…busive if inbound or outbound msg rates exceed 500/min. > > This PR modifies this to only consider inbound msg rate. > We do not want to treat a peer as abusive if \_we\_ are responsible for sending lots of p2p messages. > > Add basic "rate limiting" to \`write\_message()\` by inspecting the previous timestamp tracked by the tracker. > We ensure 150ms has elapsed based on previous sent message before attempting to send another. > In normal operation this is rarely the case. We occasionally send a ping and a block header close together and these are now separated by 150ms. > The scenario where this \_does\_ come into play is during "fast sync" when requesting large numbers of full blocks from a small number of peers. Specifically "block archival sync" against archival nodes. > We no longer flood the peer connection with hundreds of block requests and these are now spaced out to avoid hitting the abusive peer limit on the receiving end. > > \---- > > As part of this PR cleaned up the readonly usage of \`tracker\`. > The following fns have been removed, replaced with exposing a simple \`tracker()\` accessor - > \* \`last\_min\_sent\_bytes\` > \* \`last\_min\_received\_bytes\` > \* \`last\_min\_message\_counts\` > > For example - > > \`\`\` > let received = peer.tracker().received\_bytes.read().count\_per\_min(); > \`\`\` > > This makes the read locks more explicit and removes a case where we take multiple read locks in one go. > > \---- > > Introduced \`elapsed\_since\_last\_msg()\` to \`RateCounter\` for convenience. > This simplifies the implementation of the rate limiting logic. * * * Also tracked down the “block not found” error that we occasionally encounter during node restart. This is actually related to some PIBD initialization code and we were not fully accounting for a likely edge case here related to how we rewind based on the most recent archival period (every 720 blocks). The fix is minor and went ahead and merged this. > > > This PR fixes a "cannot find block" error that can occur during node startup. > W…e have some code that runs during node initialization that exercises the PIBD segmenter as we wanted to keep this code "in use". > There is an edge case during startup that we did not account for - if the node is shutdown while it is in the process of syncing then on next startup the chain can find itself in a state where traversing back to the "archive header" (our 720 block archival period) can fail as we do not have the full set of blocks for this time period yet. > This PR simply ensures the code that touches the segmenter does not fail in this situation (during startup). > > This is related to a jump from one 720 block archive period to the next due to chain height, during sync. It is possible for the jump to result in "missing" blocks as we are still syncing. This is not a problem with data integrity or missing data - its just that we were making a bad assumption during node initialization without accounting for this scenario. > > Resolves https://github.com/mimblewimble/grin/issues/3516 * * * I have some code running locally that successfully begins requesting historical blocks (currently from a known set of “preferred” archival peers). I am hoping to get some of this work into a PR’able state over the next couple of days. Roughly speaking it suppresses the “state sync” (txhashset.zip) when running in “archival mode”, forcing the block sync to begin syncing missing blocks from height 1 onwards. Now that we can successfully rate limit the p2p requests we can simply continue block sync for all missing blocks up to the current height. To release this fully we will need to support peer selection/filtering based on archival node support so there is still some work to do there. I suspect we may still want to take advantage of “preferred peer” support to ensure we have good connectivity when starting a new archival node up. But I think that’s acceptable if a node opts-in to archival mode and wants to reliably sync full history - you need to know at least a couple of archival nodes to sync from. * * * I am aiming for some kind of “beta” of this in the next week or so and hopefully we can be doing some wider testing of this functionality soon after that. ---
**Author:** ![Chronos](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.grin.mw/chronos/32/153_2.png) [@Chronos](https://forum.grin.mw/u/Chronos)\ **Post date:** [February 14, 2021, 5:39pm UTC](https://forum.grin.mw/t/antioch-status-update-q1-2021/8407/2 "2021-02-14T17:39:56Z")
Thank you. * * * ---
**Author:** ![dog](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.grin.mw/dog/32/2230_2.png) [@dog](https://forum.grin.mw/u/dog)\ **Post date:** [February 16, 2021, 8:59pm UTC](https://forum.grin.mw/t/antioch-status-update-q1-2021/8407/3 "2021-02-16T20:59:00Z")
Thank you so much for this status update! ---
**Author:** ![antioch](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.grin.mw/antioch/32/45_2.png) [@antioch](https://forum.grin.mw/u/antioch)\ **Post date:** [February 24, 2021, 9:54am UTC](https://forum.grin.mw/t/antioch-status-update-q1-2021/8407/4 "2021-02-24T09:54:58Z")
Unsurprisingly this update is late… Continuing to make progress on “full archival sync”. Tracking issue here - > > > This is a tracking issue for the sub-tasks and identified requirements. > Will be updated as investigation continues. > Related #3092 > > Introduce new BLOCK\_HIST capability > Enable BLOCK\_HIST... * * * I have a full archival sync running locally and it runs to completion with some caveats. Hoping to have this out for a wider beta release later this week. The big caveat is the chain compaction process is slow when used alongside archival sync. > > > Experimenting with "full archival sync" has highlighted an issue with chain "compaction". > A full compaction is kicked off once sync has completed.... This led down a bit of a rabbit hole exploring various options for making chain compaction more efficient (or doing it less often, or limiting volume of data being compacted etc.) > > > > This is still a continuing investigation. * * * Plan for the next few days is to park the compaction investigation and move “archival sync” forward so others can test it out. And then pick up the compaction investigation after that. Ideally archival sync runs (albeit slowly) without a significant pause at the end for chain compaction. * * * ### Why even run chain compaction if node is “archival”? What is there to compact? An archival node maintains a full block history in the local db. Compaction allows us to maintain the PMMR data structures in an efficient way (we can prune and compact historical spent outputs). The block history means this PMMR data can still be pruned and it is desireable to do so as this is effectively redundant data on an archival node (and takes up significant disk space beyond the blocks db). ---
**Author:** ![Anynomous](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.grin.mw/anynomous/32/2928_2.png) [@Anynomous](https://forum.grin.mw/u/Anynomous)\ **Post date:** [February 24, 2021, 11:38am UTC](https://forum.grin.mw/t/antioch-status-update-q1-2021/8407/5 "2021-02-24T11:38:51Z")
> [@antioch](#): > > The block history means this PMMR data can still be pruned and it is desireable to do so as this is effectively redundant data on an archival node (and takes up significant disk space beyond the blocks db). As long as it is less than Bitcoin’s full node, you will not hear me complain. Archive nodes are probably only for those interested in investigating the blockchain for which they will have space available. E.g. I would probably export/convert it to a graph database, not exactly memory saving 😛 ---
**Author:** ![antioch](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.grin.mw/antioch/32/45_2.png) [@antioch](https://forum.grin.mw/u/antioch)\ **Post date:** [March 7, 2021, 3:33pm UTC](https://forum.grin.mw/t/antioch-status-update-q1-2021/8407/6 "2021-03-07T15:33:16Z")
Quick status update/progress report. Continuing to make progress on “archival sync” support. As part of this I cleaned up and improved our peers\_allow and peers\_preferred configuration options to make these more robust and useful during server restart. > These configuration options have been invaluable during “archival sync” development and testing and will continue to be useful on `mainnet` for early adopters of “archive mode”. We have a “bootstrapping” problem here as we can only archival sync from other archive nodes and we need to both identify these and direct archive block requests to them robustly. Allowing nodes to configure lists of “preferred” peers is very useful here as we can increase connectivity with a known population of archive nodes. Over time this will become less critical. * * * It turns out legacy `v2/v3` block serialization support does not play nicely with archive sync. While an archive node maintains a full history of full blocks it still cannot rewind back beyond the horizon. And there is no way to reconstruct a `v2` full block based only on a `v3` block. We no longer need to support `v2` blocks now that we are past the final HF4. PR to clean this up and remove the old support - > This makes archive nodes quietly drop legacy block requests from old nodes on the network (pre-HF4, even pre-HF3) that still hang around periodically asking for old blocks. I think ideally we will start more aggressively banning these stale nodes but for now we can just ignore them. * * * The above PRs are the final changes needed to allow us to enable archival sync support. The task of enabling this is in PR - > Proposing to discuss actual rollout of this as part of a `5.1.0` release, potentially in the next couple of weeks. This would be dependent on commitment from enough archive node operators willing to upgrade shortly after the release to help with the “bootstrapping” issue (discussed in the PR). * * * Some people (specifically @quentinlesceller) are seeing _really_ slow archive sync performance. I suspect investigating this is outside the scope of this initial PR and looks limited to specific local environment - slow disk specifically. But we do need to investigate and there is definitely room to improve performance here over time. Recommendation would be to only run an archival node if you have reasonably fast hardware and a decent disk (and sufficient disk space). ---
**Author:** ![antioch](https://yyz2.discourse-cdn.com/flex036/user_avatar/forum.grin.mw/antioch/32/45_2.png) [@antioch](https://forum.grin.mw/u/antioch)\ **Post date:** [April 6, 2021, 3:43pm UTC](https://forum.grin.mw/t/antioch-status-update-q1-2021/8407/7 "2021-04-06T15:43:43Z")
Seriously late posting a status update, as I have not posted anything since before the emergency HF… Most of my focus has been on post-HF mitigation and improvements. A lot of it is summarized here - > > > The following PRs were merged to current/5.0.x as part of the emergency hardfork on 20210318 - > \#3598 "Removing verifier caches" > \#3600 "Rewind bad... Currently testing a “manual” approach to invalidating headers and “resetting chain state” here - > Which in conjunction to the improvement with the “sync\_head” tracking should get us to a far more stable place if we ever need to navigate a large fork (emergency or otherwise) again in the future - > Q1 came to a close a week ago and my funding period has now expired. I dropped the ball on keeping this running uninterrupted and totally forgot it was end of March until we actually hit the end of March… On my todo list is getting it together and posting a follow on funding request for Q2 to the forum. I hope to have this posted by the end of the week, hopefully in time to discuss in the governance meeting next week. I’m planning to write at least a minimal retrospective of my attempt at a “deliverable based funding request” which did not necessarily go 100% to plan (given the intrusion from the emergency HF etc.)