Funding proposal 1: Parallel Initial Block Download

Governance
1

This is a new format for governing funding, which I call funding proposals. The purpose is to first get sufficient community support, discussing deliverables and the value for the community and only move to a funding request or bounties if there is sufficient support. This incorporates some of the feedback on Grin Governance from this discussion. This new format also takes away some of the work for developers to ‘sell themselves’ via Funding requests, which can be a frustrating experience, right @davidtavarez. Note that even if a funding proposal gets sufficient support, the question is if a developer wants to take on the implementation, whether as funding request or a bounty.

Funding proposal: PIBD implementation in Grin++

What: Parallel Initial Block Download (PIBD)

PIBD is a new messaging system that allows you to download blocks simultaneously from many peers, check them, verify them, and rebuild your history. PIBD is truly decentralized, secure and robust! There is some additional overhead in data transferred, so currently PIBD is not a faster way to download the blockchain. However, as anyone knows who ever used Torrents or other decentralized downloading systems, potentially downloading from many peers in parallel can be used to make downloading faster, even with added overhead, since most users have a lower upload speed than download speed. So PIBD might in the future lead to lower initial syncing time.

Why: Currently both grin-wallet and Grin++ download the blockchain up to the horizon as a .zip file.

This is problematic for various reasons. The biggest issue is that the process is unstable and error prone. Your node asks a single peer to download the .zip file. If that peer goes offline or is malicious, he or she can interrupt your download, corrupt your download, or keep you waiting for very long!
Anyone who synced his node or wallet multiple times knows from experience that the experience is highly varying, sometimes the blockchain syncs fast, sometimes incredibly slow, and sometimes it requires manual restarting the wallet or sync process. In other words, the initial sync as .zip file makes a horrible user experience. Secondly, it is not really decentralized to download from a single peer.
PIBD will solve the above problems and is already on test-net and soon on main-net in grin rust :tada:. However, the minority of nodes out there run Grin++, meaning grin rust nodes would only be able to use PIBD to download from rust nodes which hampers its effectiveness and hampers decentralization. Even worse, Grin++ would still be stuck with the old and poor way of syncing using a .zip file. Therefore implementing PIBD in Grin++ is essential for the whole network to take full advantage of PIBD.

What we need from you:

We need your opinion, (dis-)likes, questions (also dumb ones) as community members. The purpose is for these requests is to be shared on social media, start a discussion on the value of funding the implementation of features for the community, and hopefully collect sufficient community support before initiating any funding requests. There is no fixed threshold since that would give spammers ultimate power over Grin governance. Your opinion and arguments are counted, and weight based on their validity, sound argumentation and your historical contribution to Grin.

Find here a better explanation of PIBD by @Yeastplume

Or here a more detailed technical explanation by @Yeastplume

6 Likes
2

:point_up:
The above is a funding proposal. Anyone can make one, feel free to do so :grin:!
Here I want to express my opinion and support for this proposal. In my personal opinion, implementing PIDB in Grin++ is the single most important thing for the project and the community , for a) keeping the node network unified, b) for making the network more decentralized and c) for improving the user experience for users who first run their node and wallets. Like slate-packs, PIDB is one of those changes that might appear too technical for some users to truly understand or appreciate, but one that is very important for the project. I hope my explanation might help in clarifying the value of PIDB, feel free to ask questions or add explanation of your own to make it more clear.

2 Likes
3

My position is still the same:
Pay after work is done, not in front.
Actual development cost is zero and you know it :slight_smile:

You can even afford @davidtavarez to choose the price, but my opinion - bounty should be created and putted here: Grin CC

5 Likes
4

i support it also. But i want to see if PIBD rust is running stable first and this proposal to be as a bounty.

4 Likes
5

After updating node from non-PIBD to PIBD it takes 5 minutes to start getting leaves at TxHashsetPibd status after syncing headers at Mobile: after 0 I got 17798075 from 17843253, then process is stucking at 17798075.

I noticed this process is much faster on Desktop, takes only some seconds to get 17798075 after 0. I am using same chain state/peers, so I guess its about algo complexity, will keep discovering this problem.

I am using release cargo build.

4 Likes
6

I think it will be better to make PIBD sync optionally at beginning, can be option at grin-server.toml config.

1 Like
7

I support this proposal. I think it is an important step forward in decentralization and I think it’s true we can’t really ignore grin++ any more than we would ignore grin-wallet. Infact, grin++ is arguably the more popular of wallets and this should be prioritised.

8

I think it would be best to make PIBD default if it is stable on mainnent but keep the .zip a an option in the config files and in the code as backup sync method. Perhaps for some low computing power devices, the PIDB sync method is slower due to the calculations it involves. I can image the .zip files can in the future be modified (breaking up the chain into a few.zip files) to sync new archive nodes with existing archive nodes.

Regarding funding as bounty or as funding request, I propose a hybrid solution, see below the explanation. I think this address all the concerns and covers most risk to the community funds.
@Cobragrin, @ardocrat, @l33d4n

Bounties with milestones and conditional upfront payment

This solutions is similar to how @Yeastplume is paid and to what we did with CoinSwap bounties (milestones 1,2 and 3)

  1. We break down a funding proposal into deliverables with a bounty for each milestones/deliverable
  2. We can pay milestone bounties upfront so developers do not get into problems with paying their bills
  3. To have access to upfront payment there are two conditions a) all the deliverables for the previous funding requests must be finished, b) upfront payment for a milestone/deliverable is only accessible to developers and community members with a proven track-record such as @davidtavarez and @Yeastplume .

Each time a milestone is reached and finished, we can pay the next milestone bounty upfront if these conditions are met. I think this would be the best compromise between being practical and considerate to our long term developers while strictly managing and covering risks for community funds. Let me know what you think, would this be a good solution?

1 Like
9

This is good idea.

I think motivation is lowering at this case. Full time devs are usually getting their salaries in the middle/end of month, cause they know if they will not work they will not receive salary. I think it can be no problem here if tasks will be divided in small parts.

It creates group of favourite persons with special privileges. I think it will be more fair for all people who is going to be paid by CC Fund to follow same rules and be equal. We already had problems with track-record of last 3 months, so we learned from this.

I still think we need to form bounties as it was initially at CC, so everybody can apply to help, take free tasks and so on, I just don’t support idea for only 1 dev per project, collaboration is a key. For example, @david can help @davidtavarez and get paid too or any other interested C++ dev.

2 Likes
10

I think this should be discussed widely between developers and community leaders. If it hinders end users…

What concerns me is risking Grin development pace. Community/ OC funds should be destroyed if it is needed for Grin well- being.

:+1:

Yes, rules must be equal but apply to anyone. What causes concern in this scheme is how difficult it is to bring in new developers… It is obvious @yeastplume cant reach everywhere, working solo for ex. Rust is the main repo.

Many bug fixes remain untouched for years due to lack of contributers. Will this solve our problem, or it will solidify current situation?
''we need man power ‘’ since 2020 ?

1 Like
11

In my case PIBD started immediately but stucked around 65%. I quit node and started again. It synced succesfully but full sync takes longer.

12

Regarding payment upfront, the rule applies to everyone so no favoritism, simply reputation based. First prove yourself before being able to get paid upfront for a milestone and secondly you have to finish all previously paid deliverables to keep that proven reputation

This is why condition 3a) is there, finish deliverables from earlier funding in order to keep your reputation.

Indeed, meaning paying afterwards would not be a problem if deliverables would be very small. We know from experience we do not have the manpower or know-how to use such fine grained governance nor to make so many frequent payments as CC. What you have in mind would work if we would have a team of developers and someone as team leader with knowledge who is paid to estimate hours, appropriate bounties for all tasks and deliverables.

Exactly! The biggest thread to this project is not having minor funds at risk. The biggest thread is and probably will be for some time the lack of manpower and developers.
If we would end up with a perfect governance model with zero risk to funds and zero developers :grimacing:… that is a scenario we should avoid.

2 Likes
13

Agree, reputation and years involvement in project should be taken into consideration and allow for upfront payments. Once a developer has completed a number of bounties, bug fixes and/or has demonstrated to be a fixture in the community then a process should be accessible for them to receive upfront payment.

14

i also support this proposal. as a user i always had issue to sync my grin node.

for those who don’t understand what it is, let me give you a more simple example

back in the days, i used to download movies from file hosting servers,. the internet quality was terrible, each time the connection drops, i had to go all over again.

this is exactly what is happening today when syncing a node. you rely on one single peer at a time, and if there any issue you need to go all over again.

PIBD is like downloading a movie using torrent, you can rely on multiple peers. the “node” is like downloaded chunk by chunk. this won’t make the process faster, but more reliable and way less frustrating

a lot of users think that grin is broken because they can’t sync their node and obviously they can’t use the app. PIBD will clearly improve the user experience and gain more users :grinning:

there is a lot to learn about grin, like interactivity, sync/async transfers etc, at least we need to make sure that the UX is at its best

Regarding this funding model, i support this

2 Likes
15

I never had such problems for last 3 months. I am syncing my node as developer frequently.

Problem is still here even with PIBD: when you will stop syncing at last step (TxHashsetSave) or at any validation step after TxHashsetPibd status you will need to download your state at TxHashsetPibd from where you started before again, so I don’t see how current implementation is resumable tbh. Same case I had with old zip method and TxHashsetDownload status. This problem can be easy reproduced.

Problems are known: Grin++ can not find peers and crashing at last step on Android, I am just not sure if PIBD will help in this, but we can try. I think some PIBD optimizations are needed at Grin Rust first.

1 Like
16

i was referring to syncing a node on grin ++

when testing the last release 5.2.0 alpha 2, i can easily quit and resume syncing later at the same entry.

Before

Capture d’écran 2023-06-15 à 11.10.03
Capture d’écran 2023-06-15 à 11.10.031562×50 19.3 KB

After
Capture d’écran 2023-06-15 à 11.13.52
Capture d’écran 2023-06-15 à 11.13.521536×42 18.5 KB

sure, we need more testers

1 Like
17

I caught a bug on Desktop. TxHashsetPibd synchronization step is switching to non-PIBD TxHashsetDownload in parallel, then coming back to TxHashsetPibd.

And just stucked here:

Screenshot_20230615_144833
Screenshot_20230615_1448331164×277 118 KB

At logs I am seeing:

Screenshot_20230615_144709
Screenshot_20230615_1447091169×216 105 KB

I had such bug on Mobile before too. I guess I will open issue at Github.

2 Likes
18

I synced using PIBD 4 times now. Each time without any hick-ups. Can this issue be related to you testing on a mobile @ardocrat, or did you also encounter issues when using a computer?
In any case, we need to figure out why in your case it crashed or was hanging.

19

Yes, I am testing only on Desktop now, mentioned issue above.

1 Like
20

My guess would be that this happens when a new horizon header is defined. The horizon header is 48 hours back in time, but is updated every 12 hours. So it would be important to look at the time, probably it coincides with this 12 hourly horizon update.

Just a wild guess, but it happens at block header height 2414175 according to the log.

2414175/(1260) = 3353.02083333, so 0.0212*60 = 14.4 minutes after a horizon update. Can be coincidence, most likely it is not. Probably the node gets into some conflict state since it starts to sync using the txHAshsetDownload since it updated all blocks except the last 48 hours using PIDB, but before it has downloaded the next 12 hours using the TxHashsetDownlaod method, the horizon is updated.
Hence, switching back to PIDB.

If you post a link to the issue on GitHub, I will post my ‘hunch’ there.

1 Like