Set up monitoring of our p2p network in order to detect any attacks.*

Set up monitoring of our p2p network in order to detect any attacks.
REORG attacks

As for this task, we did work closely with John @joltz setting up a monitoring infrastructure detecting REORG-Attacks.

How did we approach it?
Basically we enabled debug-log on a majority of our Grin-Nodes running on (https://Grinnode.live) and we are searching through these logs for patterns

Patterns:

(REORG!)
depth: 1

Example of a reorg = 1

1520803:2020-09-23T02:17:09+02:00 grinnodeXXXX grin[14409]: 20200923 02:17:09.709 WARN grin_servers::common::hooks - block_accepted (REORG!): 0003cf9fb286 at 883619, (prev: 0002a1ebfcb8 at 883618, prev_head: 000310fdec05 at 883618, fork_point: 0001114640b6 at 883617, depth: 1)

We then setup a special logging server for this, where all the grin-node logs are collected and we can search for these patterns.

Can I have access?
At this moment we are making these attack results only public to the core-dev team and researcher we get recomended from the core-dev team.
Until we integrated this into our https://grinnode.live/faq API we will keep it closed and once its integrated into the API we can make this public to reasearchers.

Example of the running REORG-Attack monitoring system
REORG = 1

Reorgs happening at 1 which is non critical depth

2020-09-28 

Grin-Node REORG! warning system
https://github.com/mimblewimble/grin/pull/3376
provided by https://Grinnode.live

2020-09-28 REORG occurred 6 times. at Depth = 1

Logs:

150762:2020-09-27T08:22:52+02:00 grinnodeXXXX grin[21219]: 20200927 08:22:52.058 WARN grin_servers::common::hooks - block_accepted (REORG!): 0002ca3edfba at 889722, (prev: 00019c078c65 at 889721, prev_head: 00032bfe6d15 at 889721, fork_point: 0001d4c56697 at 889720, depth: 1)
1656751:2020-09-28T04:07:32+02:00 grinnodeXXXX grin[27910]: 20200928 04:07:32.826 WARN grin_servers::common::hooks - block_accepted (REORG!): 00004dd2bf35 at 890911, (prev: 00036f134198 at 890910, prev_head: 0000afcfaf2e at 890910, fork_point: 0000128faf30 at 890909, depth: 1)
2529359:2020-09-28T04:08:00+02:00 grinnodeXXXX grin[23183]: 20200928 04:08:00.972 WARN grin_servers::common::hooks - block_accepted (REORG!): 00004dd2bf35 at 890911, (prev: 00036f134198 at 890910, prev_head: 0000afcfaf2e at 890910, fork_point: 0000128faf30 at 890909, depth: 1)
222899:2020-09-27T08:18:10+02:00 c-nodeXXXX grin[30146]: 20200927 08:18:10.300 WARN grin_servers::common::hooks - block_accepted (REORG!): 0002ca3edfba at 889722, (prev: 00019c078c65 at 889721, prev_head: 00032bfe6d15 at 889721, fork_point: 0001d4c56697 at 889720, depth: 1)
542112:2020-09-27T11:49:21+02:00 c-nodeXXXX grin[14791]: 20200927 11:49:21.388 WARN grin_servers::common::hooks - block_accepted (REORG!): 00019f4b2a0b at 889938, (prev: 0000e1069840 at 889937, prev_head: 0000a2ac0d26 at 889937, fork_point: 000026c49e95 at 889936, depth: 1)
1398771:2020-09-27T20:08:59+02:00 c-nodeXXXX grin[14791]: 20200927 20:08:59.684 WARN grin_servers::common::hooks - block_accepted (REORG!): 00033f2a1838 at 890432, (prev: 0001ec803ba3 at 890431, prev_head: 00037ecae1b5 at 890431, fork_point: 00036ed09361 at 890430, depth: 1)

When a reorg attack happens or at least a REORG bigger (>1) happening the system will alert as follows:
(this was only a test from us no real REORG attack happened here)

2020-09-26 

Grin-Node REORG! warning system
https://github.com/mimblewimble/grin/pull/3376
provided by https://Grinnode.live

2020-09-26 REORG occurred 19 times. at Depth = 1
WARNING: one or multiple REORGS! happened at depth: 5
Logs:

115994:2020-09-16T22:13:22+02:00 grinnodeXXXXX grin[28065]: 20200916 22:13:22.096 WARN grin_servers::common::hooks - block_accepted (REORG!): 00004bc76086 at 874794, (prev: 0001f23dc8b4 at 874793, prev_head: 0000048a71c2 at 874793, fork_point: 00009706826c at 874792, depth: 1)
280426:2020-09-17T10:44:59+02:00 grinnodeXXXX grin[22931]: 20200917 10:44:59.417 WARN grin_servers::common::hooks - block_accepted (REORG!): 00040c81e7f4 at 875503, (prev: 000572810f84 at 875502, prev_head: 00000afbaace at 875502, fork_point: 00054b6c03f1 at 875501, depth: 1)
513607:2020-09-18T04:30:53+02:00 grinnodeXXXX grin[24814]: 20200918 04:30:53.299 WARN grin_servers::common::hooks - block_accepted (REORG!): 0002076b0f32 at 876579, (prev: 0001e9e8b9d2 at 876578, prev_head: 000140786fee at 876578, fork_point: 000075561778 at 876577, depth: 1)
568386:2020-09-18T08:41:37+02:00 grinnodeXXXX grin[24814]: 20200918 08:41:37.602 WARN grin_servers::common::hooks - block_accepted (REORG!): 0002ccefa506 at 876819, (prev: 0000d7a858a1 at 876818, prev_head: 00001e3ac750 at 876818, fork_point: 0001c1238422 at 876817, depth: 5)
223300:2020-09-16T22:13:45+02:00 grinnodeXXXXX grin[32647]: 20200916 22:13:45.608 WARN grin_servers::common::hooks - block_accepted (REORG!): 00004bc76086 at 874794, (prev: 0001f23dc8b4 at 874793, prev_head: 0000048a71c2 at 874793, fork_point: 00009706826c at 874792, depth: 1)

Some minor problems we encountered during enabling debug-log.
Based on the grin-server toml you have two possibilities activating debug-log:

1. #log level for stdout: Error, Warning, Info, Debug, Trace
stdout_log_level = "Debug"

2. #log level for file: Error, Warning, Info, Debug, Trace
file_log_level = "Debug"

of couse, one / both have to be enabled “set to true”.

When logging to file on a Linux system and trying to use Syslog/rsyslog to forward it to a central syslog server it not working. Therefore we figured out, you had to enable the “stdout_log_level” which is talking to the syslog/rsylog in order to forward them to the special central-logging server. (whink)

7 Likes

This is great! My only suggestion is that this will detect the issues after they happen. It may be very useful to have a calculator take into consideration the current network hash rate as well as the rentable hash available on the market to determine if an attack is economically sound and for how long this would be the case. The latter of which being very important for guiding the number of confirmations people should consider prior to feeling reasonably safe. Adding this proactive measure may help ensure the reactive measures are never realized.

Keep up the great work Mike!

edit
So there is no re-inventing the wheel, this website does it https://www.crypto51.app/ and here is their github https://github.com/tdickman/crypto51 . They used to have grin on there and I posted an issue with how they were not taking into account both algos for the cost to attack.

1 Like

Great work!

A possible alternative approach to analyzing log files would be to use the node webhooks. In the node config, there is the following section:

[server.webhook_config]
#block_accepted_url = "http://127.0.0.1:7701/acceptedblock"

If this line is uncommented, the node will send a HTTP POST request to the provided URL with the full block included. This could be used to detect re-orgs, but requires some state tracking. If we receive a block at height X with hash H and later a block at height X with hash H', we know a re-org has occured. From the top of my head I don’t remember the order in which the re-org blocks would be pushed to the webhook (or they even might be pushed in parallel), but in principle this information can be used to determine the depth of the re-org as well.

If the current approach of analyzing log files works without any issues, I recommend sticking with it for now, as it is probably not worth the effort to switch. But it is good to keep in mind of this possibility, if there are any problems in the future.

1 Like

Just looked at the webhooks and we have the summary info available via the BlockStatus in the webhook code, we just don’t actually expose it via the webhook call itself.
I think we expose “depth” in the webhook but not the richer “fork_point” etc.