Set up monitoring of our p2p network in order to detect any attacks.
REORG attacks
As for this task, we did work closely with John @joltz setting up a monitoring infrastructure detecting REORG-Attacks.
How did we approach it?
Basically we enabled debug-log on a majority of our Grin-Nodes running on (https://Grinnode.live) and we are searching through these logs for patterns
Patterns:
(REORG!)
depth: 1
Example of a reorg = 1
1520803:2020-09-23T02:17:09+02:00 grinnodeXXXX grin[14409]: 20200923 02:17:09.709 WARN grin_servers::common::hooks - block_accepted (REORG!): 0003cf9fb286 at 883619, (prev: 0002a1ebfcb8 at 883618, prev_head: 000310fdec05 at 883618, fork_point: 0001114640b6 at 883617, depth: 1)
We then setup a special logging server for this, where all the grin-node logs are collected and we can search for these patterns.
Can I have access?
At this moment we are making these attack results only public to the core-dev team and researcher we get recomended from the core-dev team.
Until we integrated this into our https://grinnode.live/faq API we will keep it closed and once its integrated into the API we can make this public to reasearchers.
Example of the running REORG-Attack monitoring system
REORG = 1
Reorgs happening at 1 which is non critical depth
2020-09-28
Grin-Node REORG! warning system
https://github.com/mimblewimble/grin/pull/3376
provided by https://Grinnode.live
2020-09-28 REORG occurred 6 times. at Depth = 1
Logs:
150762:2020-09-27T08:22:52+02:00 grinnodeXXXX grin[21219]: 20200927 08:22:52.058 WARN grin_servers::common::hooks - block_accepted (REORG!): 0002ca3edfba at 889722, (prev: 00019c078c65 at 889721, prev_head: 00032bfe6d15 at 889721, fork_point: 0001d4c56697 at 889720, depth: 1)
1656751:2020-09-28T04:07:32+02:00 grinnodeXXXX grin[27910]: 20200928 04:07:32.826 WARN grin_servers::common::hooks - block_accepted (REORG!): 00004dd2bf35 at 890911, (prev: 00036f134198 at 890910, prev_head: 0000afcfaf2e at 890910, fork_point: 0000128faf30 at 890909, depth: 1)
2529359:2020-09-28T04:08:00+02:00 grinnodeXXXX grin[23183]: 20200928 04:08:00.972 WARN grin_servers::common::hooks - block_accepted (REORG!): 00004dd2bf35 at 890911, (prev: 00036f134198 at 890910, prev_head: 0000afcfaf2e at 890910, fork_point: 0000128faf30 at 890909, depth: 1)
222899:2020-09-27T08:18:10+02:00 c-nodeXXXX grin[30146]: 20200927 08:18:10.300 WARN grin_servers::common::hooks - block_accepted (REORG!): 0002ca3edfba at 889722, (prev: 00019c078c65 at 889721, prev_head: 00032bfe6d15 at 889721, fork_point: 0001d4c56697 at 889720, depth: 1)
542112:2020-09-27T11:49:21+02:00 c-nodeXXXX grin[14791]: 20200927 11:49:21.388 WARN grin_servers::common::hooks - block_accepted (REORG!): 00019f4b2a0b at 889938, (prev: 0000e1069840 at 889937, prev_head: 0000a2ac0d26 at 889937, fork_point: 000026c49e95 at 889936, depth: 1)
1398771:2020-09-27T20:08:59+02:00 c-nodeXXXX grin[14791]: 20200927 20:08:59.684 WARN grin_servers::common::hooks - block_accepted (REORG!): 00033f2a1838 at 890432, (prev: 0001ec803ba3 at 890431, prev_head: 00037ecae1b5 at 890431, fork_point: 00036ed09361 at 890430, depth: 1)
When a reorg attack happens or at least a REORG bigger (>1) happening the system will alert as follows:
(this was only a test from us no real REORG attack happened here)
2020-09-26
Grin-Node REORG! warning system
https://github.com/mimblewimble/grin/pull/3376
provided by https://Grinnode.live
2020-09-26 REORG occurred 19 times. at Depth = 1
WARNING: one or multiple REORGS! happened at depth: 5
Logs:
115994:2020-09-16T22:13:22+02:00 grinnodeXXXXX grin[28065]: 20200916 22:13:22.096 WARN grin_servers::common::hooks - block_accepted (REORG!): 00004bc76086 at 874794, (prev: 0001f23dc8b4 at 874793, prev_head: 0000048a71c2 at 874793, fork_point: 00009706826c at 874792, depth: 1)
280426:2020-09-17T10:44:59+02:00 grinnodeXXXX grin[22931]: 20200917 10:44:59.417 WARN grin_servers::common::hooks - block_accepted (REORG!): 00040c81e7f4 at 875503, (prev: 000572810f84 at 875502, prev_head: 00000afbaace at 875502, fork_point: 00054b6c03f1 at 875501, depth: 1)
513607:2020-09-18T04:30:53+02:00 grinnodeXXXX grin[24814]: 20200918 04:30:53.299 WARN grin_servers::common::hooks - block_accepted (REORG!): 0002076b0f32 at 876579, (prev: 0001e9e8b9d2 at 876578, prev_head: 000140786fee at 876578, fork_point: 000075561778 at 876577, depth: 1)
568386:2020-09-18T08:41:37+02:00 grinnodeXXXX grin[24814]: 20200918 08:41:37.602 WARN grin_servers::common::hooks - block_accepted (REORG!): 0002ccefa506 at 876819, (prev: 0000d7a858a1 at 876818, prev_head: 00001e3ac750 at 876818, fork_point: 0001c1238422 at 876817, depth: 5)
223300:2020-09-16T22:13:45+02:00 grinnodeXXXXX grin[32647]: 20200916 22:13:45.608 WARN grin_servers::common::hooks - block_accepted (REORG!): 00004bc76086 at 874794, (prev: 0001f23dc8b4 at 874793, prev_head: 0000048a71c2 at 874793, fork_point: 00009706826c at 874792, depth: 1)
Some minor problems we encountered during enabling debug-log.
Based on the grin-server toml you have two possibilities activating debug-log:
1. #log level for stdout: Error, Warning, Info, Debug, Trace
stdout_log_level = "Debug"
2. #log level for file: Error, Warning, Info, Debug, Trace
file_log_level = "Debug"
of couse, one / both have to be enabled “set to true”.
When logging to file on a Linux system and trying to use Syslog/rsyslog to forward it to a central syslog server it not working. Therefore we figured out, you had to enable the “stdout_log_level” which is talking to the syslog/rsylog in order to forward them to the special central-logging server. (whink)