libp2p performance benchmarking

"Life is all about sharing. If we are good at something, pass it on." - Mary Berry

libp2p performance benchmarking

2023-10-27

I invested some time in studying QUIC and libp2p In this article, I will show you how to benchmark the network transfer using perf module for 2 scanerios:

high latency, no packet loss
low latency but high packet loss

Let’s print help first:

 1     Running `rust-libp2p/target/debug/perf -h`
 2Usage: perf [OPTIONS]
 3
 4Options:
 5      --server-address <SERVER_ADDRESS>
 6      --transport <TRANSPORT>
 7      --upload-bytes <UPLOAD_BYTES>
 8      --download-bytes <DOWNLOAD_BYTES>
 9      --run-server                       Run in server mode
10  -h, --help                             Print help

Run in server mode:

1     Running `rust-libp2p/target/debug/perf --server-address '127.0.0.1:8080' --run-server`
2[2023-10-27T07:17:09.543Z INFO  libp2p_swarm] Local peer id: 12D3KooWPxXR3Me5pS7UjRg2gPVk9A7Se4nhEVBT9GPifjAwpiWX
3[2023-10-27T07:17:09.545Z INFO  perf] Listening on /ip4/127.0.0.1/tcp/8080
4[2023-10-27T07:17:09.545Z INFO  perf] Listening on /ip4/127.0.0.1/udp/8080/quic-v1

In another terminal, try to upload/download 100MB:

 1     Running `rust-libp2p/target/debug/perf --server-address '127.0.0.1:8080' --transport tcp --upload-bytes 104857600 --download-bytes 104857600`
 2[2023-10-27T07:21:14.922Z INFO  libp2p_swarm] Local peer id: 12D3KooWCwLib8hXRpVYNZFFnjBpQeqdKfFNgQUTj4mpJ2qyGdiZ
 3[2023-10-27T07:21:14.922Z INFO  perf] start benchmark: custom
 4[2023-10-27T07:21:14.930Z INFO  perf] established connection in 0.0076 s
 5[2023-10-27T07:21:15.935Z INFO  perf] 1.005029375 s uploaded 45.86 MiB downloaded 0 B (365.02 Mbit/s)
 6{"type":"intermediate","timeSeconds":1.005029375,"uploadBytes":48083968,"downloadBytes":0}
 7[2023-10-27T07:21:16.940Z INFO  perf] 1.005100667 s uploaded 46.42 MiB downloaded 0 B (369.45 Mbit/s)
 8{"type":"intermediate","timeSeconds":1.005100667,"uploadBytes":48671744,"downloadBytes":0}
 9[2023-10-27T07:21:17.945Z INFO  perf] 1.005020208 s uploaded 7.73 MiB downloaded 38.97 MiB (371.73 Mbit/s)
10{"type":"intermediate","timeSeconds":1.005020208,"uploadBytes":8101888,"downloadBytes":40865792}
11[2023-10-27T07:21:18.950Z INFO  perf] 1.005063875 s uploaded 0 B downloaded 46.87 MiB (373.07 Mbit/s)
12{"type":"intermediate","timeSeconds":1.005063875,"uploadBytes":0,"downloadBytes":49146880}
13[2023-10-27T07:21:19.253Z INFO  perf] uploaded 100.00 MiB in 2.1779 s (367.33 Mbit/s), downloaded 100.00 MiB in 2.1454 s (372.89 Mbit/s)
14{"type":"final","timeSeconds":4.331209541,"uploadBytes":104857600,"downloadBytes":104857600}

To simulate network latency and packet loss, we can use pfctl and dnctl

First, enable the packet filter:

1sudo pfctl -E

Then create a custom anchor in pf:

1$ cat /etc/pf.conf && echo "dummynet-anchor \"libp2p\"" && echo "anchor \"libp2p\"" | sudo pfctl -f -

Pipe the traffic to dummynet:

1$ echo "dummynet in quick proto tcp from any to any port 8080 pipe 1" | sudo pfctl -a libp2p -f -

Simulate the network latency:

1$ sudo dnctl pipe 1 config delay 10ms

Benchmark again:

1Running `rust-libp2p/target/debug/perf --server-address '127.0.0.1:8080' --transport tcp --upload-bytes 104857600 --download-bytes 104857600`
2[2023-10-27T07:58:43.515Z INFO  perf] uploaded 100.00 MiB in 8.4413 s (94.77 Mbit/s), downloaded 100.00 MiB in 8.2188 s (97.34 Mbit/s)
3{"type":"final","timeSeconds":16.689302458,"uploadBytes":104857600,"downloadBytes":104857600}

Simulate the packet loss:

1$ sudo dnctl pipe 1 config plr 0.01

1$ sudo dnctl list
200001: unlimited    0 ms   50 sl.plr 0.010000 1 queues (1 buckets) droptail
3    mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
4BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
5    mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
6BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
7  0 tcp        127.0.0.1/63691       127.0.0.1/8080  1011241 490303519  0    0 9463

1Running `rust-libp2p/target/debug/perf --server-address '127.0.0.1:8080' --transport tcp --upload-bytes 104857600 --download-bytes 104857600`
2[2023-10-27T08:33:58.098Z INFO  perf] uploaded 100.00 MiB in 9.1012 s (87.90 Mbit/s), downloaded 100.00 MiB in 2.2653 s (353.16 Mbit/s)
3{"type":"final","timeSeconds":11.3747555,"uploadBytes":104857600,"downloadBytes":104857600}

Tags: libp2p rust dnctl pfctl

Edit on GitHub