Hi all,
I'm trying to troubleshoot some speed issue which led me to this weird performance differences between IPv4 and IPv6 - hoping I can get some help narrow down the bottleneck.
I understand throughout for routing traffic going through CCR2004 is likely to be CPU bound, but I wasn't expecting IPv4 to be slower than that of IPv6, thus this post.
Setup
![Image]()
All connections are with SFP+ auto negotiated at 10Gbps - I understand LAGG will sacrifice some performance, I'm only looking to understand the discrepancy of performance between IPv4 and IPv6 which I believe LAGG is unlikely to contribute to this issue.
All hosts get both IPv4 and ULA IPv6 addresses.
All interfaces on RouterOS has Tx and Rx flow control set to auto, though this did not seem to have affected the test result.
Benchmark
Sending from Host 2 to Host 1
IPv6 - 7.73 Gbits/sec
IPv4 - 2.57 Gbits/sec
Reverse mode
IPv6 - 7.33 Gbits/sec
IPv4 - 2.87 Gbits/sec
The initial test result seem to suggest IPv4 routing is prone to a larger amount of retransmissions, in the region of 10~40x more with IPv4.
The throughput on IPv4 is very much maxed out at around just under 3 Gbps, there bandwidth on IPv6 goes up to just under 8Gbps, a difference of around 2.5x.
I have slightly more Firewall rules configured on IPv6 (25 rules) than IPv4 (21 rules) side.
NAT rules I have tried disabling most rules so both sides have 3 rules and that made no significant difference.
What could I be missing / worth looking into?
Thanks in advance!
I'm trying to troubleshoot some speed issue which led me to this weird performance differences between IPv4 and IPv6 - hoping I can get some help narrow down the bottleneck.
I understand throughout for routing traffic going through CCR2004 is likely to be CPU bound, but I wasn't expecting IPv4 to be slower than that of IPv6, thus this post.
Setup
All connections are with SFP+ auto negotiated at 10Gbps - I understand LAGG will sacrifice some performance, I'm only looking to understand the discrepancy of performance between IPv4 and IPv6 which I believe LAGG is unlikely to contribute to this issue.
All hosts get both IPv4 and ULA IPv6 addresses.
All interfaces on RouterOS has Tx and Rx flow control set to auto, though this did not seem to have affected the test result.
Benchmark
Sending from Host 2 to Host 1
IPv6 - 7.73 Gbits/sec
Code:
$ iperf3 -c host1.mydomain.example -t30 -i30 -P5 -6Connecting to host host1.mydomain.example, port 5201[ 5] local fdfd::a00:a00:27ff:fe60:e32e port 46804 connected to fdfd:0:0:a00::abcd port 5201[ 7] local fdfd::a00:a00:27ff:fe60:e32e port 46820 connected to fdfd:0:0:a00::abcd port 5201[ 9] local fdfd::a00:a00:27ff:fe60:e32e port 46830 connected to fdfd:0:0:a00::abcd port 5201[ 11] local fdfd::a00:a00:27ff:fe60:e32e port 46838 connected to fdfd:0:0:a00::abcd port 5201[ 13] local fdfd::a00:a00:27ff:fe60:e32e port 46846 connected to fdfd:0:0:a00::abcd port 5201[ ID] Interval Transfer Bitrate Retr Cwnd[ 5] 0.00-30.00 sec 4.98 GBytes 1.43 Gbits/sec 1417 653 KBytes [ 7] 0.00-30.00 sec 5.82 GBytes 1.67 Gbits/sec 1923 700 KBytes [ 9] 0.00-30.00 sec 5.37 GBytes 1.54 Gbits/sec 1624 544 KBytes [ 11] 0.00-30.00 sec 4.99 GBytes 1.43 Gbits/sec 2301 718 KBytes [ 13] 0.00-30.00 sec 5.90 GBytes 1.69 Gbits/sec 2073 502 KBytes [SUM] 0.00-30.00 sec 27.1 GBytes 7.75 Gbits/sec 9338 - - - - - - - - - - - - - - - - - - - - - - - - -[ ID] Interval Transfer Bitrate Retr[ 5] 0.00-30.00 sec 4.98 GBytes 1.43 Gbits/sec 1417 sender[ 5] 0.00-30.03 sec 4.98 GBytes 1.42 Gbits/sec receiver[ 7] 0.00-30.00 sec 5.82 GBytes 1.67 Gbits/sec 1923 sender[ 7] 0.00-30.03 sec 5.81 GBytes 1.66 Gbits/sec receiver[ 9] 0.00-30.00 sec 5.37 GBytes 1.54 Gbits/sec 1624 sender[ 9] 0.00-30.03 sec 5.37 GBytes 1.54 Gbits/sec receiver[ 11] 0.00-30.00 sec 4.99 GBytes 1.43 Gbits/sec 2301 sender[ 11] 0.00-30.03 sec 4.98 GBytes 1.43 Gbits/sec receiver[ 13] 0.00-30.00 sec 5.90 GBytes 1.69 Gbits/sec 2073 sender[ 13] 0.00-30.03 sec 5.90 GBytes 1.69 Gbits/sec receiver[SUM] 0.00-30.00 sec 27.1 GBytes 7.75 Gbits/sec 9338 sender[SUM] 0.00-30.03 sec 27.0 GBytes 7.73 Gbits/sec receiveriperf Done.Code:
$ iperf3 -c host1.mydomain.example -t30 -i30 -P5 -4Connecting to host host1.mydomain.example, port 5201[ 5] local 10.10.11.143 port 33874 connected to 10.10.10.213 port 5201[ 7] local 10.10.11.143 port 33888 connected to 10.10.10.213 port 5201[ 9] local 10.10.11.143 port 33890 connected to 10.10.10.213 port 5201[ 11] local 10.10.11.143 port 33892 connected to 10.10.10.213 port 5201[ 13] local 10.10.11.143 port 33896 connected to 10.10.10.213 port 5201[ ID] Interval Transfer Bitrate Retr Cwnd[ 5] 0.00-30.00 sec 2.24 GBytes 641 Mbits/sec 18934 49.5 KBytes [ 7] 0.00-30.00 sec 1.61 GBytes 462 Mbits/sec 16267 73.5 KBytes [ 9] 0.00-30.00 sec 1.87 GBytes 537 Mbits/sec 16611 24.0 KBytes [ 11] 0.00-30.00 sec 1.67 GBytes 477 Mbits/sec 16438 58.0 KBytes [ 13] 0.00-30.00 sec 1.59 GBytes 457 Mbits/sec 16502 38.2 KBytes [SUM] 0.00-30.00 sec 8.99 GBytes 2.57 Gbits/sec 84752 - - - - - - - - - - - - - - - - - - - - - - - - -[ ID] Interval Transfer Bitrate Retr[ 5] 0.00-30.00 sec 2.24 GBytes 641 Mbits/sec 18934 sender[ 5] 0.00-30.03 sec 2.24 GBytes 640 Mbits/sec receiver[ 7] 0.00-30.00 sec 1.61 GBytes 462 Mbits/sec 16267 sender[ 7] 0.00-30.03 sec 1.61 GBytes 461 Mbits/sec receiver[ 9] 0.00-30.00 sec 1.87 GBytes 537 Mbits/sec 16611 sender[ 9] 0.00-30.03 sec 1.87 GBytes 536 Mbits/sec receiver[ 11] 0.00-30.00 sec 1.67 GBytes 477 Mbits/sec 16438 sender[ 11] 0.00-30.03 sec 1.66 GBytes 476 Mbits/sec receiver[ 13] 0.00-30.00 sec 1.59 GBytes 457 Mbits/sec 16502 sender[ 13] 0.00-30.03 sec 1.59 GBytes 456 Mbits/sec receiver[SUM] 0.00-30.00 sec 8.99 GBytes 2.57 Gbits/sec 84752 sender[SUM] 0.00-30.03 sec 8.98 GBytes 2.57 Gbits/sec receiveriperf Done.Reverse mode
IPv6 - 7.33 Gbits/sec
Code:
$ iperf3 -c host1.mydomain.example -t30 -i30 -P5 -6 -RConnecting to host host1.mydomain.example, port 5201Reverse mode, remote host host1.mydomain.example is sending[ 5] local fdfd::a00:a00:27ff:fe60:e32e port 39440 connected to fdfd:0:0:a00::abcd port 5201[ 7] local fdfd::a00:a00:27ff:fe60:e32e port 39456 connected to fdfd:0:0:a00::abcd port 5201[ 9] local fdfd::a00:a00:27ff:fe60:e32e port 39458 connected to fdfd:0:0:a00::abcd port 5201[ 11] local fdfd::a00:a00:27ff:fe60:e32e port 39474 connected to fdfd:0:0:a00::abcd port 5201[ 13] local fdfd::a00:a00:27ff:fe60:e32e port 39488 connected to fdfd:0:0:a00::abcd port 5201[ ID] Interval Transfer Bitrate Retr Cwnd[ 5] 0.00-30.00 sec 5.19 GBytes 1.48 Gbits/sec [ 7] 0.00-30.00 sec 5.09 GBytes 1.46 Gbits/sec [ 9] 0.00-30.00 sec 5.08 GBytes 1.46 Gbits/sec [ 11] 0.00-30.00 sec 5.16 GBytes 1.48 Gbits/sec [ 13] 0.00-30.00 sec 5.09 GBytes 1.46 Gbits/sec [SUM] 0.00-30.00 sec 25.6 GBytes 7.33 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - -[ ID] Interval Transfer Bitrate Retr[ 5] 0.00-30.03 sec 5.20 GBytes 1.49 Gbits/sec 107 sender[ 5] 0.00-30.00 sec 5.19 GBytes 1.48 Gbits/sec receiver[ 7] 0.00-30.03 sec 5.10 GBytes 1.46 Gbits/sec 626 sender[ 7] 0.00-30.00 sec 5.09 GBytes 1.46 Gbits/sec receiver[ 9] 0.00-30.03 sec 5.10 GBytes 1.46 Gbits/sec 142 sender[ 9] 0.00-30.00 sec 5.08 GBytes 1.46 Gbits/sec receiver[ 11] 0.00-30.03 sec 5.10 GBytes 1.46 Gbits/sec 314 sender[ 11] 0.00-30.00 sec 5.16 GBytes 1.48 Gbits/sec receiver[ 13] 0.00-30.03 sec 5.17 GBytes 1.48 Gbits/sec 7 sender[ 13] 0.00-30.00 sec 5.09 GBytes 1.46 Gbits/sec receiver[SUM] 0.00-30.03 sec 25.7 GBytes 7.34 Gbits/sec 1196 sender[SUM] 0.00-30.00 sec 25.6 GBytes 7.33 Gbits/sec receiveriperf Done.Code:
$ iperf3 -c host1.mydomain.example -t30 -i30 -P5 -4 -RConnecting to host host1.mydomain.example, port 5201Reverse mode, remote host host1.mydomain.example is sending[ 5] local 10.10.11.143 port 59334 connected to 10.10.10.213 port 5201[ 7] local 10.10.11.143 port 59340 connected to 10.10.10.213 port 5201[ 9] local 10.10.11.143 port 59342 connected to 10.10.10.213 port 5201[ 11] local 10.10.11.143 port 59350 connected to 10.10.10.213 port 5201[ 13] local 10.10.11.143 port 59356 connected to 10.10.10.213 port 5201[ ID] Interval Transfer Bitrate Retr Cwnd[ 5] 0.00-30.00 sec 1.99 GBytes 570 Mbits/sec [ 7] 0.00-30.00 sec 2.02 GBytes 578 Mbits/sec [ 9] 0.00-30.00 sec 1.99 GBytes 570 Mbits/sec [ 11] 0.00-30.00 sec 2.01 GBytes 576 Mbits/sec [ 13] 0.00-30.00 sec 2.01 GBytes 576 Mbits/sec [SUM] 0.00-30.00 sec 10.0 GBytes 2.87 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - -[ ID] Interval Transfer Bitrate Retr[ 5] 0.00-30.03 sec 1.99 GBytes 570 Mbits/sec 9278 sender[ 5] 0.00-30.00 sec 1.99 GBytes 570 Mbits/sec receiver[ 7] 0.00-30.03 sec 2.02 GBytes 579 Mbits/sec 9186 sender[ 7] 0.00-30.00 sec 2.02 GBytes 578 Mbits/sec receiver[ 9] 0.00-30.03 sec 2.00 GBytes 571 Mbits/sec 9067 sender[ 9] 0.00-30.00 sec 1.99 GBytes 570 Mbits/sec receiver[ 11] 0.00-30.03 sec 2.02 GBytes 577 Mbits/sec 9239 sender[ 11] 0.00-30.00 sec 2.01 GBytes 576 Mbits/sec receiver[ 13] 0.00-30.03 sec 2.01 GBytes 576 Mbits/sec 10164 sender[ 13] 0.00-30.00 sec 2.01 GBytes 576 Mbits/sec receiver[SUM] 0.00-30.03 sec 10.0 GBytes 2.87 Gbits/sec 46934 sender[SUM] 0.00-30.00 sec 10.0 GBytes 2.87 Gbits/sec receiveriperf Done.The throughput on IPv4 is very much maxed out at around just under 3 Gbps, there bandwidth on IPv6 goes up to just under 8Gbps, a difference of around 2.5x.
I have slightly more Firewall rules configured on IPv6 (25 rules) than IPv4 (21 rules) side.
NAT rules I have tried disabling most rules so both sides have 3 rules and that made no significant difference.
What could I be missing / worth looking into?
Thanks in advance!
Statistics: Posted by burnduck — Tue Apr 02, 2024 3:45 pm