Feature requests - CHR on Bare Metal for faster Network throughput
Note: This topic is about achieving very fast network/Ethernet throughput ( as in 10-Gig and faster network routing throughput)
I would like to see a CHR that can be installed on a Bare Metal server.
Here is my reasoning and justification why I would like to see a CHR that can be installed on a Bare Metal server.
Mikrotik's CHR ROS runs on a Hypervisor ( such as VmWare or ... ). This is a 64-Bit operating system and also does not have the ability to be directly installed on a Bare Metal computer/server
thus:
- CHR , does not run on Bare Metal
- CHR , no install ISO ( or any procedure ) to install CHR on a Bare Metal computer
- CHR , can be installed as a 64-bit operating system in many Hypervisor environments
Mikrotik's x86 ROS can be installed on most Bare Metal computers and under most Hypervisor environments. However, x86 ROS is a 32-Bit operating system.
-
When a virtual CHR sends an Ethernet packet out through the virtual Ethernet interface, the packet is first delivered to the Hypervisor virtual vSwitch , which will always require at least one ( 1 ) physical CPU clock-cycle - thus so far , the packet has not actually been sent to anything yet.
Now that the hypervisor's vSwitch has the packet , the hypervisor will always require at least one ( 1 ) physical CPU clock cycle to send the packet out through the physical network interface card.
* So far , we have consumed at least two ( 2 ) CPU clock cycles or more to get the packet from the CHR out on the physical network switches.
if the packet is destined for another CHR on another different hypervisor , then there is at least one ( 1 ) CPU clock cycle to get the paket from the physical Ethernet interface and place the inbound packet on the vSwitch , and one ( 1 ) more CPU clock cycle to move the packet from the vSwitch to the CHR.
The effective throughput result is that when a CHR is sending or receiving packets out or in through physical network interfaces, we are running at half CPU speed. Thus , it is almost impossible for a CHR to achieve sustained network speeds of 6-Gig or faster ... because of the hypervisor CPU clock cycle overhead to transfer packets.
Depending on the physical CPU processor GHz clock speeds that the hypervisor is running on , it is easily possible to have a wall of 6-Gig or 8-Gig that the CHR is limited to in throughput. This wall is there because of hypervisor CPU clock cycle overhead when moving packets. Even if you have 100-Gig physical network cards on the hypervisor , the packet transfer throughput wall is still there. ((( And - there is also the CPU consuming clock cycle resources of the hypervisor which may also be slowing down the CHR throughput - because the hypervisor is also doing other things at the same time)))
ROS x86 32-Bit does not have this hypervisor CPU overhead when installed on a bare metal box. However, this is a 32-Bit ROS router and may not have enough RAM memory for large RAM needs such as multiple BGP tables and multiple OSPF tables and/or other RAM consuming things such as firewall rules and large address-list tables.
I think CHR might support SR-IOV interfaces ( which should eliminate the hypervisor CPU clock cycle overhead) , but I do not know if a SR-IOV driver is in the CHR ROS software. Also , not all hypervisors and network cards and BIOS configurations have SR-IOV support.
If Mikrotik were to support a CHR install on a bare metal install, the packet throughput should be possibly double the speed or more.
To backup my claim of packet speeds and hypervisor CPU clock speed overhead, there is a VyOS open-source router that can be installed on bare metal and also has SR-IOv support. There are many reports of VyOS router installs sustaining 40-Gig and much faster (nearly up to 100-Gig) network transfer speeds.
My ISP network CHRs are hitting that wall. I am currently upgrading my physical servers and switches to 100-Gig. I am also upgrading my four BGP Internet feeds to 100-Gig. My current CHRs can sustain 6-Gig but it's impossible to layer-3 route faster than 10-Gig.
So , I ask ... Would Mikrotik please consider creating a CHR platform that can be installed on bare metal - and have drivers for 40-Gig, 100-Gig, 200-Gig and 400-Gig network cards -and- also include support for SR-IOV network interfaces?
If the future of CHR will not include bare metal install features, then I will be forced to migrate my layer-3 CHR routers to something else.
North Idaho Tom Jones
Note: This topic is about achieving very fast network/Ethernet throughput ( as in 10-Gig and faster network routing throughput)
I would like to see a CHR that can be installed on a Bare Metal server.
Here is my reasoning and justification why I would like to see a CHR that can be installed on a Bare Metal server.
Mikrotik's CHR ROS runs on a Hypervisor ( such as VmWare or ... ). This is a 64-Bit operating system and also does not have the ability to be directly installed on a Bare Metal computer/server
thus:
- CHR , does not run on Bare Metal
- CHR , no install ISO ( or any procedure ) to install CHR on a Bare Metal computer
- CHR , can be installed as a 64-bit operating system in many Hypervisor environments
Mikrotik's x86 ROS can be installed on most Bare Metal computers and under most Hypervisor environments. However, x86 ROS is a 32-Bit operating system.
-
When a virtual CHR sends an Ethernet packet out through the virtual Ethernet interface, the packet is first delivered to the Hypervisor virtual vSwitch , which will always require at least one ( 1 ) physical CPU clock-cycle - thus so far , the packet has not actually been sent to anything yet.
Now that the hypervisor's vSwitch has the packet , the hypervisor will always require at least one ( 1 ) physical CPU clock cycle to send the packet out through the physical network interface card.
* So far , we have consumed at least two ( 2 ) CPU clock cycles or more to get the packet from the CHR out on the physical network switches.
if the packet is destined for another CHR on another different hypervisor , then there is at least one ( 1 ) CPU clock cycle to get the paket from the physical Ethernet interface and place the inbound packet on the vSwitch , and one ( 1 ) more CPU clock cycle to move the packet from the vSwitch to the CHR.
The effective throughput result is that when a CHR is sending or receiving packets out or in through physical network interfaces, we are running at half CPU speed. Thus , it is almost impossible for a CHR to achieve sustained network speeds of 6-Gig or faster ... because of the hypervisor CPU clock cycle overhead to transfer packets.
Depending on the physical CPU processor GHz clock speeds that the hypervisor is running on , it is easily possible to have a wall of 6-Gig or 8-Gig that the CHR is limited to in throughput. This wall is there because of hypervisor CPU clock cycle overhead when moving packets. Even if you have 100-Gig physical network cards on the hypervisor , the packet transfer throughput wall is still there. ((( And - there is also the CPU consuming clock cycle resources of the hypervisor which may also be slowing down the CHR throughput - because the hypervisor is also doing other things at the same time)))
ROS x86 32-Bit does not have this hypervisor CPU overhead when installed on a bare metal box. However, this is a 32-Bit ROS router and may not have enough RAM memory for large RAM needs such as multiple BGP tables and multiple OSPF tables and/or other RAM consuming things such as firewall rules and large address-list tables.
I think CHR might support SR-IOV interfaces ( which should eliminate the hypervisor CPU clock cycle overhead) , but I do not know if a SR-IOV driver is in the CHR ROS software. Also , not all hypervisors and network cards and BIOS configurations have SR-IOV support.
If Mikrotik were to support a CHR install on a bare metal install, the packet throughput should be possibly double the speed or more.
To backup my claim of packet speeds and hypervisor CPU clock speed overhead, there is a VyOS open-source router that can be installed on bare metal and also has SR-IOv support. There are many reports of VyOS router installs sustaining 40-Gig and much faster (nearly up to 100-Gig) network transfer speeds.
My ISP network CHRs are hitting that wall. I am currently upgrading my physical servers and switches to 100-Gig. I am also upgrading my four BGP Internet feeds to 100-Gig. My current CHRs can sustain 6-Gig but it's impossible to layer-3 route faster than 10-Gig.
So , I ask ... Would Mikrotik please consider creating a CHR platform that can be installed on bare metal - and have drivers for 40-Gig, 100-Gig, 200-Gig and 400-Gig network cards -and- also include support for SR-IOV network interfaces?
If the future of CHR will not include bare metal install features, then I will be forced to migrate my layer-3 CHR routers to something else.
North Idaho Tom Jones
Statistics: Posted by TomjNorthIdaho — Fri Jan 26, 2024 10:54 pm