MikroTik CCR2004 PCIe NIC in Proxmox
Vložit
- čas přidán 21. 03. 2023
- A smart PCIe network interface card that adds full-fledged router capabilities to your servers.
Proxmox.
Druvis.
Everything you need for unlimited knowledge in another episode of #MikroTips! - Věda a technologie
I think we need more videos about this card. Id like to better understand usecases and how it works.
Mikrotik, yes please..
This card very useful in data center or network exchange environment where you pay per U. Rather than installing separate router and server that needed 2U spaces, with this card you only need 1U space.
@@hexatested Brilliant! You know of any server models that can accommodate this card? And do you have any idea what that could mean in terms of savings? Like, if all you needed was 1 server and 1 router, then you would cut costs in two? I imagine flexible rates - paying for bandwidth and electricity are common practice.
Very-very good, troubleshooting-style video! I’m not familiar with Proxmox, but it was interesting for me. Thanks.
Another good thing to try if you want to maximize throughput to a single VM is to directly assign either individual interfaces or the whole PCIe card to a VM. This lets you skip the linux kernel bridge as a possible bottleneck.
and QEMU/KVM. There's at least 3% perf loss on CPU/RAM from those but potentially more on i/o, maybe also IOMMU groups related stuff. Likely the issue here is the load on the CPU i/o controller handling the NVMe disk on the same group. Another theory is PCIe bottleneck/overhead: this card appears to be x8 electrically, likely PCIe 3.0. That's almost exactly 8GB/s _bidirectional_ which is what we seem to be getting. Even though traffic generated shows beyond 8GB/s on the external router, only really
@@cldptpcie 3.0 8x is yes 8 Gigabytes/s but networking is Gigabits/s, 8GBps translate to 40Gbps. The pcie 3.0 8x is able to transmit both sfp28 to the host no problem, as it would be illogical to have chosen this interface other wise.
@@majordragon995 omg I could swear I saw the traffic generator window show GB/s but it's really kbps and Gbps. My mistake
Thank you for this video!
MT should fulfill it´s promises and support BSD for the CCR2004-PCIe. It would go great with a PFSense or Opnsense Firewall!
Mikrotik router inside BSD router.., interesting
Did this happen?
Ideally you have SR-IOV support and poke that into the VM directly rather than use a virtual Ethernet card in KVM. I think otherwise you won't get the full capability of PCIe because the KVM guest has to jump through the kernel in both directions.
One thing I have been curious with these cards - is it possible to make the card work basically separately, and then communicate back to the host system via one of the PCIe interfaces? Think like a security appliance where normal packets just come in one interface and go out the other and don't touch the host system's CPU, but packets you want to inspect make a trip through the host system.
using SR-IOV you run into IRQ issues due multiqueue - at least with Intel X520 series
I don't know, if this card event supports SR-IOV
another great monitoring app for CLI is glances, also give you good overview
Give us EVPN in these cards and you'll see stock go out next day. What an easy enabler of full L3 underlay, especially considering the price.
I would love to try this is in a one of my Lenovo / IBM servers
Through many different evolutions of traffic generators we finally found that TREX was the most cost effective way to test devices at our ISP. A Dell R610 can generate about 10Gbps in ASTM mode using Intel Optical cards. TREX has been tested up into multiple tens of gigs and there are even anecdotes of it being used at 100Gbps but I cannot verify this.
The issue is none of these cheap cards have any offloading so you can never hit anywhere near wire speed anyway because your bottleneck will always end up being in the CPU shuffling bits around for no reason.
WONDERFUL!!!
An update video would be greatly appreciated, this is a good card for mikrotik as well as for other open source router OSs.
Example, would something like this work with my lenovo tiny p330, could i use this and a switch to have the ultimate router + proxmox + whatever else ?
About Performance: Try to use openv-switch and set the cpu type of the vm to host.
Would be cool to hear the outcome of the suggestion
I ordered 3 of this 7 month ago to do exactly that. Haven't received a single card 😢
do we need license for this?
It would be great to have similar video with the VMWare ESX and Hyper-V :)
Can the switching acceleration hardware on the board used to make a high-performance firewall? Can multiple boards be used on a signle system with the acceleration hardware on the boards to make a larger fabric across the boards?
Thanks for the video. Since I saw this NIC announcement, I thought the idea was to run CHR directly on the nic, and not so much use it as a passthrough to other VMs. Is that possible?
This is in the style of other “SmartNIC” or “DPU” cards - having your network card do some amount of helper offload for you, although in Mikrotik’s case its just RouterOS and not a system designed to do trivial data manipulation on the fly or similar.
I wish you could run containers on the card (for use in other systems, not with a hypervisor) but it only has 128MB internal storage and no USB.
The limit could also be the number of pcie lanes available.
Product definitely looked interesting. However, the fact it simply stops working whenever it's rebooted kinda kills all use cases. Also, I experienced some kernel panics while running it. I suppose if they can fix the PCI-E initialisation issues (e.g. allow it to re-initialise after the host system has booted), it becomes a much more interesting product. Currently having two of these cards but not deploying them as it simply wasn't stable.
Try upgrading to the latest RouterOS version. I rebooted my 2004 a couple of times while leaving the host running and it seemed to work fine.
It might depend on the motherboard though.
@@RB01-lite Thanks for your reply. So there's still hope. I'll give it a go again soon. 🙂
Can this card work with ESXi 8?
I just need to be able to get hold of the damn thing... been on pre-order for nearly a year :(
Would be amazing as an integrated SAN controller of sort. Well, Network controlled SAN.
So I was just wondering if you have tried tweaking the MTU size to fit 25Gb speed ???
I know for 10Gb the MTU can be shaped to 9000 but in my experience leaving it default in production environment is easy troubleshooting.
When it comes to the PCIe card itself, to attain the maximum possible throughput jumbo frames are required, but it should be possible to improve the throughput without resulting to that.
ordered it 06/2022 - still waiting. not available like many other products. i am certified for your stuff and need them for customerprojects, but cannot buy them anywhere. i am really pissed
Isnt virtio limited to 10gps in the driver. The only solution is to pass through the hardware using iommu and making a dedicated VM driving the NIC.
Product seems really interesting, but a bit hard to imagine solid use case for. As it lacks some features that other DPUs have. Albite this is more affordable. Make a version with more RAM and Storage, comparable to nVidia Bluefield and add NVMe-over-TCP support. Or show how it can be used to offload traffic encryption or firewalling.
Make a video of more use cases for CCR2004-PCIe.
NVMe over TCP already possible with the ROSE package ;)
@@RB01-lite That's nice, but it's only a part of it. DPUs can interpose as a regular NVMe device to Host machine. This is the missing link, or it already can do it ?
@@andiszile If I understood you correctly a DPU could just have the host load an NVME drive on bootup, that is physically elsewhere. In the current ROSE implementation you can only access the NVMe-over-TCP drive after the boot process.
@@RB01-lite Ok. Looking into it. Maybe even other DPUs can't really be used as boot devices (unless UEFI can wait for drive to bootup :D ) but they can be used as storage device.
Gain is that Host CPU doesn't need to process nvme-tcp protocol by itself.
@@RB01-lite But probably i am too focused on this one particular use case. That's why i would like to see showcase of more use cases that utilize this as more that just a NIC to broaden my view.
Where can I get it?
Any use case?
Is the problem on performance solved? Could you consider using SR-IOV please?
If @mikrotik made a follow up video in 24-25 with SR-IOV backed pass-through to the Proxmox VM. The CPU probably could not coke eggs anymore. ᕕ(⌐■_■)ᕗ ♪♬
9:18 - THX for real values.
That's the router running traffic generator and not the card.
@@RB01-lite Doesn't *cpu-used-by-cpu=100%* limit performance / transfer ?
@@Miesiu It limits how much traffic can get generated in the first place, here the issue was that the throughput was lower than what gets generated.
@@RB01-lite Now understand. THX!
Am I right in thinking that this thing does not support sr-iov?
And more importantly - since main selling point is that it's a router - what kind of speed one can expect when this thing is being used as a router?
Don't know much about sr-iov, but routing performance depends largely on setup. However it is safe to say that routing with the 25G interfaces will not deliver anything near the wire-speed that is possible in pass-through mode.
@@RB01-lite 1) the card is a router itself, it runs ROS as on it's own.
2) seen in the specs, max of 12G routing in best scenario
Synthetic load is a not VM friendly by any means, maybe try passthrough the whole pci-e slot to the VM, or at least with IOMMU try to individually pass one of the cages
when we talk about performance, some words pop in my head,
SR-IOV, multi-queue, OVS, DPDK
as proxmox is a .... I mean compare to proxmox, vmware vsphere is a (more) enterprise ready platform, should perform best result out of the box (don't know if DirectPath I/O nic helps. but we seen vmxnet3 in vmware gives better performance compare to x520sriov, cause x520sriov driver only support 1 queue.)
how ccr2004 pcie card running on that?
if you can emulate atl1c, how about emulate more pcie interface for sriov use, will be useful.
If I am not mistaken, Linux is the only platform this is supported on. This card requires extra drivers that are not available on VMware. If I am wrong, then please someone let me know, but I recall reading this on the Mikrotik website.
@@masterTigress96 Yes, one year later, this card is still not supported by enterprise virtualization platforms, It's a software emulated card without any hardware offload supported. If they can improve driver, it will be very promising, and we have seen the benefits brought by Bluefield and Amazon Nitro.
Card is similar to Radeon RX 6400 !
Despite my best efforts I cannot find one. I have checked several distributors and they are all telling me they have not had had one for close to a year
Does this card work with XCP-NG?
In PCIe passthrough it should, but I have yet to get my hands on one to test. Looks interesting.
What @jblow530 said, so no not for XCP-NG itself if you want to use it to e.g. migrate VM's to another host in a speedy fashion. Linux is as far as I know the only OS this is supported on. XCP-NG and VMware also run a modified version of Linux, but you need something like Proxmox (which is a more standard, full fat Linux distro) to get the drivers.
Maybe a custom kernel for XCP-NG of VMware could get it to work, but I haven't tried it.
I wish that it we can just stick to any windows pc then run winbox just like any other mikrotik router :P
Uztaisiet, lūdzu, video, kā dabūt AWS site-to-site VPN'a pieslēgumā ar Mikrotik aktīvus abus tuneļus.
I never had great luck with proxmox XCPNG I've had much better network stability
This video is confusing. No agenda and clear scenarios....waste of time of the creator...