ONE HUNDRED GIGABIT - MikroTik CRS504-4XQ-I9
Vložit
- čas přidán 19. 06. 2024
- To try everything Brilliant has to offer-free-for a full 30 days, visit brilliant.org/CraftComputing/ - Plus the first 200 of you will get 20% off Brilliant’s annual premium subscription!
As my storage needs grow, so too does the need for speed. And thanks to MikroTik, I'm well equipped to give my home network a substantial boost. In this video I install two 100gb MiktroTik switches, run all new fiber lines, and ensure my new network cards can handle the torrential flood of bits. My server rack has never been faster, and it's going to speed up all my services, even cloud gaming.
But first... What am I drinking?!
Well to kick things off, I got me a can of Pub Beer from 10 Barrel Brewing. But as weeks turned to months, this shoot wrapped in the midst of Mixology March. So I crack me an egg and enjoy a cocktail I've never tried: An Ace. A delightfully creamy and playful potion which will satisfy your holiday hankerings quite nicely.
Links to items below may be affiliate links for which I may be compensated
Check out today's parts:
Mikrotik CRS504-4XQ-IN 4-port QSFP28 Switch - amzn.to/3JPX4mW
Intel 100GBASE-CWDM4 QSFP28 SMF Transceiver - ebay.us/ZIFi5q
LC to LC Single Mode Fiber Patch Cable - amzn.to/3yPPVN3
100G QSFP28 to 4x 25G SFP28 Passive DAC: amzn.to/3ZZpMas
Mellanox ConnectX-4 PCIe 3.0 x16 QSFP28 NIC - ebay.us/T7XqXa
Mikrotik CRS317-1G-16S+ on Amazon - amzn.to/3p5k5Gp
Mikrotik Desktop 5-port 10Gb Switch - amzn.to/2TvVezP
Mikrotik CRS328-24P-4S+ 24-Port PoE + 10Gb Switch - amzn.to/3fHpGjf
Ubiquiti UniFi UDM-Pro Router/Firewall - amzn.to/3fA3Spm
T568B Cheat Sheet T-Shirt available here: vkc.sh/product-tag/t568b-chea...
Grab yourself a Pint Glass or Hoodie at craftcomputing.store
Follow me on Mastodon @Craftcomputing@hostux.social
Support me on Patreon and get access to my exclusive Discord server. Chat with myself and the other hosts on Talking Heads all week long.
/ craftcomputing
Music:
George Street Shuffle by Kevin MacLeod
Link: incompetech.filmmusic.io/song...
License: filmmusic.io/standard-license
0:00 - Brilliant
1:05 - Welcome Back and Unboxing
5:50 - Hardware Overview
8:10 - Installation
11:09 - Welcome Back... Again. How'd it Go?
14:43 - Network Cards
18:54 - TESTING!
22:11 - Real World Uses
25:04 - What do you think? And Sign Off
26:13 - Beer and Cocktail Review - Věda a technologie
To try everything Brilliant has to offer-free-for a full 30 days, visit brilliant.org/CraftComputing/ - Plus the first 200 of you will get 20% off Brilliant’s annual premium subscription!
Your shirt is brilliant!
You pronounced it correctly (not that anyone minds if you didn't, just nice to see)
why does your intro have the sound of water pouring into a glass??? you would think you'd have at least used a carbonated beverage pouring sound for it... right??? You have the MIC... you have the BEVERAGE.... now fix it!!! (and lemme have the beverage when you're done plz?? ... that's the whole reason I wrote this... I wuntz bierz plz :P) :)
Forgot to put the command in the video description - as you said at 15:45 ;)
those mad lads at Mikrotik just sent not just one but a couple of their flagship 100gb switches plus accessories, no questions asked. Jeff truly wields the power of the gods
Facts haha
i Still cannot justify the 2x 2.5 giabit switches i need in my basement and office, this guy is doing 100 gig! Good for you great content! Keep it comming!!!
I have a 36-port 100 Gbps Infiniband switch in my basement.
yeah, really is frustrating that consumer networking has been dragging ass for the better part of TWO decades now. I swear they've introduced 2.5Gbps and 5Gbps just to nickle and dime us instead of going to 10 where they can keep overcharging businesses
@@ewenchan1239 ah stop trolling
@@bojinglebells it hasn't, you have no idea what are you talking about. most consumers don't even use 1gbs looking at how widespread is wifi. 2,5 and 5gbs was introduced for entrerprise, to reuse cat5e cables.
anyway there are already 400gb devices available or even 800g, of course not for consumers or average enterprise...
@@wiziek they don't put 2.5G on consumer products so that enterprise can use them. But sure, keep being a know-it-all ass.
iperf3 tip: it natively supports multiple parallel streams with the -P flag, no need for multiple instances ;) great upgrade though, it gave me the itch to dust off my own 100Gbps cards I have lying around
THIS
Your shirt... That little saying is grilled deep into my head. After many years of making thousands tens of thousands hundreds of thousands of patch cables over the last 20+ years.
This is the level of absurdity that I love from the channel.
I was finally able to upgrade my NAS and main pc to 10 gig. Because I needed to upgrade my NAS and didn't want to deal with migrating 30TB the data over a 1g connection.
I’d like to see RDMA in Linux and Windows through SMB Direct, as well as iSCSI and NFS. RDMA should remove all CPU bottlenecks since the transfers will not longer use traditional file stacks. Make sure when doing any tests with iSCSI to turn off synchronous writes with TrueNAS, it will allow better performance for tests-although it shouldn’t matter once you get NVMe.
Well, not quite correct... RDMA itself removes the TCP stack and goes RAM to RAM via Infiniband, like DMA does from let's say sound card to RAM, but between machines. File-systems are still involved, especially in userland program space, where fopen, read, write and fclose operations are used.
It's super driver dependent and the drivers suck ass. Tried to implement it for a fileserver project a while back but couldn't get it to not drop packets.
NVMe ROCEv2 requires converged enhanced ethernet support to avoid flow control issues else reliability issues may result - I do not think these are CEE capable.
Great video. With 100gb you could pretty much run all the VM storage over iscsi. Would make for an interesting project.
That's what i did but with 40Gbe on my proxmox
most of the clients I work for have 10Gbit (either fiber channel or ethernet) for VM storage.
"over iscsi"
or NFS
The way you are installing that is making my DC senses tingle. MicroTic designed this to be a back of rack device. I bet the fans are blowing the wrong way for your configuration. That's why the power and QSFP are on the same side. Edge switches have power on one side and ports on the opposite side because you generally link them to patch ports for your end devices off the front.
Also MultiMode (OM4) is fine and spec'd up to 100G over 125m(400ft). The problem as you saw is the QSFP28's at that configuration want 8 or more likely 12 strand MPO and they are not cheap.
I really wish you had been clearer that the switch to SingleMode infrastructure wasn't a limitations of the OM4 itself but a budget limitation of buying the hardware to allow OM4 to operate at 100Gb speeds. OM5 is the same issue, however old Datacenter infrastructure that I supported ran either 24, 48 or 96 strands to each cabinet using 1x12 MPO so even with the cost saving of SM which is a relative recent thing, it was still cheaper to buy the MM compatible 100GBSR QSFP28 modules than to replace the cable plant would be.
It's great to see a good use of 100gig. I upgraded to 10gig with the crs326 and crs317 switches a few years back and think I am good for the foreseeable future.
I bought a 2u wall mount rack and bolted it under my desk for my PDU. Works perfectly and keeps everything out of the way. Also installed some under desk pockets to hold power bricks and the like.
On systems with only one X16 slot, I would recommend running your 100G NIC in the X16 and your GPU in the X8 or X4 slot, the GPU might only lose a few per cent of performance, but your 100G NIC will thank you.
It took a long time for me to finally try out some MikroTik hardware. It was their Wireless Wire kit, which is ironic given my long-standing distaste for infrastructure wireless, but genuinely the best choice.
Other than some configuration bobbles - they're the link in the middle of a double-NAT setup - it's been _very_ nice. Incredibly reliable and simple to install. I know double-NAT is bad, but I haven't been able to successfully argue for taking over the ISP-provided external router yet.
Awesome video, Jeff. LOVE Mikrotik!
I have been waiting for this video so much!! And it's 30 minutes! fun!
Keep creating amazing content!
Having a personal, local fiber network in you home just sounds so cool to me lol
Great Scott's. That is some seriously impressive gear, thanks for sharing.
The fact that all of this is as (relatively) inexpensive as it is is freaking crazy to me
Love to see the Rack Studs, they're so great.
It's crazy how far transfer speeds have gone. It was only like 15 years ago we were splitting DS0's off of DS1's to backhaul voice on cell sites.
I appreciate the shirt, after doing thousands of cable ends in my life thus far.
For file transfers and real applications (such us streaming) you would need to setup NFS over RDMA or Samba over RDMA. I don't think the bottleneck is in the SSD raid. Standard ethernet works fine for synthetic benchmarks using specialized applications (like iperf), but for getting similar speeds with network filesystems and non specialized apps you really need RDMA.
Remote direct memory access
My goodness. I have a hard enough time using all of my 10GBE setup, even between servers doing large VM backups! hahahaha. Good video, looking forward to more videos on it.
never clicked so fast on a video
Awesome Shirt in the 2nd half. Cerritos Strong!
Good job wearing green! This comes from a Professional Brewer in Ireland who was raised in The PACNW! Happy St.Patricks day. Let me Know if you ever want to Homebrew. Super fast switch I just ordered the 570/80 you just presented.
Looking forward to the Truenas testing with Cache and all that 🙂
That is fast. I know less than Linus about the linux universe but it is always fun to see the hardware. Also glad you mentioned the tree grinder across the street. For most of the video i was sure that you had a noisy fan near a microphone. Cheers
Awesome video, I have a question regarding safety though:
Could you please be more clear on the dangers of leaving SFP/QSFP lasers open? It can permanently blind someone, and proper precaution should be taken when handling lasers.
I grew up having a BBS on 9600 baud (9.6 kbps) and the move to 19,200 baud was a giant leap. That was still all over a circuit switched telephone network, i.e. not IP networks yet.
The speeds you reach in this video are like 10 million times higher... Talk about progress...
Nice network, good job!
vkc.sh/product-tag/t568b-cheat-sheet/
Looks to be from the CZcams channel "Veronica Explains" 🙂
Damn --- I have not been this jealous in a while lol.
That's sick tho
Youre setup gives me something to aim towards... maybe one day
Wow. Overkill - Love it!
One thing I'd really like to see is inter-vlan routing speeds. Hardware offloading etc.
There seems to be very little information around correctly setting this up without just killing your switches/routers cpu.
Usually if you're geeky enough to have 10 or 100gb networking then you're going to have vlans. :-)
It would help me get on the 10/100gb train!
I was going to ask how he setup inter-vlan routing with that Ubiquiti UDM-Pro. I went to 25 Gbps and I had to collapse my VLANs to put all of the 25 Gbps clients in a single VLAN if I still wanted to use the UDM-Pro. An alternative might be to use the Mirotik as the inter-vlan router but I don't know what the speeds are. The UDM-Pro just couldn't cut it for me.
Was waiting for someone to review this switch in the wild. Interesting regarding the cpu bottleneck. Assuming there’s a hardware DMA workaround that doesn’t involve CPU. We’re getting close to RAM speeds let alone storage.
I do not have the kind of speed requirement, files, workloads, etc., that said it is definitely drool worthy. I agree on the ' typical ' workload usage and could do with that kind of speed and storage for my Steam Library and wonder what kind of load times you could get using a net-box on that network.
I also had the same issue with Mellanox ConnectX2 and 4 Cards on some HPE Workstations.
I've very happy with my MikroTik gear.
Jeff,
In this video, you were just like a kid at Christmas, so Merry Christmas! enjoy your massive bandwidth.
Love the shirt!!
If you want to push the network, storage, and CPU a bit, database ETL (extract, transform and load) will be a good general stress test.
Not only are you capable of saturating the network link, but for how long, and for how much data that has to be process, structured, and stored in a database that can then be queried quickly. I'm not sure what's out there for "canned" large footprint ETL benchmarks though.
I got a Celestica dx010 and can't even get to 25. Still love the overkill. I went with bcachefs for testing my storage in a tiered configuration, I don't have all that many nvme drives to put my data just there.
To avoid the scheduler on FreeBSD bouncing around on diff cores, use the command cpuset with iperf to lock down the process to a specific core. I was able to achieve much better rates when benchmarking like this.
is really amazing... readly proud for latvian guys... )) mikrotik well done )))
I would love to see a iSCSI netwoork boot comparisson between a sata ssd and nvme. In addition to that mount a GameDrive for steam and see what kind of cpu usage is generated through the streamed block storage with that much bandwidth
Great video! At this point, do you really even need to copy back the footage to edit it, or can you do that directly on the remote drives (using copy-on-write so you don't overwrite the originals or have to create a copy. Deduplication might net you some interesting savings there, too). I have a funny feeling you might run into issues and bottlenecks with SMB, the File Explorer equivalent or similar. But still, that would be a use case I'd like to see. Maybe you could also try playing some disk-intensive games directly from remote storage and seeing if they remain playable? Just a "totally overkill hardware for gaming" kind of idea. And thanks for the idea of running single-mode over multi-mode if you want "cheap" future upgradability. I'd like to run some fiber lines around the house for the connection between the desktop and the NAS. I'm looking at 10Gbps right now, but it would suck to be stuck there in the future.
When you said 100 gigabit full duplex it gave me a full duplex. I've been considering a network upgrade for a bit and I'd very much like to get 10g links in place, but 100g is almost an incomprehensible number.
If your full duplex lasts more than 4 hours, make sure to call your doctor.
Beautiful, absolutely beautiful 😘
When those lights first came on, Jeff was grinning like a kid who just had the best Xmas of his life! 😆
It was like getting my Super Nintendo all over again 😁
@@CraftComputing - There is an unmistakable look of pure joy, that can’t be faked. You had that look just then. 😆
Curious what distance those single mode optics are rated for? Did you put attenuators on the (preferably) receive ends of the links? You are going to shorten the life on your optics if you are blasting 10km optic power levels over 3meter cables.
Nice!
I'm a sucker for fast networking ^^ Awesome video, Jeff.
Why would RAIDZ-1 be preferrable to a striped mirror (RAID10) configuration in your use case? Other than the the fact that you have more storage space.
Wouldn't RAID10 be better for write speeds and potential rebuilding of the pool in a worst case scenario?
Took me a minute to realize what that shirt means. I'm drinking and eating as I'm watching this and thinking about that shirt. I like it.
Oh my gosh, love the t-shirt!!! When I first noticed it I just laughed and laughed. Mostly only network geeks will get it. Well done!
Quite outdated for *this* video though 😂
This comes up on my CZcams feed the day two new 32x200Gbps switches where delivered at work 😂
I really want to see how you maximize the network.
I've been meaning to pick up the same router, but I'm not sure how I should be setting up NVME drives to maximize.
Should I be using a 16x pcie 4.0 slot with a 4x4x4x4x PCIE bifurcation card? Striped + Mirrored?
What about sata ssds, how fast can those be made in ZFS.
What about tiered storage with an HDD, SSD, and NVME pool. With the stuff you're actively working on being moved to the NVME pool?
(
So envious :)
I love the t568b shirt and the Boimler shirt and I'm totally jealous of your new network setup. I can barely saturate the 10g on my home network, but my homelab storage server is a whole lot smaller than your monster.
But don't most people say "orange-white" rather than "white-orange", etc???? Because it's orange insulation with a white stripe on it?
@@Hitek146 I learned it as "white-orange" because verbally it's easier for me to differentiate. Saying "orange" twice in a row makes me lose track. Plus, I don't think the design would have worked as well with the colors all on the left-hand side of the shirt.
@@VeronicaExplains I agree that the colors all being on the left would be less visually attractive, but I think you have it backwards about the verbal repetition. Putting "white" first means that you are saying "orange" twice, which is what you say is what makes you lose track. Saying "white-orange, orange" puts the two oranges together, while saying "orange-white, orange" separates the two orange words. I always say "orange-white, orange, green-white, blue, blue-white, green, brown-white, brown", only putting the two blues together, rather than putting the two oranges and browns together. Plus, in my experience terminating old-school telephony cabling, where there can be hundreds of pairs in one bundle of many various colors, including purple, the stripe is always said last...
@Hitek146 I don't think I have it backwards from my angle, since I know what I remember in the server room (you do you though). Besides, it's a t-shirt, the design was the most important part for me.
You made me an alcoholic. Always drinking your episodes with beer :)!
Oh, that t-shirt. I want one!
I was thinking the same!
vkc.sh/product-tag/t568b-cheat-sheet/
The one thing I like about Mikrotik is I don't have to fight with GBIC and SFP compatibility! Intel, 10Gtek, Nokia (FTTH SFP). It doesn't care and it just works.
With 100G and DPU you could boot off network as it was local nVME drive. That would be interesting project, albeit expensive one.
Our mikrotic gear is cheap but unfortunately buggy. Their OSPF implementation has a memory leak. CCR2004 has a few problems that severely limit bandwidth. Also ROS7 still has some blocker bugs for us.
FINALLY!!!!!!
Looks like it is Christmas at Crafting Computing!
Well I guess I'm done waiting for the 40gig Brocade icx-6610 part deux!
cheers!!! thanks!
MikroTik hasn't touched SwitchOS for two years, so no surprise they haven't ported it over to the faster-than-10Gb switches. The latest version is 2.13. And their site is saying the CRS504 is RouterOS v7 only, so they've probably abandoned SwitchOS.
Given my experience trying to use RouterOS as well, I can say that performance will fall through the floor if you try to actually use that as a router. I had that problem with the aforementioned CRS317, which is why I have an OPNsense router instead.
That is one snazzy shirt you got, Jeff!
I'm surprised how long it took me to understand that shirt... I always think OrangeWhite-Orange-GreenWhite-Blue.... in my head
100Gb very cool for most businesses let alone home, but that mains lead about to pull out of the wall, can you get 90° NEMA plugs so the strain is downwards?
Damn 100gig... I would only need that between my two servers... Despite I was only cnsidering a 25g upgrade...
Thanks alot Jeff... My bank account is getting drained
I get about the same, ~ 35 Gbps per stream with iperf3, and about 16 Gbps with plain SMB. Using RDMA (SMB-Direct) I can achieve 40 Gbps file copies, but only in one direction. (Uploading from W11Pro4WS to Svr2022) Tried enabling PFC+DCB on my equipment, speed went down. More troubleshooting needed.
I would like to see the peak of cloud gaming
Nice Video! Would it be possiable to see how long it takes to do a Snapshot of a VM?
So it's 5 months later, and I have only now picked my jaw back up off the floor after hearing you say you got those 16 Intel 100gig transceivers for 5 bucks each.
If it makes you feel any better, I still can't believe it myself.
Nice network t-shirt... It's a sign of the times now that when I see an awesome rack setup I think God that'd cost £££ in electricity bills to run each month. 😞
I NEED to know where you got that shirt, it would be an absolute hit at work.
What MTU do you run? Jumbo frames can help with reducing the packet per second rate which can affect everything from number of interrupts the NICs have to handle which can have beneficial side effects to limit the effect of single-threaded performance on the overall benchmark results.
As for testing, I would want to know the CPU usage of the tiks while running these tests, its most likely the switch chip will handle the entire data path there can be exceptions. Can you also comment to what type of SFPs you got for that price, I assume nothing past 10km stuff (and normal 1310nm). I am happy to hear that the intel SFPs work in the Melanox cards as past experience with Intel+SFPs is they don't work with anything but themselves (the tiks accept pretty much anything). We have been heavily using broadcom cards as a result unless the setup specifically calls for Intel (seems usually for their offload functionality).
I don't know if I've seen Jeff this giddy before!
I'm a simple man. I see 100Gb lights, I smile.
Nice cable shirt!! Where did you pick that up at: I want one!
I couldn't help but spit my coffee out when you mentioned how much you paid for those optics....
Imagine Mikrotik being nice enough to send you two of these - we can but dream. Amazing stuff
And I just finally upgraded my workstation from an AMD Phenom II x6 1100T to a Haswell i7. (Yes, Haswell). My IP Camera network is Fast Ethernet based (10/100) - and has no bottlenecks. Man I wish I could justify upgrading. Oh well, that's what makes something like so fun to watch. I'm not a huge fan of Mikrotik/Router OS. Much prefer pfsense.
I had trouble using the connectx-4 on a board as well. It wouldn't boot with my U.2 16x bifurcated card installed. They can definitely be finicky.
Is there a reason you aren't using the -P (parallel) flag on iperf3? -P 30 would run 30 parallel streams... Not sure if that changes your single thread testing though... Great video, love watching this stuff!
Ok, this is probably dumb. But I'm curious if there is any practical use for a ram drive on your server shared over the 100 Gbps network.
I'm also curious how a 100 Gbps connection compares to 10 Gbps in latency.
Also, regarding your future plans to use an NVME pool, I've heard that ZFS actively hinders NVME performance (according to a presentation by Allan Jude from last year), so a different file system for comparison may be interesting.
i like that EIA/TIA 568B shirt :)
I like the Ethernet colour code T-shirt.
Great video as allways!
But why, MikroTik, did you put the AC power inlets on the same side as the network ports?
a lot of networking rack enclosures don't have access in the rear so hotswapping the PSUs would be impossible. This switch is trying to target the broadest possible audience
some rack cannot access the back of equipment like the telecom operator use in bts
@@marcogenovesi8570 this ^^^ a lot of places you simply can't get to the back of the gear (without disturbing a bunch of other stuff which would defeat the purpose of having the hot swap power supplies)
Which sfp+ and sfp28 nics work with both truenas Core and scale?
Those bottom power plugs shown at 9:59 & 13:39 aren't in all the way which is a fire hazard since it could lead to arcing so definitely get those in all the way, and you should get that dust off too so it doesn't get into the sockets.
Anyway, great vid, it's always fun to sit back and watch your videos
I bumped them while wiggling behind the rack. I did fix them.
@@CraftComputing Good to hear 👍
does it work for improving self hosted Multipath TCP (MPTCP) -& MPudp- throughput on the lan.
-does it have working openWRT drivers-
Assuming since those pci express 100gb/s nic's support proxmox out of the box. Is there guarenteement that those same nic's could work with an actual Ubuntu LTS that is not in a virtual environment? I'm looking to build a datacenter from ground up within next year, speed is priority. Already have the switches in mind that I'm needing to get.
I think the elephant in the room is how the hell you got those optics for 5$ each
Thats crazy LOL
I have a 25GB Raid 0 with a mix of music/documents and mostly under 10GB media files server and on a 10Gbe connection that takes most of a day to backup LOL
Question, is there any place I could find a NR1200 storage server? Your link for it is down so I was wondering if you can still buy them for cheap. Thanks!
Hi
Where can i find the link for the commands you used to update the Mellanox card firmware
Thx
Jeff if possible make a video on how to flash the bios of the mellanox cards from infiniband to ethernet in windows 10 thanks.
Recent convo with my self: I'm thinking of upgrading the house to a 10gb network network infrastructure.
Me: cool your gonna need a new switch and nic.
Also me: yeah I can get a 2x10, 4x2.5 switch and a 10gb rj45 nic, plus cable and supplies to do the house runs for like 500$
Me: wow cool do you work an IT or Tech job or something?
Also me: I'm a welder
Me:........
Also me: I have a problem.
please add the command (and maybe a link to the documentation) to change the switching mode of the network cards :)