What the NUC Kind of Cluster is This???
Vložit
- čas přidán 30. 08. 2023
- forum.level1techs.com/t/3-nod...
www.engeniustech.com/engenius...
**********************************
Check us out online at the following places!
bio.link/level1techs
IMPORTANT Any email lacking “level1techs.com” should be ignored and immediately reported to Queries@level1techs.com.
-------------------------------------------------------------------------------------------------------------
Intro and Outro Music By: Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 3.0 License
creativecommons.org/licenses/b... - Věda a technologie
Reminds my several 'Beowulf' clusters for scientific computation that I built 10+ years ago by stacking miniITX small for factor I5-3450s (4core 16Gb ram each) and hooked them to a gigabit switch. Node0 was running Ubuntu with storage pools, management tools and the other nodes were booting by tftp with all the files hosted node0. DRBL linux package was handling the whole boot, os config of the nodes, network mounts magically. This cluster has served us well for dynamic fluid/structure simulation using MPI for more than 7 years and was totally unbeatable in perf/dollar. Maybe I could do that with those mini pcs. With the current CPU perf / watt and core count this ciuld be great..
@Wendell TRY THIS ^^ !!
I used to have a single big box with a 6800K and dual Titan Xps for simulations, but I remember the computer science lab in my university having a stack of 1L office PCs with i5 3450Ts. IIRC there were 3 stacks of 6 each for 18 nodes or 72 CPU cores and a combined 144GB of RAM. It was really funny hearing them all spool up their fans. You could tell how hard somebody was hitting them by how many of them spun up.
At some point HP released Open Single System Image (openSSI) Linux Cluster Project, which was also kind of interesting, but sadly didn't seem to go anywhere.
I'd love to see some more stuff specifically on hyperconvergence itself, what software supports it (XCP, Proxmox, eventually TrueNAS etc), how to use it, when to use it and maybe even using it with mixed hardware setups for the homelabber with a need for more PCIe lanes than these mini PCs can provide.
I personally would love to see more Thunderbolt (or USB4 for that matter) adoption in desktop motherboards.
Thunderbolt adoption was slowed by Intel's exclusive-use deal with Apple when it was first released. I'm surprised that it hasn't taken off since then.
Yeah lack of USB4 even on expensive boards is simply horrible, to add just regular type-C adoption is also horrible, everyone slacking.
ehhh idk, we are using docks these days at work for employees with their usb C connection points and honestly... I thought usb C was suppose to last long but it kinda sucks, they become so weak, never had that with usb A or B.
@@Decenium agree, had a couple usbc ports bend in ways usba never would
I bought my mother one of the MinisForum "game" PC's to replace the MicroITX computer I built her a couple years back. For the money, I couldn't build something better and certainly not as small. I was really impressed with the little thing.
When it breaks you cant even run around to get replacement parts. While a desktop computer can be repaired easily and cheaply.
I’m in the process of building a new lab and I have been struggling on what to get. This has helped me a bit. Thank Level1techs!
I had no idea thunderbolt could be used for networking like this. This is genius!
USB also supports networking as does Bluetooth.
@@autohmae How would you use USB?
@@physx_yt1062 for this application ? I wouldn't maybe USB4 is good enough. Just saying that all of them have IP support.
It feels like the logical endpoint of Thunderbolt. Thunderbolt is PCIe, which is basically a packet-switched network, so why not use it as the carrier for another packet-switched network (IP)?
So happy to see another happy outcome! Your hard work is making a difference!
Yeah, the video I've been waiting for ever since NUCs popped with two TB3 interfaces 😊🎉 Real sad, that the latency is that high though. That makes it totally unusable for replicated storage...
Wendel looking good Sir! Great content from the team. 2.5 G in a NUC is a win.
Useful info amongst the rambling
Yay finally the NUC cluster, I have been looking forward to this since you mentioned in another video
I have a Genoa 9124 with an air cooler. After boot is is actually very quiet. The cpu has a max power of 200 watts, but the chip area is enormous, so it has no trouble getting rid of the heat.
That's pretty cool, no added power consumption and heat from extra networking cards and for a cluster like this, the limit on the cable length is not an issue.
I would be interested in the possibility of using a thunderbolt 4 hub to take all of this to the next level, but that would add cost and complexity and I suspect that a hub wouldn't work with multiple hosts in the first place, as that's not what it was designed for.
Love this stuff, I looked into it a while ago but with IP over USB3 which is a very accessible connector (a lot of homelabbers don’t have modern thunderbolt systems but old laptops / mini pcs).
Would love to see if that’s possible (haven’t got round to trying it myself).
Great thing for a K3OS or similar Kubernetes cluster on bare metal OS...
That in inconsistent ping is um... concerning
In the days of old, I ran a NUC
Now I'll need at least 3 those when I muck
Cause I do a ping-NUC 1, NUC 2
Then I save a NUC for when there's nothin' else to do
FYI old lenovo micro pcs have PCI so you could add a network card :D
Cattle not pets! Build availability into the solution, not into each device! Love it!
I have often wondered why there is no daisy chain cable? For nucs with only 1 thunderbolt port this would be nice.
Love this!
I wonder, and I know it sounds crazy, if introducing a TB to PCIe chassis with a proper NIC would work better or just increase the latency and overhead. It could show if it's the interface or entirely the driver.
Also is the ping introducing significantl latency that manifests when using the cluster? I need to do more home labbing....
If you only end up getting about 10Gb anyway you could use that GoWin R86 that has two 10 gig SFP cages on it already.
You're just great!
Nice idea for 5 node lenovo m90q mini cluste, but using 10gb and no switch. This could save a lot of money. I think this could be done on plain ethernet NICs.
This is incredibly cool
In my opinion/experience, thunderbolt was a promise never even close to being realised. I'd avoid the headache personally. Maybe more mini pcs will start being designed with SFP ports which are slower but without the bs.
Like so many Intel developments over the last decade or so
Is there an active occulink m.2 card that would work with this kind of stuff?
How does the ngenius system compare to the Linksys omada?
not been here for...a year, maybe two...
What happened you look good man!
He got bit by a tick and got sick and had to change his diet.
Oh I'm not sure. It sounds like a setup for a complete Clusternuc to me.
Silly question - how do you install a PCIe 3.0 x16 card into any one of these mini PCs???
Now do it with firewire
1tb of cache? would that help an os #3d cache clockfriquency issues(handler prioraty)?
Would this work with USB3.2 (gen2x2,or whatever the 20Gbit version is called)? Even if it’s not explicitly called Thunderbolt? I’m assuming it would definitely work with USB4.
Thunderbolt is not USB.
"Soon" showed up!
The Minis Forum Quorum 🥳
While running this in cluster with 2.5gbps, doesnt it get slow?
400,000K SUBSCRIBERS WELL DONE TEAM @ LEVEL 1
How about bridging the thunderbolt interfaces on each nuc, and running spanning tree. Now you have a flat br-thunderbolt network with just one ip per nuc.
Spanning tree protocol essentially works by logically cutting links until the network has no loops. You'd end up with one node handling all the traffic between the others, even though they have a direct link going unused. That also adds the full latency to all the forwarded traffic. Running one IP per NUC is fine, as long as the routing is correct (multipath for redundancy).
ask eta about the gamecube and think makita toolbox plz and Thank you
Can this work with USB4?
I''m surprised this wasn't on the Linux channel
Level1Linux probably does not have the algorithmic reach that the main channel does, so this will do better here.
What I am hearing instead is don't bother with TB networking unless you have nothing above 2.5Gbps
or go retro, use firewire (400/800) was faster than (in theory) than 100Mb, but also only point to point.
The Jetson nanos can be clustered together to build a mini gpu cluster. Beastly for inference and more than capable for training small NNs.
I've toyed with the idea of building a luggable/cyberdeck with 5 nodes and an ebike battery in a briefcase.
I'd leave a positive comment but I'm all outta sorts without my morning new... Links with friend sharing...... I agree with guy who said we need Krista to fill in with Wandall and Ryu puppets....
I think I may have missed it, but what's with those ridiculous pings (on a local network??). Is this just a poorly written driver?
15:57
Similar things happen sometimes with regular 10Gbit LAN also, don't remember exactly what fixed it last time but when idle like only pinging then also huge ping times, but when doing iperf all is fine. In my case the problem was between one Linux and one Windows machine.
Those latencies (ping) is unacceptable for using Ceph. That's worse than a WAN connection. Sequential transfers should be fine, but for clustering, that's a no go :(
So basically iSCSI, but with extra caveats? :D
I would actually like to put 8 orange pi 5s in my system that has a 7950x, and I'm wondering how easy this setup is gonna be with the way people talk about it. This seems somewhat daunting, and I may also opt to use the m.2 slots for interfacing instead of type c or on board ethernet
What's my lowest-wattage and cheapest cost to having a home machine with: ECC RAM, 10GbE, Intel Quick Sync, and at least six cores/12 threads?
still pretty high, you are looking at Xeon D boards
The Xeon chips aren't APUs too, right? So I'd need an Intel GPU w/ Quick Sync support as well? Seems like a lot of PCIe connectivity to give up just to gain Quick Sync, that's a shame.
@@FrenziedManbeastIntel does make some consumer chips with ECC but you'll probably not get 6 hyperthreaded cores (they only do this on low end chips where there's no competing Xeon), so you'll need to decide if 4 high performance cores will do compared to say 6 older generation, more power hungry cores.
Do you need QuickSync specifically though, or just hardware accelerated encoding? Because AMD's 7000 series will run really efficiently when underclocked slightly, is readily available in 6 core configurations for *relatively* cheap, they all support ECC and have onboard GPUs so they *should* be able to do hardware encode (do double check this first but as long as the encoders are there AMD GPUs are generally well supported on Linux).
@@bosstowndynamics5488 I should have specified that I'm looking for fully Registered ECC. I'm still curious as to what the lowest Watt + lowest cost solution would be. I understand that many consumer CPUs support ECC on the data sheet, but finding motherboards with support for Registered ECC for those chips seems...difficult? Impossible?
Many i3s support ECC RAM.
ClusterNUC.
Bisides solar bitflips cancer is the main catalyst
can you do 5 node on that setup? please?
Three node is a practical limitation because there aren't enough Thunderbolt ports available.
@@CataclysmZA Only for full interconnection. You can still run a ring topology with two paths between any nodes, though forwarding through networks with this poor performance isn't going to be fun.
Now that intel is discontinuing NUC... do we now call them PUC?
what the NUC?!?
Why no level 1 news.. I mean level 1 sharing links with friends??
DeeeZ NUCz
3d vcache vgpio? pathrought? think capturecard but quantom arm meets x86 #northbrydge (threadripper ipad pro brydge keys?
We still have a situation where a low cost nvme will saturate ethernet socket becuase those mini pc's insist on not using 10gbe :/
Anyone actually run iperf or netperf over a thunderbolt-net interface like this? In order to do highspeed ethernet a lot of driver/sw tricks and hw offloads are needed to run ethernet efficiently and im not sure if just raw thunderbolt is enough but maybe it is
Couldn't you bridge the interfaces and assign an IP so it's the same IP space. Simpler config with the same design.
P2P is better. STP bad
software networking, I mean it can work but its gonna suck down half your cpu cycles and run half the speed
what is death? a blesing
Pcie turning pi 2
This video title is far cleverer than it has any right to be.
First, I finally got some internet points!
Cheers ;)
all the points, give them to him/her/they/it
I just imagine youtube sending firsts certificate
So out of the box we get 1Gb and 100ms ping? Or we could use the 2 2.5Gb interfaces for 5Gb aggregate, and 0.1ms ping?
2x10gigabit for a little setup when you first get the nucs vs 2x2.5gigabit oob is an easy choice for me, that 10gigabit is gonna be much better for me
If you are not using the thunderbolt it's not worth the 100-150$ markup over similar devices without thundetbolt.
Also if you use both ethernet ports to do this then what network interface is used to talk to the outside world?
@@marcogenovesi8570 perhaps talk to the outside world vua a wifi card in a slot where a 2nd ssd would normally go
1tb virus or "Ai" xD #uranium forge eeprom updater
Any dude in I.T. named Wendell has to be the man! LOL
risc-v cores can run an os at 1mhz zo can i volt it up to like 120dc and have the chassis run on 24v #capasotor pack+ discharge tool arcing prev.
You don't want with low security thunderbolt an hacker to use it and overheat until it catch fire😁
Jajajaja thunderbolt en linux y con AMD or nvidia hahahahade, please don't. Yet. Early adopted back & no Bueno
Escape from tarkov cheatercard fixes vigpo?
I wish this guy would learn, that the background music is very annoing... Otherwise, I love his videos.
I'm not interested in music, when watching computer-videos, and when I want music, I prefer to choose it myself