SFP+ DAC VS Fiber Latency: Copper Is Faster Than Fiber
Vložit
- čas přidán 19. 06. 2024
- Amazon Affiliate Store
➡️ www.amazon.com/shop/lawrences...
Gear we used on Kit (affiliate Links)
➡️ kit.co/lawrencesystems
Try ITProTV free of charge and get 30% off!
➡️ go.itpro.tv/lts
Use OfferCode LTSERVICES to get 5% off your order at
➡️ lawrence.video/techsupplydirect
Tesla Referral Program Offer
🚘 www.tesla.com/referral/thomas...
Lawrence Systems Shirts and Swag
👕 teespring.com/stores/lawrence...
Digital Ocean Offer Code
➡️ m.do.co/c/85de8d181725
HostiFi UniFi Cloud Hosting Service
➡️ hostifi.net/?via=lawrencesystems
Protect you privacy with a VPN from Private Internet Access
➡️ www.privateinternetaccess.com...
Google Fi Service Referla Code
📱g.co/fi/r/TA02XR
More Of Our Affiliates that help us out and can get you discounts!
➡️ www.lawrencesystems.com/partn...
Twitter
🐦 / tomlawrencetech
Patreon
🔗 / lawrencesystems
Our Forums
🔗 forums.lawrencesystems.com/
Our Web Site
🔗 www.lawrencesystems.com/
SFP+ 10 Gigabit Connectors: LC Multi-Mode Fiber, 10GBASE-T, and DAC Cables.
• Comparing SFP+ 10 Giga...
www.arista.com/assets/data/pd...
#DAC #Fiber #10gbe #10GB - Věda a technologie
A huge consideration for fiber: If you ever need to maintain electrical isolation between rooms and or buildings - especially if you're going outside or into a noisy RF environment.
I did IT in a printing press factory where the printers power cables interfered with cat5e cabling back in the day. Solution was to run the TP cable away from the power cables. Fairly impressive common ground noice being able to overpower the TPs built in common ground noice rejection.
Base-T Ethernet is also galvanically isolated
I found:
"Galvanic isolation in power equipment refers to the fact that the output power circuit is electrically and physically isolated from the input power circuit. Electrical isolation is accomplished using an isolation transformer. Physical isolation means that the output power wiring does not touch or connect to the input wiring.
All personal computers already have galvanic isolation between the input power and the computer logic built in. This is a requirement of international safety agencies in order to prevent shock hazard. Therefore, the addition of another transformer is redundant.
Many people mistakenly believe that galvanic isolation corrects noise on ground (earth) wiring. This is not correct. In fact, all galvanic isolation transformers only isolate the power wires, but pass the ground wire straight through.
Some UPS systems provide galvanic isolation. Most on-line UPS units do not provide galvanic isolation, despite the widespread misconception that they do. For example, Exide, Unison, and ON-LINE (Pheonixtec) on-line models do not provide galvanic isolation. The standby ON series by Oneac does provide galvanic isolation. Therefore, isolation is not a function of the type of UPS, but rather a feature that can be added to any UPS.
The actual benefit of installing an isolation transformer is that common mode noise fed to the computer is greatly reduced. Common mode noise can also be reduced by using noise filters, such as in the APC Smart-UPS series. The filters can work as well as the isolation transformer, especially at high frequencies where computers and networks operate. The isolation transformer is better at very low (audio) frequencies.
Computers and computer peripheral equipment are not affected by audio frequency noise on the power line. Therefore there is no benefit of the isolation transformer over filters for computer applications. The disadvantage of the isolation transformer is the extra heat which shortens UPS battery life if the batteries are in close proximity. Another disadvantage is that the UPS weight dramatically increases with an isolation transformer."
@@squelchedotter I'm far from being an electrical engineer but can you shed light on what some say about the difference in ground potentials be a big issue and starting a fire?
@@Beachnative42 A voltage is a difference between two potentials. A voltage is relative (that's why birds can sit on high voltage lines without any issue). As the ground (lower potential) can by anywhere, the lower potential between 2 different voltages can be greatly seperated. If you connect now those two lower potentials, you get another voltage, where there should be '0'... And now a current can flow and you might be in a dangerous situation.
The speed of light inside of glass is actually 2/3 of the speed of light. While signals inside copper can propagate at anything between 50-95%, and radio waves 90-99%.
It's 0.65c in twisted pair. No idea about twinax though.
The signal propagation speed in copper is actually the speed of light through the surrounding insulating material.
The plastic insulation in UTP gets you that 65%.
10G twinax cables use foam insulation which is mostly air. The speed of light in air is close to the speed of light in vacuum.
So you should get something like >90% in good 10G cables.
DAC is used in intra rack connections in high frequency trading. The order entry/market data switch will be in the middle of the rack with 0.5m DAC to the closest servers, then 1m and 2m DAC to the top and bottom of the rack. If you're renting space, you'll pay more to get better placement in the center of the rack just for those few extra nanos.
Nice, please keep exploring SFP+ and DAC/Fiber
HERESY
(Grabs torches and pitchforks) 🤣
Great Article, I noted that people should pay attention to you explanation in video that DAC is not the same as running standard copper connections; but my take-away is use DAC (as short as you can) on your network infrastructure where you can and also I notices a correlation by Arista in that an extra 5.5ns per 1 foot of of latency on standard fiber, noticed around 4.5ns per foot on DAC
I built my home rack with OM3 fibre. I showed it to my network engineer friend and her first response was "you know DAC exists right?". I just like fibre. I didn't know about the latency thought, I thought they'd each suffer similar losses due to converting into the SFP format.
I'd even install "fiber keystones" in a patch panel and put in small fiber patch cables from patch panel to fiber switch. Makes it look neat imo.
thanks tom :D
Wow the timing on this video! I just recently got into 10Gb networking and currently have fiber installed on my home lab, I was wondering about what had the lowest latency when compared to DAC cables and then this video shows up on my CZcams home page. Thank you for the information!
protocol is a factor... i work with mellanox infiniband switches that provide sub-ms latency for large HPC clusters.
That hardware works with truenas? Could you do any comparision or video? xD
wow , great video, u deserve a Hugh thumbs up .
Nice Topic. Very Cool.
This is so true.
I running 100 Gbps 4x EDR Infiniband network in the basement of my house and between nodes, I am using copper DACs but to my headnodes, I am running fiber.
Forget 10 Gbps.
Go with 100 Gbps. :D
@Bruce Mckenzie
I'm confused.
Doesn't 4.5 GB/s = 36 Gbps? (4.5 GB/s * 8 bits/byte = 36 Gbps)
40 Gbps / 8 bits per byte = 5 GB/s.
WIth my four SATA SSDs in RAID0 (Samsung 1 TB 860 EVO), I can only get around 16 Gbps max on it (around 2 GB/s) on my 100 Gbps Infiniband.
I don't have NVMe SSDs to test because I will burn through the write endurance limit of consumer grade M.2 NVMe SSDs in probably 6 months or less. (I burned through the write endurance of SATA SSDs in as little as 1.7 years out of a 5 year warranty, so I don't really use consumer grade SSDs anymore. And enterprise grade U.2 NVMe SSDs that can support a minimum of 3 DWPD are still a little rich for my blood at the moment.)
@@ewenchan1239 Samsung 980 Pro hit 7GB/s (56Gbps), I understand the write durability problem for some use cases though, but in my case I just need the speed for short bursts.
Excellent
Tom I need to correct you on one little thing. Electricity is not slower than light, electricity is exactly as fast as light. Therefore it just makes sense when you convert to another medium it adds latency. But otherwise great video. Live to see more.
Exactly! I believe it was Maxwell who demonstrated this if Im not mistaken
The transmission of electrical signals/power is the same. Electrons move cm/s.
@@carlsjr7975 but to be completely correct you need to consider the impact of the electrical and optical resistance. For example light is slower in glass than it is in a vacuum. But in a perfect world electricity is as fast as light.
@@niklasp.5847 Technically electricity in copper is faster than light in fiber.
@@carlsjr7975 Well that depends on length of the copper conductor. And the material of the fibre. For example the true glass fibres have a lower optical resistance than the plastic ones and therefore a higher speed. And theoretically if there is no electrical resistance and no optical resistance the speed would be exactly as fast. But the light electrical conversion does take time too.
We use SFP fiber to get ethernet to areas where long copper would never work due to interference due to proximity to other equipment.
Thank you so much for the video! I would never have imagined that the fiber patch cord was slower than the metallic pair, for me this is something new and something I will share for all my colleagues working with Infrastructure.
That's not actually what he said... Because of NRZ or RZ data modulation in a fiber transmission, you have to add some extra methods for interpreting the light source that comes from the fiber, therefore in order to have the minimum amount of errors in a fiber transmission, certain processes are implemented that take time (latency). This only means that fiber is in fact more reliable than copper and faster in long distances since DAC can't go long distances because of energy dissipation, magnetic field interference and other factors that limit the length. Since DAC is a more straightforward transmission method in a small distance, that extra latency that light modulation takes, makes a small diference, but nothing that obvious... Also DAC cables are more expensive, which in a big scale environment, could be a downer.
@@Daniel-wr9dw Yes for integration between distant switches I will always use fiber, I work this way for years without problems (from one sector to another, always fiber), but I had no idea that for integration between switches and servers the cable would be better and that's why I'll start thinking about using.
I had a project that would connect a server to a storage using fiber optics, today with this knowledge would not do that anymore!
Finally, a clear answer to the question with a strong source. Thank you!
Yeah too bad it's wrong
Which fact exactly? Link your source.
@@ashkaanhassan5398 Where to start, he gets a lot wrong in this video, but let's just go with the title. "copper is faster than fiber" - I take this to mean lower latency, since no one will argue that fiber has less bandwidth. Unfortunately yt won't allow me to post a link, but do a google image search for "fiber vs dac latency" and you'll find a table from multiple sources that shows 100ns port latency for fiber and 300ns for DAC. Yes transmission down copper is slightly faster than down fiber, but there can't possibly be a DAC long enough for that to overcome the 200ns advantage of fiber.
I asked around for suggestions of for applications where nanosecond latency on short cables could matter. I received the suggestion high frequency trading where first to respond wins, implemented on an instant response FPGA. But you really need direct to attachment between endpoints or cut-through-switching to ensure you aren’t getting stuck in the much more “slow” store-and-forward switching if you are competing about a few ns.
Also keep in mind how small nanoseconds are. I just did the calculation of the latency introduced by my 50 meter run of Fiber from my basement up to my office, apparently that's 300 nanoseconds, or 0.0003 ms (milliseconds). Considering one's first hop to their local router is within the 1ms range, or to ones ISP is 2-6ms. less than 1/1000th of a ms is introduced by the fiber, I don't think it would make a noticable difference, even in ultra-low latency requirements. Especially noting the difference at 4:16 in the video, a 1.641 ns difference between a 1meter DAC and 1 meter of MMF is 0.000001641 ms, an absurdly tiny number, when 1 ms is 1/1000th or 0.001 of a second. Go with whatever your requirements are. :)
There are some things regarding high frequency trading where some Stock Exchanges have boxes to introduce latency by having a 38 miles of fiber coiled up within a box just to add a slight delay too. See Tom Scotts video here: czcams.com/video/d8BcCLLX4N4/video.html
It is big deal for high frequency trading.
But they use radio, not copper cables. Because radio through air travels faster.
There is a microwave radio link between the New York stock Exchange and Chicago Commodity Exchange. Essentially a bunch of radio towers built in a straight line between the two.
Because radio through air travels faster than light through fiber.
I believe there is a similar radio link between the London and Frankfurt exchanges.
And a company placed a new fiberoptic cable across the Atlantic, using a slightly more direct route between the New York and London exchanges than all the existing fiberoptic cables. Just to shave a little bit of latency compared to their competitors.
Cory is a legend
That is showing a few ns difference. Typical ping times between two computers connected via direct DAC/fiber/cat6 cables is 0.2-0.5 ms. Saving 2 ns out of 200000 ns is not going to be noticeable.
Both yes and no.
It isn't really noticeable for the average consumer.
but as you start using unbuffered switches and have multiple jumps to go through, it gently adds up. And on top of that, a properly managed server will have far better IO response times compared to a cluttered consumer computer. Between servers, and when accessing drive enclosures, the difference starts to become noticeable.
Today on the Bench Right, because in any situation where that kind of latency matters for your application with that many hops, you’re using unbuffered switches. Lol. I’m going to say the scenario in which this happens is absolutely zero.
@@BlownMacTruck Well, it can be noticeable for an HBA or raid card talking to an external drive enclosure, especially through an FC switch or the like. Though, for regular IP networking it surely isn't an important thing to take into consideration.
Some clustered computing applications could also benefit, but for latency sensitive applications one wouldn't use IP as an interconnect protocol regardless. (PCIe, Infiniband and Infinity fabric among others are far more popular.) But not every clustered application benefits from reduced latency.
Today on the Bench Fibre channel and interconnects aren’t even close to what this video’s purportedly about, and even then, nanosecond latency is not a concern as the serialized packet architecture is designed to be for completely different workloads and the scenarios those entail. You’re so far into apologizing for this travesty of a video that no one can see you anymore.
@@BlownMacTruck I am though still stating that for most networking applications there isn't any benefit to be had. And the only IP related benefit is in larger networks where the latencies do stack. But this is still rather far from a reason to use DAC.
Another reason is that DAC cables can be cheaper, and easier to maintain (dust/gunk typically doesn't disrupt the connection), and can also have better physical rigidity compared to a fiber optic cable. (A tighter than 4 inch diameter bend can lead to the fiber breaking.)
And for a raid card it can still have a noticeable impact in some applications. For an example if one needs to read one piece of data to figure out where the other piece of data is, and sometimes one got to iterate a few times before knowing where the final location actually is. (indexing indexes, etc for an example. For spinning disks this isn't meaningful, for Flash storage it is a different story.)
But the vast majority of applications will not care. So buying DAC for latency isn't the main reason to buy it.
Just like the main benefit of fiber optic cables aren't that they are non conductive and can be strung between large electrical potentials without the risk of large electrical discharges. In some applications this is the main spec, but for the utter vast majority of cases it kinda is "meh".
This video assumes that I know what the various pieces are called and what their purpose is. I don’t know what the end piece things are, or what type of cable ends you have, or even what the black cable is. This is a very advanced video, I’ll move on but thank you.
Just get really short om3 patch cords if it’s hard to find compatible Dacs.
How much latency does a spf+ to rj45 module add? Moca adapter add 3-4 ms. I’m assuming it’s going to have a similar latency penalty.
But... a few 10's of nanoseconds of latency difference is on the order of memory access times, orders of magnitude smaller than pretty much any I/O operation being done on the server. What's left out here is that the greater latency difference is also due to the extra fixed overhead of error correction and other signal processing done over longer links. This is why the DAC cables are so short. There's probably more jitter in the interrupt processing time by the NIC in your computer trying to get across the PCI bus to the CPU.
And yeah, over longer lengths, you have to consider the differences in velocity factor (how much slower the propagation of light/RF through a medium vs. in a vacuum). But you have to have kilometers of difference to perceive this. You can also operate some of these links with varying FEC (forward errror correction) which requires blocks of data to operate on, and thus increases latency in exchange for working over paths with worse link margin.
I think the first-order design goal is how you get a reliable link between the two pieces of equipment. If you're on a marginal path where you lose a packet, then these few nanoseconds get flushed down the toilet when TCP has to wait for its retransmit timer to fire on the order of milliseconds, 5 orders of magnitude slower.
I totally agree the differences are way to short for a discussion that dac make a real performance difference in small links. The latency of different switches may have surely a bigger impact than fibre vs dac.
And no one buys the switches only because of the lower latencies.
There is always a trade of in features but it is a nice discussion in a more academic area.
The discussion why to use a multiple parity solutions like raid 6 in case of an drive failure in a raid 5 rebuild has much more real world impact than this.
You sure don’t want a store-and-forward type of switch or a normal server on a network where these ns delays matter, too many other factors that are greater latencies. I imagine the primary application that cares about this is supercomputing low latency interconnects.
No, the supercomputer guys are probably way more interested in throughput and probably 2*RTT worth of buffering in the switches to absorb a TCP window of data before end-to-end flow control kicks in. Switches with cut-through forwarding (that is, they start to forward the frame on the egress port once enough of the header arrives to make a forwarding decision) are what the high-frequency traders use. These guys are lunatics about the microseconds.
The reason you use DAC cables in practice is because they're cheaper and they use less power. (That heat on the twisted pair media SFPs is because more power, of course.) In $DAYJOB, we have a total power envelope of between 8.6KVA and 12KVA per rack, depending the specific data center facility's requirements. So the choice of the server CPU is based on the Tdp of the chip, CPU cores, clock speed, memory channels and how many server blades you fit into the rack and not exceed the power profile. And also how many ethernet switch ports in the top-of-rack switch as adding the 49th port means another pair of switches. So you do this multi-variate optimization problem based on the characteristics of the application(s) you're running - more cores, lower clock speed? If you don't do multi-thread execution in your application efficiently, then fewer cores, higher clock speed? Are you memory bandwidth constrained - so more memory channels, but more $$$ for those CPUs. The cost of the interconnect cabling isn't that big, overall, but why be stupid and throw $$$ away when you only have to go 6 or 10 feet away to the top of rack and save some watts of power while you're at it.
In doing this optimization evaluation, we've NEVER worried about a few 10's of nanoseconds of latency difference. We deploy servers a rack at a time and have a well understood application that scales out horizontally with hundreds of instances. Your mileage may vary, but I'll bet that latency of different 1G/10G/25G/40G/100G media interfaces doesn't really factor into the large scale system decisions. You choose the media based on flexibility, and cost. Do I need to go 10 meters? 10 kilometers? Something in between? How many different flavors of spares do I want to keep around?
Louis Mamakos apparently with MPI optimized InfiniBand application, practical delay on operations are on usec scale not nsec scale, so for InfiniBand MPI there is little cable impact. Not sure if there is any even faster low latency application in use today - was 20 years since my HPC studies, so I need to google decades of new info ^_^. You don’t use TCP in low latency applications, the point of them is different from ”slow” technologies because the command modes can speak to workers so fast that they almost appear like if they were on the same machine. For applications were you always need to know results of other computations that matters. If you just need a cluster for easily scaled applications then TCP and Ethernet will do just fine of course.
@@randomgeocacher IB isn't really required anymore since you can get 10G NICs with RDMA and every standard after that from what I remember has RDMA in the spec. IB used to kill Ethernet in latency because you could have RDMA where Ethernet did not. 25G/100G + Lossless Ethernet is quite compelling now as you no longer have to run dual stack.
As for DAC vs Fibre I'll pick Fibre every time for dense applications, if you've ever had even as few as 40 DAC cables to the same point it's a right pain to work with. Unless you've got some seriously nice cable management accessories in the rack you'll curse those cables as soon as you need to pull one out or add one. DAC cables typically have limited range of lengths compared to MMF and if you have to go custom length Fibre is far cheaper. DAC is great, just to a point.
According to the results the difference between the cables with the lowest (7.8345 ns) and highest (42.2407 ns) latency is 34.4062 ns, which is equal to 0,0000344 ms. :) Just FYI.
So in my rack to connect on my 10 gigs switches together I use 1 meter fiber cables multimode and ubiquiti transceivers so I should probably switch all those over to direct attach copper ? Are there any good right angle direct attach copper patch cables ? the only reason I didn't use direct attach copper is because it's harder to make look clean and my teenie brain thought fiber faster lol
What will be the latency difference if 10G SFP+ fiber vs DAC?
1:06 - 1:15
I don't suppose there'd be a way to measure exactly how much latency it adds? I'm working on something where using an SFP+ to 10Gbase-T module like that may be my only option.
Long as it's sub-millisecond I'm good, and I know that's not a very high bar to reach, but there doesn't seem to be a lot of info out there about it and I'd rather have a pretty clear answer before investing a few hundred bucks on some gear that may not even suit my needs.
Thanks for these videos, btw. Very educational.
The easiest way to answer this would be for you to tell us what your needs are
1. With DAC cables you are limited to 7m and it depends on the switch you can be even limited to 3m, been there and seen the problems
2. DAC cables you are limited to compatibility between venders, not so much with Fibre as you can use different SFP transceivers at each end
3. Distance, even at home I have 20meter fibre cables with SFP+ transceivers for about £100 pound per run total cost between HP switches.
4. No EMI issues, with Copper you'd need shielded ethernet cables
Fibre's even cheaper these days than six months ago. From fs.com, $24 CAD per 10 gig SFP+ transceiver, $17 CAD for 20m of OM3, total of $65 CAD (£38) for a full run.
This is really one of those economies of scale thing and frankly it's all relative to a term called velocity of propagation. Fiber is great for covering long distances but the velocity of propagation of light through fiber is less than the velocity of propagation of electrons over copper (meaning light travels through fiber slower than electrons travel down copper). What travels faster than electrons down copper? Light in free space (which still isn't exactly perfect to light in a vacuum) but that brings us into microwave radios which can cover distances up to 20 km with latencies less than 0.050 ms.
Where it really comes into play is not when trying to cross racks or campuses but when trying to cross entire states/countries/continents. Say you need to tie in a campus or location that is 1600 km away. So you strech a fiber pair and find (due to the velocity of propagation) that your minimum latency is close to 8 ms...but also find the minimum latency for copper is only 7 ms. Further, you almost never are able to have a direct path (unless you go microwave) so the actual path of the fiber or copper may be an additional 1,000 km so all of the sudden that 1 ms savings based on distance turns into a 4 ms savings by the time you get the point to point connections ordered up.
Again, latencies aren't going to that important between the two mediums for the average net engineer. Large enterprise, oil/gas, HFT, yes...it can be important but trading throughput for speed just isn't as big of a deal for the average admin.
I know you just did a video on the differences between media types, but it bares repeating for those who just stubble across this vid. and are reading comments. The Copper SFP+ to RJ45 modules are not discussed here, and in terms of power draw and latency in early 2020 it's a bit... Well I wouldn't want to have more than a couple of those modules in a microtik 4 port. They get pretty warm and ... physics. As far as the subject of this video, latency I think they might be a bit worse then DAC.. not sure where they place when compared to fiber though. I expect this to change with better power efficient / less latency modules going forward into the decade.
What looks cooler though? DAC or MMF?
So we are talking nanoseconds? 1/1,000,000,000 of a second, right?
So the difference between 1m of DAC vs 1m of MMF = a difference of < 2 billionths of a second...
Sounds negligible to me (considering I only need to connect adjacent switches).
Question:
Can you give me a real world scenario where a ~0.000005ms different would make a difference or potentially be a problem? I know in the consumer space there's no way there would be a distinguishable difference, but what about in the enterprise space? I always knew DAC was faster than fiber in terms of latency at the module, but didn't know the numbers were in the nano second range. Crazy.
Yeah, considering we are talking nanoseconds, it seems I was worrying over nothing. Even microseconds would be no big deal. If it was milliseconds, then I’d start worrying. I already have OM4 MMF. I was gonna get DAC, but it seems pointless considering my fiber is 0.3m long.
This becomes a problem when you start scaling to Fortune 100 level companies. For example, Twitter has MILLIONS of transactions per second flying through their clusters. Any sort of latency mitigation is important when youre talking about that much data flying around, especially for latency-critical uses.
Other angle: Sharing RAM, PCIE, or CPU hardware across a clustered network. If youre trying to access RAM from another physical box, then you want NO latency.
@@DaedalEVE Microseconds? 1us is the instruction clock of an 1MHz CPU, a bit clock of an 1Mbit line. The bit clock of 10 Gigabit/s is 0.1ns. Transfering a small 64 byte frame would take 64ns (a bit more due to overhead, but you get the idea).
5ns latency is 7.8% of that time.
High precision time synchronization for cluster communication and thus high availability/virtual synchrony.
Despite corosync doesn't need nanosecond precision that much, it is always good if you can do better
Real world scenarios where nanoseconds matter: High frequency trading and particle accelerators.
Can you make a video where you install several wireless access points in one location? Like several Unifi AC-HDs in a building or home. How to get the best coverage, how do you set them up? Mesh networks are the "new thing", but dedicated APs, where each AP is connected with ethernet is better. I would love to see this with "real world" results.
why are you not using DAC your rack in your background?
I thought Tom was talking about a Barista testing cables at the beginning.
where is the link
Have you performed analysis of real world impact on short cables, where does it actually matter? These ns delays are similar to necessary delays such as the inter packet gap in most networking protocols, which rarely are huge culprits to total delay. And a store-and-forward switch would add much more delay than the cable delay. So I have some doubts about a few ns latency mattering in applications except supercomputing interconnects. The entire stack would need to be low latency optimized and latency constrained for a few ns to matter. I’d imagine most applications unable to demonstrate any performance impact due to cable latency on a short cable.
It matters in storage servers connected to hypervisor's
Lawrence Systems / PC Pickup hmm... wouldn’t impact be limited on such applications unless your I/O per second is awesome? E.g. 10 nanoseconds extra RTT on one million operations would be 10 milliseconds total. So accessing a 1M IOPS disk the cable delay would increase 1s I/O time to just 1.010s, I think unless I’ve confused my maths. But maybe the disk server is all SSD and IOPS are crazy high, both end points are crazy optimized, maybe there is some impact. I have a suspicion that even on a pretty high IOPS disk system you wouldn’t notice any real world impact by changing cable, but I certainly haven’t tried myself or been close to any gear useful for demonstrating :)
DAC is cheapest, lowest power, and lowest latency.
So it doesn't really matter if 3ns of latency is meaningless to you. There is not much reason to not use DAC.
I use armoured OM3 LC Duplex mode fiber from ebay and i have a mediaconverter with sfp+ at the ISP switch and 15m of this cable connected to a TP-Link T2500G-10TS and i also have a wireless router 0.5m of cat6 connected,that i connect my Samsung phone to and test with the app speedtest, and i have only 4ms in ping time but i have FTTH that we as a house share and we have 500/500 and 3 ITV boxes and 3stationary pc and 3phones connected to different AP's and there is generally very low ping at any of these points, I used to have cat.5e of 10-15meter and with copper i had higher ping than with fiber, so if you do it right you get lower ping and lower power with fiber so you must either done something wrong in your video or used a dirty or faulty fibre... or those sfp media converters have you checked them with a microscope for dust and contaminants?? You'll be scared/surprised of how much dust and grime that gets inside of the fiber connectors on the fibercable and inside your sfp converter/s
Yes UTP is slower than fibre. DAC copper is faster than fibre.
I do have a Mikrotik switch running SFP+ and OM3 fiber, running to my 2 servers in my basement, from my home office/lab. I like the 10 gig transfers, and latency doesn't seem to be an issue over 20 meters. I did notice that the 1 gig POE from the Mikrotik to my TP-Link switch slows down my gigabit internet 100-200 megabits into my PC connected to it (the Mikrotik). Other computers not connected through the Mikrotik, get full bandwidth from their connection to the TP-Link switch, which is also connected directly to my router. (1 gig)
@04:25 What the hell? What SFP's did they use?
and what testing methods, we have access latency of 1.3ms at 7 kilometres with OS2
I suppose if you pay 20 buck for a cheap Chinese sfp module like g10tek or the like, anything is possible :)
They also only tested OM1, no OM3, 4, or 5.
ns not ms.
@@richardallen08 they tested SM also.
Question: are SPF+ transceivers compatible with each other? or they need to be always the same brand? (example, buy a HP 10Gbe SPF+ nic card , can you mix them with a microtik SPF+ transceiver?)
Supertamai2000 - No, not all SFT+ transceivers are compatible with one another; and Not all DAC cables are created equal. If you are running Cisco Switches , you will have an easier time finding off brand DAC cables, and have less compatibility issues but if you are running those cute Purple Switches from Extreme Networks, you better be using Extreme Certified /Branded cables; especially if you need to call Extreme for Support. Somehow they can tell when a non-extreme DAC cable is being used. We have even seen ports just not light up at all. There are firmware differences in the connector chipsets.
@@jasonklems8584 I see, so some brands have harder/harsher restrictions and try to lock down the usage on their systems. The question is how can we find compatibility sheets? Because I've seen quite a few brands, the problem is that NICs are also dependant on the chip/controller they have.
I use copper DAC for ISL, Single Mode OM4 for almost everything else.
an idea for your next project. rgb lan cables and if you can control the lights in them you get even more kudos for doing that
My only reason using fiber is just for distance
Very interesting results, I primarily use DAC cables for in my cabinet due to cost, reduced power use and increased robustness. Any idea how it would stack up to 10Gb/s RJ45 latency?
As you said at the beginning of the video, way more latency for rj45
@@LAWRENCESYSTEMS Thx for the response, It's good to know, I had assumed it was the SFP+ module to UTP conversion adding latency but missed the section where you also included built in ports.
RJ45/UTP ports have a conversion latency of 50-500ns.
It varies between different brands.
Receive typically has higher conversion latency than transmit.
Thoughts on Unifi's new "enterprise" switches? (USW Leaf and USW Pro Gen2)
Leaf is cheap, but the biggest downside at first sight is while it has redundant power supplies, they are not exchangeable, that's quite a no-go and I wonder why they took this route if they really want to play in the enterprise market, otherwise I don't know who this is made for. As for reliability and configurability I can't tell right know
The Gen2 switches are very decent, Pro models kinda pricy but 10G and PoE++ as well as (at least promised) layer 3 seems like a nice package
You're using the word "DAC" is that a brand or are using that as acronym? If acronym would that be "digital to analog converter"? If that is the the acronym, is it truly converting a pulsing light to a electrical analog signal? I thought those adapters were named "media converters for a reason.
Direct Attach Twinax Cables (DACs)
Do they make a fiber to coax converter? I want to run fiber in my rooms, but want to piggyback off coax lines throughout my home; much like a MoCA adapter, but for fiber.
Not that I know of
Lawrence Systems / PC Pickup Darn!
RFoG exists, but I’ve never seen it used outside of a telecom.
The main advantages of Direct Attached Copper is generally that it needs no expensive optical conversions.
Nor does it need a relatively expensive fiber. But can instead rely on a simple yet effective pair of copper wires surrounded by a foil shield.
But honestly, the main reason copper is faster than light, is due to the fact that the light in an optical fiber isn't even traveling close to the speed of light in vacuum.
A coaxial/twinaxial cable, can reach speeds closer to the speed of light in vacuum far more trivially than what a strand of glass is able to. Mostly since one can use materials that have a lower relative permittivity than that of glass.
Also, all optical sensors and lasers have their own response times as well, adding additional latency regardless of cable length.
Now, a signal in the electrical domain will also need a transceiver, since the LVDS signal from the SFP port isn't really ideal for just sending over a twinax.... So some impedance matching will be in order, but that is usually not that hard. (It does add latency, but it isn't like the LVDS signal were ideal for driving a laser to start with either, so similar tech will be needed for fiber too....)
Also, the velocity factor of a typical optical fiber is around 0.65 to 0.8
While the velocity factor of a typical coax/twinax is around 0.8 and up. (Some coaxial cables are above 0.98, while other types of coaxes have VF as low as 0.4, but they are optimized for other applications where latency obviously isn't important.)
Though, the main advantage of DAC over fiber is the simple fact that lasers wants a fair bit more power, and are more temperature sensitive compared to a handful of transistors doing impedance matching and signal boosting.
Also, it would be fairly trivial for a DAC cable to have an amplifier along its length to bring its abilities to even longer lengths.
Could someone reference the acronym DAC. Is it Direct Attached Copper, or is it Digital to Analog Converter?
@@bjre.wa.8681 In networking DAC is Direct Attached Copper.
In electronics and audio it is Digital to Analog converter.
It is an overlapping acronym, but at least it is in fairly different fields.
@@Ormaaj Can't find what the acronym "TIA" refers to. (Only find a disease related to stroke...)
If you mean variations in the time of arrival, or commonly known as jitter.
Then yes, it does effect peak bandwidth of a given system.
Then there is also the spread of the time of arrival, though this "isn't a thing" in single mode fiber, since the fiber is simply not wide enough for the light to consider it anything other than a straight tube. (oversimplifying a lot there...)
Though, for certain low latency connections, fiber isn't an ideal solution.
But for regular networking applications, DAC or fiber isn't a noticeable difference.
If one builds a super computer shuffling cache coherence checks back and forth, then going to Coax for that could have advantages. (And here we would practically sidestep the whole idea of a kernel and network drivers, since our connections would link between the cache controllers directly. (and yes, consumer and even enterprise grade CPUs don't have such features. But the CPUs in a 100+ million dollar super computer is a bit of a different story.))
@@Ormaaj Trans impedance amplifiers don't add much latency.
Especially not 10-100 ns.
Since if the amp were that slow, then it would struggle to work with signals up in the multi Gb/s range.
All SFP modules I have interfaced with use standard PECL drivers.
They are really boring.
It is a couple of PNP transistors amplifying a differential signal to drive a laser diode. The amount of latency in these circuits is almost nothing. (single ns is a comparatively long time.)
Or use another diode for sensing incoming light. Amplifying it, and making it into a differential signal.
There is usually some temperature compensation for driving the laser, and thermal protection too. But that is fairly analog in its implementation. Nor does it really interface much with the data path itself. Even the signal loss/strength indication is largely separate.
To be fair, one can build a fully functional SFP module out of a handful of discrete transistors, and get good performance. Downside is that it wouldn't really be the same size. It is though not hard making it fairly small. Though, more expensive than a fully integrated chip. (Not to mention the nice space reduction too, and usually a bit better power efficiency on top.)
The only really digital part in the typical SFP module is the EEPROM storing the "firmware". (typically only manufacturer name, serial number, peak bandwidth, and some boring characteristics about how it indicates signal strength.)
For QFSP and multi fiber modules, the story can though be different. A lot of QFSPs have a mux making them a bit more active, but in general, it is a similar story there.
In short.
SFP modules are in general just high speed differentially signaled optocouplers. (and one can even use them as such.)
@@Ormaaj Some modules especially QFSP ones can have some actual processing in them. Same story for the SFP to Ethernet modules.
But SFP to duplex fiber is usually as simple as it gets. One can even just get a single ended to PECL driver and use them as a fancy optocoupler, the SFP module doesn't care. I have done this myself a few times. (Reading the "Firmware" is also largely unimportant, unless one wants to know who manufactured it, when, what serial number it has, its peak bandwidth, etc. (Or just vendor lock what modules work's in one's switch.))
QFPS to regular duplex often is just including a mux/demux. And then just following standard ordering of the channels, though to my knowledge a lot of switches just considers a QFSP port as 4 SFP ports under the hood.
Since some QFSP modules just break out into 4 duplex fiber pairs. (Ie, it is practically just 4 SFP modules in one. Sharing the same "firmware". That one can then connect to 4 separate devices.)
Though, the laser driver technically does some processing, in regards to thermal management, but usually in the analog sense of the word. Similar story for the data loss signal.
The SFP standard is fairly simple, and that is honestly one of its large strengths. Since one then knows that the modules only really do one job and not much more.
Biggest exception to the "SFP module are fancy opto couplers" is the SFP to Ethernet ones. In my experience one can't just send arbitrary data to them and actually get a link between two such SFP modules. (Ie, they expect something that is in line with the Ethernet protocol. (They are also not "optical", but they are galvanically isolated since they have transformers on either end. And can therefor largely be used in the same fashion over smaller voltage differences.))
I want to bring up a Idea for a new video, because there is no info around this toppic at all. If you have SFP (no plus) - a DAC should provide a 4 GB/s link while a Module only provide 1 GB/s. But i found not a single Ressource where this is tested, if you can establish a 4.5 GB/s link over SFP DAC?
DAC ( non plus ) can't exceed 1GB
@@LAWRENCESYSTEMS There are G-PON (2.5 GBit/s) modules for SFP availible and working at full speed. The Interface link speed to the module ist 4.5 GB/s in SFP.... So theoretically DAC cables in SFP should work at full 4.5 GB/s.
@@ravenvg They only work if the switch supports them (not all do). Also G-PON are fiber NOT DAC.
@@LAWRENCESYSTEMS You are right of course. The G-PON was only a Example to show that at some switches a DAC might reach 4.5 GB/s even on SFP. But found not a single Comment online about one doing this. So the idea was it is worth an experiment, but I dont have two switches with SFP and SFP+ here to do it myself. Thought you might have the equipment for this test and this would be of interest if it really is working... In theory it should...
Copper is faster indeed. However cost much more to ground the wire. That speed aint going to add up if you have a surge and wipe out your line, nics, and any equipment living on the edge. Rule of thumb fiber for long runs or runs that involve going under ground or are easily exposed to environmental surge factors. Copper for local links
But I love Fiber.
He wasn't against fiber. Near the end of the video, he explained that fiber is the way to go if DAC isn't long enough for your connections, and if you're doing 20G or 40G connections.
Could you make a video about the differences and benefits of PON vs SFP vs SFP+ (fiber) and why ISP's usually use PON for fiber connections to consumers.
It’s so ISPs can split fibers and don’t have to have an individual strand going back to the Central Office/OLT for each customer.
You need to ask?
What about 10GBase-T (aka CAT6A copper) latency? How does that compare?
Same
CAT6A should actually be worse than 10G twinax, but better than fiber.
latency != speed.
Indeed, how does latency effect the T it takes for a large batch of files over common standards like rsync , NFS, iSCSI and CIFS ? I know latency in terms of lag for gaming but in the enterprise space, it's just dBase like MariaDB quarries and the like, no?
Latency has not much todo with speed. Its like fps has not much todo witj refreshrate of a monitor or shutterspeed vs fps on a camera. Can be linked to each other if one is to high.
Where is this Cory person that got called out?!
Can we get linus torvalds cosplay as vin diesel wearing one of those t-shirts. That would be great.
Consumer home networks: Thunderbolt is way faster, get a nas with thunderbolt.
Did I hear that right, electricity through a conductor doesn't travel at the speed of light?
You know what does? Light. In a vacuum.
@@jpdalvi WHY WOULD A AIR PRESSURE HAVE ANYTHING TO DO WITH LASER BEAMS
Schlomo Shekelstein physics
Yes
Inside copper, signals can propagate anything between 50-95% of the speed of light, depending on many factors. Inside glass, light travels at 2/3 of the speed it does in a vacuum.
The distance / latency is a thing because also inside the fiber the signal is traveling with 2/3 speed of light. A good coax cable is a bit faster and copper (with plugs) is slower. (may I find the time to test it on my multiservice meter - thanks for the r&d dokument link)
This is also the reason why brokers all get the same exact length to the brokage servers. All have to have the same minimum latency, especially for High Frequency Trading, so everyone have the same possebilities.
Yes DAC is lower power, I've seen tests with 400G QSFP-DD - Copper 3,6 Watt and SMF Fiber with 40km optic it were @ 9,4 Watt and the standard says 12W. But there are "maybe" 15W DWDM 80-120km modules in developement.
Damn. Cory got Called out.
I'm not smart enough to deal with this.
Honestly you seem like the person who would know where to find an sfp (not sfp+) Dac thats 6 inches ish long. I am wanting to use between a UDM and a unifi switch the switch is only sfp and I have read in multiple places that I can't use a sfp+ dac between them. Can you demonstrate this or would you know where to find an sfp dac?
I have also heard the same thing. I purchased a UDM Pro and was saving up for the USW Enterprise 24 PoE. In the mean time, I plugged an SFP+ DAC into my old TP-Link switch (SFP) to see if it'd work. The UDM Pro automatically changed the port to 1Gbps FDX and the TP-Link switch worked fine. There shouldn't be any problems between two Unifi products. As for a short SFP+ DAC, .3 meter is the shortest I found/use www.amazon.com/gp/product/B08Q7QW6KC/ref=ppx_yo_dt_b_asin_title_o03_s00?ie=UTF8&psc=1
Is it true that RJ45 switch is hotter than SFP+?
Yee
@@LAWRENCESYSTEMS thank you for your comment. Can i ask you why RJ45 is hotter?
Uses more power.
❤❤❤
Copper is not faster. Tens of nanoseconds are meaningless when the kernel schedules tasks with a granularity of tens of milliseconds. That's six orders of magnitude slower.
Your videos I don't need rise the reprodution velocity :D
Been saying it for years, DAC > Fiber (Depending on your needs/layout)
Fiber also moves in waves and elections moves like a train, it is basically instant. Waves takes longer than a small electron train. Very simplified!
ofc its faster, 0.95c vs 0.7c (c=speed of light)
Fiber is so cheap these days I see no reason to buy DAC's as a reason to "save money". They're harder to route and in this day and age have no real benefits. Optics can also be used with any length of fiber and are easier to route. Fiber really isn't *that* delicate. If you're really worried about it you can buy armored cables for about 15% more than normal cables ;)
Modern fiber is much stronger than people think. Corning makes fiber that you can wrap around a pencil. Not to mention incredible crush resistance.
A fiber connection also has more points-of-failure, since the transceivers can go bad and also fiber lines are kinda sensitive when it comes to bending them (unintentionally of course). If there is no need for very high transmission speeds, long distances or electrical separation, copper is the way to go
Fiber is not light speed. It's still light moving in a glass medium. (less than speed of light).
HOW CAN A COPPER IS FASTER???
If doughnuts are involved, all bets are off!
That title is a total clickbait lie.
Wow so DAC has less latency than having transceivers?! Who would have thought? haha
No seriously, is anybody surprised about this?
@feldim2425 Electromagnetic waves travel at the speed of light as well. In fact light is an electromagnetic wave.
Moreover as you said simple electrical current propagation time is very close to the speed of light en.wikipedia.org/wiki/Speed_of_electricity
Light in fiber optic cable also doesn't travel at speed of light in vacuum. "In normal optical fibers (silica glass), light travels a full 31% slower." www.extremetech.com/computing/151498-researchers-create-fiber-network-that-operates-at-99-7-speed-of-light-smashes-speed-and-latency-records
But as we noticed all of that does not matter. ;-) The real latency for cables in DAC length is in the transceivers, switches and routers. Even the switching delay of low latency switches will be at 10 times higher than the propagation time in the cable.
By the way 10GBASE-T is reaaaally slow latency wise. Because of the complex transceivers. So by that logic copper is slower than fiber if you use up 100m CAT6A twisted pair copper. ;-)
@feldim2425 I understand. Thanks for your input and comments!
Light/photons are faster than electrons!
Copper also consumes more power (measure it yourself).
That is for the MEDIA, fir the SAME length, fiber beats copper in speed, BUT, there are other electronics and data coding used to transmit on each, which could make a difference.
Bottom line:
Unless you are doing NASDAQ transactions, it does NOT matter (except for the power used, again measure it YOURSELF).
You're correct that light travels faster than electrons in a copper cable, but moving electrons aren't *exactly* what transmits the information. The data transmission comes in the form of electromagnetic waves, which travel at roughly the speed of light. Electrons do move, but at what's called the drift velocity, which is slower than the propagation of the signal.
Speed of light through glass is around .66C and is actually slower than then electrical signaling rate in copper which is around .76C.