Add a DAS to your NAS! USB Direct Attach Storage with ZFS
Vložit
- čas přidán 12. 07. 2024
- In this video, I explore the Terramaster D6-320 USB hard drive enclosure. Often called a 'DAS' (Direct Attach Storage), this type of device can be used to expand the storage on your existing home server setup, and I show how to setup and use it with a zfs pool in Proxmox. It's a good candidate to be a Proxmox Backup repository, extended file storage, or all of the storage in a mini-PC.
Terramaster D6-320:
www.terra-master.com/global/p...
amzn.to/3RELZsX
There are also smaller models with 2 and 4 bays which also use UAS (USB Attached SCSI), the D4-300 and D2-300 use the same USB architecture without an onboard RAID controller (perfect for ZFS).
Support me on Ko-Fi if you enjoy my content and find it useful:
ko-fi.com/apalrd
Feel free to chat about my upcoming projects on Discord!
/ discord
No blog post for this one, but here's my blog:
www.apalrd.net
Some links to products may be affiliate links, which may earn a commission for me.
#homelab - Věda a technologie
I love that you have a 6 disk stripe with 3 different size and types ...
At least you know the risk involved 😊
Thanks for making this no BS , info dense video. Thoroughly enjoyed it.
Glad you enjoyed it!
This is one of the greatest tech vids I’ve ever seen. I have no Freaking idea what just happened but I enjoyed watching. I was going to ask a question but I am way too embarrassed to ask here as Im a total noob . 🎉
Explains exactly what I was looking for :) great video... thanks!
Thanks for the great video once again Apalrd :) Also, thunderbolt 3/4 DAS with SATA controllers are pretty good too, and you don't have to worry about UAS. But they are a bit on the more expensive side.
I don't have much that supports Thunderbolt, so it would be hard for me to use, but yea they are totally a thing. Some with RAID controllers as well.
See this is a cool storage set up. I think this might be purchased soon.
You are a great teacher. Not too fast not too slow. Just enough details and background information. Thanks 👍
5:22 I just finished the comp TIA exam and man the USB standards are so messed up. I don't know what their marketing department was doing but really they could have simplified it so much.
Thank you, I was looking for this exact solution!
Cheers from OZ
12:08, a video on using PBS to backup PVE would be awesome. I've read about this being possible, and it would be great if you could do a tutorial on it, along with a disaster recovery process.
You make great content. You are good at explaining things. Really like your versatility and focus on open source. One complaint though: when you talk into the wireless lave mic it's often difficult to clearly understand what you are saying (that is, at least for a foreigner like me). Many times I have to go back and listen again once or twice to finally get it. It's frustrating but because you make such great content it's still worth the effort. Sure, when I crank up the volume it's easier to understand you. But that makes the 'louder' parts in your speech quite unpleasant to listen to. Use of a sound/voice compressor in your audio pipeline might be an idea to improve sound quality. That's only my opinion and not meant to bash your channel. Keep up the good work. Can't wait for new uploads from you. Good stuff.
I'll try to get the audio levels better
@@apalrdsadventures Thank you, Apalrd! Much appreciated! There's probably several different ways to accomplish that goal, but with varying efficiency, complexity and cost. In the unlikely case you have a pile of money in your basement waiting to be spent on crazy expensive audio gear you may have a look at the new Rode Wireless Pro mics. They take away much guesswork and headaches regarding audio levels. Plus real time recording with time stamps.
Thanks for the video, it s inspired me to get the 4 drive version and some WD reds for my single board storage needs.
I hope you read comments from old videos! it looks like there is considerable room inside that box. I would love to see you take out the
USB stuff! and install a Zima Blade (Box) with a HBA card. I use Unraid but you could do a second video just on the NAS Software you could install and the performance of each. Hope you see this and appreciate the idea for a cool video!
I do read comments on old videos, they all come to my comment feed.
It looks like they are using a PCIe board edge connector between their backplane and the USB card, which is similar to their other units (so they probably have both a USB and NAS version of the host card for a single backplane). So I'd also need to replace the backplane with cables, or just use the USB board and connect it to the Zimaboard (or ZimaBlade...)
@@apalrdsadventures doing it both ways and showing the difference in speed would be cool as well?!
HBA would suffocate in there, especially SAS3
I was thinking the same thing. There is enough space inside, I know the ZimaBoard could fit in there, but what about going for a cheaper mini PC? something like the lenovo m720q. Lenovo Tiny is really cheap and powerful and quiet. it has USB3 and PCI E port as well.
@@EduardoSantanaSeverino That's exactly what I'm planning..get a mini PC and stuf fit in there..maybe take the board out fo the mini PC case for more cooling..
I was thinking of using Quebec's OS, with a 6 cart dock, so that I could use VM and hot swap between tasks. If I did something storage heavy, I'd add a tape drive for backup and archiving.
I don't recall hearing what chipsets are used in this DAS for the SATA-USB bridge and USB hub. Can't find anything online either. ASMEDIA, maybe VIA? Hopefully, not JMICRON.
Was using this kind of setup for about a year. Switch to stand alone Synology 4 bay NAS.
DAS started to play up, sometimes after proxmox restart half of drives wasn't showup up. And one day ZFS crap it self with these drives.
Moved to Synology NAS about 4 months ago and not one single issue.
Most of the time the PSU is underspecced and the manufacturers rely on capacitors on the PCB where you connect the drives to compensate and allow the drives to spin up.
When either the capacitors or the PSU start to fail you get all kind of issues, including data loss.
It doesn't matter for the usecase you showed but the block size of the pool you built on the command line was 8k. If you're storing VM images you probably want to use 4k blocks (ashift==12) which is the default in the PVE GUI. If the ZFS block size is greater than the guest filesystem's block size your deduplication won't work consistently.
That's a great note. Appreciate your comment
I think one use case of these DAS enclosures would be if you wanted to upgrade 6x HDDs in your NAS, but didn't have enough space to fit in the new ones.
Would be interesting to know how well TrueNAS CORE / Scale handles the DAS enclosures.
He mentioned near the end of the video that TrueNAS did not 'play well' with passthru of USB drives all on the same path...
This is a real shame .. my fear with any NAS has always been if the NAS itself dies and I need to plug the drives into 'something' ASAP to use the data (like when the CPU on my old Synology died without warning) .. the DAS coupled with a micro PC running proxmox TrueNAS or something seems like the solution @@mdd1963
hey! thanks for the video! does this DAS passthrough S.M.A.R.T. data? Are you able to see temps and other details of each drive inside Proxmox?
I wouldn't be surprised to find out this breaks eventually. Every time /I've put ZFS on a device connected by USB ZFS kicks drives out of the pool and a scrub / rebuild etc is required. Takes a few days for that to happen though. I recall some ZFS USB bug about it but not much effort given it's not that common. Would be great if reliable though.
Using UUIDs for disk identification in ZFS is a smart choice! It ensures consistent and reliable disk recognition, making our systems more stable and resilient to changes. A true game-changer for system robustness!
ZFS uses its own internal metadata after you build the pool. It can handle drives being moved around just fine.
Seems interesting, how is the noise level of this device? I live in a small flat and my partner would kill me if i my homelab stuff becomes the noiselevel of a busy street
Windows: "you cannot use that command" user sweats: "please?" windows: "no!"
Linux: "you made a mistake, you didn't use proper punctuation and capitalisation..." user: *checks homework and correctly retypes* Linux: "you cannot do that..." user: "I am root!" *casts a -f command* Linux: "OK!"
I'm looking for this either as a 4x or 6x 2.5inch only version, or as a 4x or 6x nvme version. I've seen a video about a PCI express 4x nvme adapter showing drives independently but I do like the idea of usb for easy connection onto smaller hardware like raspberries etc.
USB3 is not all that fast, compared to NVME or even SATA. So a few 2.5" SSDs will easily outperform the USB3 uplink.
Whats the power consumption with the drives spinned-down? Thanks!
I purchased one of these after watching this video as it met my requirements, however I am getting a lot of logs in dmesg about resetting the USB handler, and the speeds are inconsistent. When doing a large transfer from it I will hit about 450MB/s for a few minutes then it drops to about 150MB/s and stays there. Wondered if you had the same experience ?
- Do i have any problem if the Truenas server is installed on bare metal hardware and not in a VM? Can I use this box as I would with a SAS enclosure?
- In proxmox could I use this with chefs?
Hi, I am thinking DAS for TrueNAS but you said there were problems.
How about zfs on it as in this video, then TrueNAS as VM, and second disk to TrueNAS from this zfs? So not passing through usb devices but assigning space from this big pool on PVE as hdd in TrueNAS ?
That power supply gives me the shivers... I'm very concerned about power quality with any USB based solution, but maybe i'm wrong?
Sure it's risky but sometimes needed. I use single stripe for game installs. If something goes wrong, I can reinstall them.
How can i learn more about nas im confused by all the info
Like trunas unraid
Different raids for the hard drives
Docker
Proxmox
Is there a good channel or video to learn this stuff thank you!
Love your contents.
Question, if you didn’t raid 0 these drives, and you wanted to use them as individual drives, is there a way to put the individual drives that are not being used into sleep for better longevity? I was wondering if this’d be possible in a usb multiple bay enclosure.
The normal Linux utilities *should* work for that (since it should just be a SATA command over UAS), but there's no explicit hardware spin down feature like the large units with staggered spin up.
Would this work for bare metal Truenas zfs? Looking to provide as backend vm storage for proxmox cluster either via iscsi or nfs over 10gbe
It will, unless they have blocked usb disks entirely on their end.
Does this DAS auto power-on after the power outage?
If it was on before the power outage, in my experience
I have a bunch of drives of different sizes, speeds and ages that I want to use. Is there HDD pool configuration that I can use with some form of parity that I can use in Proxmox directly? I don't want to install a VM with unraid as I want to share the storage between containers but I am not too familiar with RIAD, ZFS and HDD pool configurations.
.... Striped it is :P
in zfs, you can mix and match redundancy in the same pool. Effectively, zfs is a stripe of all of the vdevs, and the vdevs are one of {single disk, mirror, raidz1, raidz2, ...}.
So ideally you can pair up drives in either mirrors or raidz1, and add that up.
Remember though that you cannot remove vdevs from a pool, so if you get larger drives later and don't like the same arrangement then you have to create a new pool and copy the data over.
@@apalrdsadventures thanks! I'm watching your other videos which are massively helpful too 😂
This is for a non critical home NAS so redundancy is nice to have but not essential. More important is flexibility to add/remove drives.
I was wondering if a simple LVM setup is good for my use case.
true nas might not recognize such stuff. I've experienced in like this one, some times truenas sees only one of two disks, and even restarts and re-connections are not working. And as I understand true NAS does not even support USB-connected drives. That is a small part of knowledge after a short time of usage. Luckily I have not faced issues with corruption or lossing data as described plenty of times on each corner where mentioned disks via USB
TrueNas and USB are not an enterprise solution but home lab is fine. Im running TrueNas scale bare metal not thru Proxmox and the USB support is OK. I’ve be running the Terramaster 6 drive DAS connected to an HP microserver gen 10 plus v2 that only has room for four disks. The DAS using USB 3 gives me another pool with spares (Raid v2) for replication without saturating the network to another server. The DAS usually stays connected at all times but if you want to disconnect it hot I would export the DAS ZFS and let the drive sync scan finish before pulling the plug.
12:31 "so I did some testing off camera"
Is there any DAS that can be connected to two PCs at the same time? :D
Not USB, but SAS supports redundant operation. It's purely failover redundant though, not for you to mount the filesystem in two places at once.
How is the cooling on it?
There's a fan behind the drives and it's barely audible
the only thing better is 2 das - the price of used spinning rust is getting low enough now that everybody van have 10-20 tb pretty easily and affordably but also keep in mind nvme is down in price - many people will just want to go with a bunch of nvme - much much faster and no moving parts - i still contend people want/need dual nas with 40g dual port cards - when you have a lot of big data plus images then the faster networking is a huge plus
Such a shame its priced around the 300 mark. Half that, and it would fly off the shelves.
When you say that usb passthrough to TrueNAS was not a good experience, what do you mean? Unreliable?
Do you think in general drive passthrough doesn't work well or do you think it's related to this device, i.e. the 2 usb hubs used internally in the device?
Passing through the individual SATA devices at the SATA layer works fine. Passing through all of the USB devices at the USB layer caused the TrueNAS (SCALE) VM to crash qemu on the host. It seems like qemu is struggling to keep up with passing USB frames through the xhci emulation layer with the emulated chipset.
@apalrdsadventures ah okay thanks. This could be a problem for me then because all I have are Intel NUCs, so all of my drives were going to be over USB 🤨 (as they have lots of USB 10Gb ports and no PCIe)
Without the extra pass-through layer I was able to saturate the 10G link, and with zfs compression I was able to get quite a bit over 1000MB/s write speed, without any USB issues. It's also possible that it's a bug in either qemu, my hardware selection in qemu, or the old kernel that TrueNAS is still using, I can't be sure.
For the price of a $300 disk shelf, you could buy a proper case to expand and move your existing NAS into, plus all the necessary accessories. And you wouldn't be relying on rinky-dink USB connectors, or cheap brick power supplies. And if you didn't have a NAS, you could easily build one for that price (minus the hard drives).
I think you are missing the target audience for DAS.
Some companies don't have the resources to deploy another NAS, or to build another NAS.
A lot of use cases don't require a NAS, but just extra storage.
Also sometimes you don't want to overload the network, so DAS works in all those instances, and takes very little time to deploy and understand.
For instance per day I usually generate 1.2 TB of data that needs to be accessible for 6 months or so to me, but only 1Gb for everyone else. At the end I just push it to tape. Our network is only 1Gbps. So if I transfer it to the NAS I make everyone's life miserable, while via DAS I don't even think about it as an issue.
I think @NetBandit70 meant buying a case in which to fit the components of an existing NAS plus the additional hard drives. In this case of expanding PBS Storage that may also be a valid solution. In other cases (i.e. expanding a compact NAS-Box) it may be not.
I think it's interesting to see that there are such big USB cases for hard drives.
I just transplanted a Server from a 1U in a 100 buck 4U case that can hold 15 3.5" drives instead of 4.
But expanding that further is a bit pricey where I live. The 24 bay solution would be 500 and the 36 bay case is around 700 bucks. Now I am looking into some sas JBOD shelves :}
@@juliansbrickcity5083 You have a link on that $100 15 drive case? Sounds really good!!!
Just buy a 20TB HDD and be done with it. Way cheaper in the long run being only 1 HDD to power up instead of 6 and the built in USB hub in the DAS. Get the full bandwidth of the HDD when connected directly to the computer. People work out the pricing, running costs and extra complexity and use the KISS theory.
Have you not heard of the word redundancy? There's a reason we don't go with one big drive.
@@rpm10k.Exactly, and especially true for uses like Unraid. Your huge 20tb drive would essentially be your parity drive, so now you're buying another 20tb drive for your array. But to the above point, and in the Unraid scenario, putting all your eggs in one big basket is setting yourself up for one long ass rebuild.
The sweet spot these days seems to be the 4-10tb range. Something like an 8tb parity and a bunch of 4 and 6tbs in the array. Grab a pair of 1tb SSDs for the cache and you're good to go.
Then, if (when) a drive goes down, you're looking at hours of a rebuild instead of potentially days.
Such a pity it uses a power brick. Could have been so much neater and I’m on a mission to get rid of anything that doesn’t power off mains or USB.
all that empty space in the side.. and they couldn't just internalize the PSU.
Hell, you could even just shove that AC/DC adapter in the side, it looks like it'd fit.
What a shame.
They use the same case for a NAS, which has a bigger CPU board inside
sik RCA audio gen, brah
5:45 don't rotate drives like that while they are spinning!
Increase your voice volume.
Turn your volume up
@@zxcvb_bvcxz it's full. Other tech channels are fine with mobile phone speakers. Not this channel. I am always looking for earphones with his videos.
I'm looking to add more storage to my to proxmox home server on a asrock mini x300 , 5900g , 64gb , limited on sata ports but it does have 2 usb3.2 gen 2 or what ever they are calling it this week ?
I did get a usb3 to 4 x 2.5 dock , but it passes the drives this :-
usb-SABRENT_DISK01_DB98765432146-0:0
usb-SABRENT_DISK02_DB98765432146-0:1
Which you would think is brilliant , but if you add or remove a drive it reorders them , so forcing a resilver just by adding a new drive , so a 3 drive raid and remove a drive and it's dead .
Sound like passing the full drive and it's disk/by-id should solve the problem .