Getting the Most Performance out of TrueNAS and ZFS
Vložit
- čas přidán 13. 05. 2024
- After setting up your TrueNAS server there are lots of things to configure when it comes to tuning ZFS. From pools, to disk configuration, to cache to networking, backups and more. This guide will walk you through everything you should do after installing TrueNAS with a focus on speed, safety, and optimization.
Video Notes: technotim.live/posts/truenas-...
Support me on Patreon: / technotim
Sponsor me on GitHub: github.com/sponsors/timothyst...
Subscribe on Twitch: / technotim
Become a CZcams member: / @technotim
Merch Shop 🛍️: l.technotim.live/shop
Gear Recommendations: l.technotim.live/gear
Get Help in Our Discord Community: l.technotim.live/discord
2nd channel: / @technotimtalks
Here are some items I found useful when building my NAS
- Supermicro NVMe Add-in Card - www.ebay.com/itm/375100900409...
- Dell NVMe Add-in Card - www.ebay.com/itm/164864777644...
- M.2 Heatsink SSD Cooler for PS5 PCIE NVME - amzn.to/4bwsYAq
- Intel OPTANE SSD P1600X Series 118GB M.2 PCIE - amzn.to/497Np53
- SAMSUNG 990 PRO Series - 2TB PCIe Gen4. X4 NVMe 2.0c - M.2 Internal SSD - amzn.to/4buzeZz
- Seagate IronWolf 12TB NAS Internal Hard Drive HDD - amzn.to/3usqnH8
- Seagate Exos X18 ST14000NM000J 14 TB Hard Drive - amzn.to/3SC6ntT
- APC UPS 1500VA UPS Battery Backup and Surge Protector - amzn.to/499JvZx
- Tripp Lite SMART1500LCD 1500VA UPS Smart Battery Backup (rackmount) - amzn.to/3SOXhLr
(Affiliate links may be included in this description. I may receive a small commission at no cost to you.)
00:00 - Intro
01:09 - Pool Configuration
02:11 - Mirrored VDEVs
03:25 - Dataset Configuration
04:37 - Sync vs Async
05:20 - What is ZIL
06:15 - The More ARC the Better
07:11 - ARC Tuning with TrueNAS SCALE
08:03 - L2ARC (Increase Read Speeds)
09:11 - Increase Write Speeds
10:32 - What is SLOG and How It Helps
12:36 - Networking Considerations and LACP
14:11 - IOPS vs Network
15:05 - VLAN Routing Considerations
15:37 - Snapshots
16:47 - Backups
17:22 - Alerts
17:45 - UPS
Thank you for watching! - Věda a technologie
Great video, of note TrueNAS Scale will have automatice ARC cache to be more than %50 later this year! :)
Thanks Tom! I saw that's coming in 24.04 after creating this! Thank you for all of your wonderful TrueNAS content!!
Is it going to be a configuration setting you can define or just larger than 50%?
@@justinknash The documentation says it will match CORE, which uses up to 90% of available RAM.
Well its meant to be on dragonfly (or is it dragonfish?) beta, which i think is out now, ive not tested just wat i heard, think i'll wait till its in stable myself. Toms vid works for me for now, definitely looking forward to it though
That was a great health check for me, we must have watched the same videos from Tom :)
This was fantastic and will be a great resource to reference down the road. Great job Tim!
Thanks Colten!
Over all, not a bad ZFS video, but I took some hasty notes while watching, here they are for you, happy to expand on anything or clarify on anything that I did a poor job on while not paying 100% attention.
You say faster resliver times on miorrs, but not if you are using mechanical media where the act of resilvering is very hard on the disks and if they where deployed at the same time are more likely to fail togeather during a heavy operation like that. That is why you tipicaly only go raidZ2 or raidZ3 on mechanical.
Also as far as miooring goes, you get 1 disk of write speed per vdev, but 2x the read, or more if you have a 3+ wide miorr.
Would only use encryption if you hae a proc that supports AES-NI.
Compression: LZ4 is safe to leave on and use everywhere, I prefer ZSTD-3 in most cases.
ARC: The 1G of RAM per 1T of disk/data is more to do about metadata managment, the goal, keeping all your block lookups in RAM. You would want to stack extra if you have a heavy workload, you also need to know block size affects this requirment, 1M blocks need less that 16K blocks.
IX is working on fixing ARC on Linux, give it a bit.
L2ARC: Yes, your disk layout needs to be faster than the pool or it is the new bottle neck, you should also be carful about L2ARC, it requires RAM to function also as the block map in ARC has to refrance the L2ARC, iirc it's something in the neghborhoot of 1:5 RAM:L2ARC.
Write speed: you are mixing your units a bit funky I think, 150MB/s is ~143MiB/s and 1G eth is ~125MB/s (119.2MiB/s) but realworld you are looking closer to 1G being ~960-980Mbps, or around ~120-122.5MB/s. 10Gbps would be ~1.25GB/s (~1.164GiB/s) but real world, maybe closer to ~8.6Gbps on 1500MTU and maybe ~9.6Gbps on 9K MTU.
SLOG: Their use to be stuff like ZUSE RAM drive with a SAS interface, but now a day we have things like 3DxP or PMEM, the goal here is lowest commit latancy, hense the prefrance for for NVRAM/3DxP, if you are using cheap or low qualty SSDs that do not have PLP and do not report a write as good until it's in nand not just in the DRAM on the SSD, huge latancy problem, if you get a crappy consumer SSD that says it has PLP or just reports the data is commited even if it's only in dram but not protected, you can have data loss or coruption. You want something that can ack writes as fast as it can to really be of use here.
Network > disk: You also need enough RAM to hold ~5 sec (the default transaction group timmer iirc) of inbound data in RAM before it goes to SLOG/Pool.
You can also add vlan interfaces on the ports and restrict both client IPs and services to each as well. This also means you don't have to make static routes to avoid going out of the default gateway if you do something like have your UI/mgmt on 1G and all data serviecs on a 10+Gbps link.
Boy, your comment NEEDS to get pinned
The best thing I did for performance personally is moving the ZFS metadata off the rust to a Special Metadata Device (SMD) on mirrored Optane NVMe's. Having the metadata on SSDs makes browsing files, moving files, batch renaming files, etc etc, so much faster. It's incredible. PLEASE remember that if you lose the metadata the whole pool is dead, so definitely use reliable SSDs in at least 2x mirror if you want to use SMD in production.
Sounds cool! How do you set an SMD to store ZFS metadata?
Agreed, really shines when you are chugging through lots of tiny files. No metadata lookups means you get like at least a 2x speedup with small files, and I set any files under 32kb to be stored on the optane ssd's. Honestly was not expecting such an improvement, kinda odd this video doesn't really mention it.
Hey Tim, that was hands-down the best overview I've watched on TrueNAS to date. Ticks all the boxes for a self-confessed geek who's taking his first steps into building my first home Server/NAS on a £80 HP Elitedesk 800 G3 SFF i7-Gen7 256GB SSD 16GB. Just setting up TrueNAS Scale as a VM on a Proxmox 128GB boot SSD with 2x4TB Seagate Iron Wolf (mirrored) plus a 1TB Gen 4 NVMe L2ARC. Your presentation style is truly engaging, simple to understand for tech-savvy newcomers and clearly illustrates the core concepts whilst framing them with real-world examples. Thx again ..sub'd👋
Thank you so much!
I've read/watched a ton explaining zfs architecture concepts in my journey to setup a Scale build, but this (especially with your illustrations) has been by far the most clear/helpful. I will say, for the average homelabber there seems to be a lot of "general consensuses" about slog, l2arc, and some of the other stuff not being impactful or helpful enough to worry about doing. Would love some commentary on when/where each is useful or not. Thanks!
Excellent video. I use TrueNAS at its most basic functionality because I'm not savvy, but this guide really helped explain a lot of the questions that I've had.
as someone who has been using truenas for a while now, at least a good few years, this was STILL helpful for understanding things better, thank you so much for the highly detailed explainers here, even having watched other videos explaining things, you still helped me better understand some things I thought I knew well enough and did not, thank you.
Thanks Tim, very informative and just at the right time as I'm in the process of switching over to truenas scale!
The combination of @TechnoTim and @LawrenceSystems in providing all of us with such an incredibly detailed yet easily digestible explanation of these systems is invaluable. Hats off to you both!
SCREW SAFETY! MORE SPEED 😁
Great job on this one Tim!
🚀 Thanks Jeff!
Great Video Tim, I am new to ZFS & didn't know much but you cleared up so much for me like L2 Arc, Slog, ZIL. Thanks and always sending good vibes.✌🏼
Really enjoyed the presentation style AND the content, right speed of delivery and technical detail, with the all important explanation of each part of the UI and whether it's needed. Im going to explore your other videos for LAGG, i'm having terrible trouble setting this up on Scale 23.10 and a Unifi XG16.
Great job. Next maybe a video on how to utilize said UPS with something like NUT. Can't wait to see the colocation part of your adventure.
agreed explaining NUT would have been nice to expand on when it comes to UPS.. being able to have truenas automatically power itself down after passing a certain battery percentage or time frame of being on battery would have been good to talk about..
I'll have to watch through this again and take notes. Hopefully it will apply to scale as well since I'm switching to scale here soon.
This is SUUUUPER helpful! Perfect timing for me on my HL15 tuning, appreciate it!
Perfect timing Tim. I am in the process of transferring 25TB of data off my Truenas Core server and install Truenas Scale. This video answered many questions I had. Thank you for the quality content.
Can't you just import your pool in Scale? I am switching this weekend also.
@@Makaveli6103 yes, you can. I just want to make a clean install.
One of the best most useful videos I have watched in a long while. Thank you! TrueNAS Scale is something I am really interested in but very limited videos recently. It is changing fast and I can understand how hard it is to keep up. Would love some future app setup and setting up apps with a commercial vpn like PIA for reverse proxy.
Awesome video Tim. I learned a lot about ZFS and TrueNAS advanced features.
Wooo this is gold! Thanks for that very detailed and clear video.
probably the best video i've seen to discuss the basics of zfs (truenas)
Outstanding Video! Thanks a LOT! Learned some many new things with this video. Really, thanks for putting the time and effort.
I super appreciate this video. Thank you so much for making it!
This guide was great, nice work
I went through ten other videos and didn't understand anything till I got to this one, thank you for explaining everything so simply and so clearly!!! ! How do you handle security?
Brilliant video mate! Loved it!
great synopsis for any NAS setup, really cut through various actionable topics. good stuff
Dear Tim, Thanks for making this video, outstanding!!
I love this video. I would have mentioned Special Vdevs though. They seem to be all the rage.
EDIT: I also think it should be noted that a loss of a slog will only result in the loss of IN-FLIGHT data. So if you run a single drive (or a stripe) and it dies, you'll only lose about 5 seconds of data. But, at 10+ Gbps, 5 seconds might be a lot. Also, SLOGs only need to be about 2/8s of your ARC size max (tx group is 1/8 of ARC by default, slog can store two transaction groups before syncing, by default)
Thank you for mentioning special VDEVs. I thought about it, but the safest place for this is on my main pool since if you lose this you lose your Pool. I would have to create a pretty redundant special VDEV for this and I don't have the space! If I could have a do over, I would have at least mentioned it. Thank you!
@@TechnoTim it's all good. You could spend several hours going over all the ZFS features. This was a fantastic overview
@@TechnoTim the recommendation I've seen is for minimum of a three wide mirror (three drives all storing the same data), so it definitely takes up space. I'm also not sure how much it really speeds up reads. I imagine it's not worth it for most use cases.
@@darkpixel1128 The performance boost will vary depending on what type of data you're storing and how much of it you have. When you get into tens or hundreds of terabytes of data, directory operations can start to slow significantly which can get very annoying. I'm going to be consolidating some servers soon and I'll be adding drives for storing the metadata, but I'll be going against "standard practices" as my tolerance for data loss is high, and the possibility has already been mitigated. I've got two spinning rust pools that are ~100TB each. I'll be using a single Intel 2TB NVMe drive on each. (mainly because I'm out of expansion slots and PCIe lanes... unless Wendell over at Level1Techs reveals the specifics of his NVMe carrier boards) I'm comfortable with that for two reasons. 1) Those intel drives are pretty darned reliable. Orders of magnitude above the other drives in the array to the point where it's still more probably that I lose 3 drives on a RAIDz2 VDEV before the Intel drive dies, and 2) I have a near-live onsite backup online, so all the data is still accessible, so restoring the original pool and accessing the data in the meantime is pretty trivial even if the Intel drive does die.
@@TechnoTim As I mentioned in my (wall of text) post, non-enterprise admins should absolutely NOT be using L2ARC. It is a trap.
The only way to get real performance gains is to understand a little more about what each pool will be used for and the types of files/access on that pool...
...Then set up Metadata and Special Metadata VDEVs using that understanding.
NOTE: You can add Metadata and Special Metadata drives to your pool in-place. It will start using them immediately. It just won't move existing Metadata/Files to them. You will need to fiddle with stuff to get the stuff moved, similar to how we will need to jump through some hoops when RaidZ expansion finally rolls out.
ZFS special device (SSD mirror) can also help you increase speed and improve on IO … definitely a use case for those who have tons of small files on spinners.
Thank you! I mentioned this in another comment but I was a little scared to move this off the pool since I don't have a lot of additional space for all the additional drives I need to make this redundant enough not to lose my pool. If I could have a do over I would have at least mentioned it and why I chose not to do this. Thank you!
Don’t see it mentioned in the comments so I thought I’d mention. The backup capability in trueNAS is VERY easy to tie to popular cloud providers for offsite backup. I use back lade for the data I consider critical so I have a backup if the house burns down. It’s quite affordable if you think about it as ‘insurance’.
Great video Tim! Thanks for sharing it with us!💖👍😎JP
Thanks for your insights!
Pretty solid overview. Just one nit to pick, the 1gb per tb thing is kind of a myth based on memory wanted for dedup and even then it wasn't meant as a rule. Depending on hardware, network speed, number of users and exactly how Truenas is being used a lot of people who use it purely for storage can get away with 16gb memory and people who are running jails/VM's 32. That said 'More memory, more better' so it doesn't hurt to give ZFS more memory.
I guess 2 nit's, consumer NVME don't make the best drives for ZFS L2ARC/Slog/special devices. Get a couple Optane drives to compare with and it's a night and day difference (also helps to have drives with capacitors in case of worst case power loss if we're talking best practice). P1600x's are relatively cheap and plentiful right now and either size should be more than enough for SLOG up to a 40gb network. That said you also want to be very careful with special devices, lose that vdev and you lose the pool. And after having played around with ZFS for years now honestly most home/home lab users probably wont see much improvement regardless, but as someone who likes to tinker I can understand wanting to try something out if for no other reason than to learn.
Thank you! Not nit picking at all! It's a deeply technical topic and I felt I was in over my head in some parts. I appreciate comments because it helps viewers too! As far as SLOG, mine are on Optane! (But my L2ARC is not) I didn't mention is though, I should have!
@@TechnoTim Yeah the RAM requirements being a myth are really something we need to continually be shouting out loud to help fight against the ingrained false information. That's the problem with "common knowledge" that is completely false.
More memory is great, but ZFS doesn't actually NEED any memory. Like, at all.
This is a really important thing to realize when we start using ZFS for system drives.
You probably don't want Your Proxmox Host using 64GB of your 128GB of RAM for ARC, just because you installed it on a ZFS mirror.
Your VMs will probably be trying to cache stuff anyway. There really isn't much need for the hypervisor to ALSO be using RAM to cache the same stuff.
More of this type of intro! Thank you.
what a fantastic video! I really needed this about a year ago rofl
Nice video Tim
Hi Tim, Great video as always.
It would be really interesting to see a video of how other external appliances/VMs/containers within a network use this sort of storage. Currently I have a RAID array which I share through VM in Proxmox and create files in containers within a VM, however I often encounter file permission issues, or missing/corrupt data. I find it quite difficult to find resources outlining this full lifecycle, explained in a simple manner. I Have used SMB previously but maybe NFS is better, but I am not sure. I find it quite easy to find resources on setting up truenas, but not so much when it comes to other areas using it.
Thanks for all your work, I have learned so much from your channel.
Perfect timing! I am about to rebuild my TrueNAS server in a new JBOD, and this video helps clear up many questions I had. Thanks!
Thanks a lot for this video. I am planning to build a TrueNAS server to replace my 10+ yo Synology. I will use a refurb HP DL380 with 768GB of RAM, SSDs for boot, L2ARC and SLOG, and 12*12TB 5400rpm WD NAS drives, and 2*10GbE + 4*1GbE eth ports. Looking forward to see it in action :)
Wow this video is so good. I already knew most of these concepts but were still confusing. Great stuff!! Beautiful balance between deeply technical but easy to understand.
3:36 For me a reason to turn encryption off is write NOP optimization.
ZFS can just skip writing to file if current content is the same as what you are writing. Very handy in some cases.
ZFS does this by comparing block hashes. But it isn't possible with encryption.
Another annoying thing is that truenas doesn't allow you to have unencrypted dataset inside an encrypted one. So the only way to have any unencrypted datasets is to disable encryption on the root level.
Great video, thank you for explaining TrueNAS in a deeper way. I’ve only used it a bit as I’ve been on unRAID for more than a decade and haven’t been too unhappy to make a move. Now I’d like to build a second server with only SSDs and this will probably be TrueNAS.
One point on backups from a backups nerd. 😜 It’s worth noting that the backup server doesn’t really need to have any drive redundancy at all. Sure it’s convenient to have in case of a drive failure, but since the data SHOULD already be on 2 other instances (3-2-1 rule) it’s not necessary. Maybe do RAIDz1 just for good measure. 😅
That should at least soften people’s thoughts on backups, as you really don’t need a complete replica of your source system hardware and drive setup. That means that you “only” need to buy 3 new drives to expand your pool with another 2 drive mirrored vDev, and the third drive goes into the backup system.
AND, thank you for mentioning that snapshots isn’t backup! 👍
Thanks for the great videos. Would you recommend TrueNAS on Proxmox VM at all? My scenario is to edit video off NAS just hobby and up to 4K. I will have 10gb direct data connection
re: backup
Depending on how much data you are trying to back up and how fast you want it to be backed up - LTO tape can be an option for some people.
The initial cost for the faster tape drives can be quite steep (I think that my LTO-8 tape drive was somewhere just shy of $5000 USD when I bought it 4 years ago), but I think for a 12 TB raw/30 TB compressed LTO-8 tape now, it's like $75 (I think) per tape, which is cheaper than a 12 TB HDD (new) and 30 TB hard drives only exist if you know and ask the right people.
Thus, as a backup solution, it works well, if you can stomach the initial cost.
The more data that you want or need to back up, you end up cost-averaging down and becomes more cost efficient to do tape than really any other storage medium.
Sync in edit datasets, do you have it on standard or always? 😊
Nothing better to train your brain like watching videos like this one..😆
Thanks, I used your proxmox things to do post installation ❤. At 01:45, what happens to the pool if the cache drive fails ?
Simple google would have worked .... 😅 it just rights to the vdev
I'm having write performance issues for a raidz1 pool of three nvme drives... Which I find very confusing :D.
I'm wondering whether moving two of those as a mirrored SLOG to the spindle pool and then putting the VM NFS datastores there would actually improve things...
I believe the recursive option for snapshots is for any data sets under the data set you're taking a snapshot of. I don't believe the recursive is for folders inside of a data set.
Will striped RAIDZ1 VDEVS be an alternative to your approach?
Since it gets more efficiency and still have redundancy
At 7:30 in the video you indicated that you would even have a link in the documentation below the video to the Tom Lawrence video where you learned this. I don't see that link.
Hey Tim,
What size and characteristics would you recommend for the SLOG mirror ssd's? I have 200TB in spinning rust and 256GB of ECC Ram. The use case is mostly to read/write large files (20GB-100GB) and backups with 20+ sources reading and writing to it at any given point.
I use 2 Intel OPTANE SSD P1600X Series 118GB in a mirror. Works great. Decently priced, links in the description!
@@TechnoTim the SLOG need max 32GB so you can use over provisioning on these drives, can make there life even longer!
This video gives you a good but somewhat luxury NAS example to using ZFS. Since 2019 I run ZFS on my Ubuntu desktop using a Ryzen 3 2200G and 16GB DDR4, I limit the memory cache (L1ARC) to 4GB. I have a 512GB nvme-SSD (3400/2300MB/s) and a 2TB HDD (2 partitions) with a 128GB sata-SSD cache with 4 partitions as L2ARC (90GB + 30GB) and ZIL (5GB + 3GB). I use a lot of Virtualbox VMs, loading Xubuntu takes
Also when setting up your pools, it's worth considering your drives. I use a LOT of used enterprise drives, and would never, ever choose less than a Z2 because they're old drives, they will die sometime. Edit: ZFS pool expansion is now available in TrueNas Scale, at least in Cobia. You can now add individual drives.
Yeah, most of my drives are used enterprise drives. I keep a stack of cold-spares on hand too. You don't want to wait for a drive failure to order your replacement.
My failures have been pretty rare. Of my 6TB drives of which I have 48 running, I've had about 3 drive failures over the last 5 years.
@@ajhieb Luck can vary though.
I've had no failures with the 12x 10TB drives I've been running for the last 18ish months.
I've had 16 failures on the 12x 4TB drives over the 4 years I've been using them.
Yes...16 drive failures on a 12 drive array. The 5 year warranty is coming up soon and that pool is going away...
@@Prophes0r That's not really luck (random occurrence) so much as it is variation in drive manufacturing. Quality/reliability can and does vary between manufacturers, models and even batches. All the more reason to keep cold spares around (or hot spares if you have the bays available)
Low budget/Powersaving option:
After running truenas systems for many years and paying for the disks and the power consumption I decided to switch things up.
I now have a low power dedicated TrueNAS system that runs 24x7 with a single SSD that serves my important files and a single large drive to serve my media.
For backup I have a older system that powers up every sunday running Unraid and using rsync I sync everything from my truenas system to unraid. (I also rsync a truenas box I have with my dvd-movie collection that is only powered on when needed)
Using unraid I can mix and match drives to create one big pool andas my backup size increases and I don't have to keep up with matching 100% of the drives that I have running on my truenas boxes. I used zfs snapshotting for years but the older snapshots take a toll on your usable diskspace.
Please make a follow up on this, for the pitfalls eg. SLOG, Mirrored VDevs etc etc
Quality content.
Thanks Bill!
That comment that ZIL is inside your pool and outside of it called SLOG was a nice touch. Always confusing those two.
At 12:53 how come and you use MTU of 1500 sand not 9000 since you dealing with large files?
wouldn't you want to stagger the age of your mirror'd drives? so that they don't fail at the same time when they do fail? even just different batches.
Are you virtualizing your TrueNAS on Proxmox? I’m building a server soon and that seems like the best option for virtual machines and GPUs, and was wondering what your experience was. I saw you had a video from 2020 but wondered if there was anything more recent.
openzfs2.2 added the ability to add a single drive to raidz1,2,3 etc
Another optimization on the networking part is to increase the MTU from 1500 to 9000 aka. enable jumbo frames (must be enabled on switch level as well).
You can enable encryption when you create a new dataset, though I'm not sure if TrueNas will give you that option. You can also replicate an existing dataset to a parent dataset that is encrypted, keeping snapshot history, etc. while adding encryption. Definitely easier to do in advance.
Great point!
does truenas support dynamic raid expansion yet? i take it we're still waiting since you talked about adding 2 drives in a mirror at a time. i was really disappointed to realize they didn't support this after doing RAIDz2 of my 4x 4TB....
planned before the end of the year... crossing fingers...
Will sync work with SLOG?
Zfs dedup is great when you have a couple of VMs on the same OS. Those VM disks use a lot less space.
Storage space is a lot cheaper than RAM and not that limited in max amounts per socket.
Can you get the proxmox helper type scripts for truenass, that would be a game changer for me
Are you still using Cat5E for your 10gig connection?
A question on slog. Since it's a critical vdev unlike l2arc to are you using DC-grade ssds for it? Do you have any suggestions on how to choose those (models/volume/TBW)? These are pretty expensive. I assume optane would be best :)
I use Optane for this, I have a few links in the description if you are interested! Decently priced (ATM)
@@TechnoTim thank you. I've checked, and here in Mordor even second hand ones are pricey
@@BoraHorzaGobuchul I've got mine new on ebay (P1600x 58GB): 2 for $91.78, $14.64 Shipping from US to France, $21.29 french tax = $127.71 not cheap but It's really a big increase in speed. For me the mirroring may be a bit luxurious because I think that the risk of data loss is only when the SLOG fail and the serve crash in the few second later (but I'm looking for a confirmation...)
If i remember correctly SLOG is writecache but not as you described. SLOG is not inflatable. SLOG is by default in ZFS FORCED to clear itself (write all stuff in SLOG to zpool) every 5sec. In real life, it will speed up first 5s of writing to max of your network, but than you will bump in to speed difereces od ZVOL(slower) and SLOG(faster). SLOG will be still half full and new data from network still incoming. If your network is 10Gb(bit)/s, SLOG is NVME drive with R/W speeds over 2GB(Byte)/s (20+Gb/s) and your ZPOOL wite speed is 500MB/sec (cca 5Gb/s) u still bump in to write wall of zpool after few seconds.
SLOG is mainly for synchronous writes like iSCSI, DBs etc... SLOG fastly receive data, confirms data writes, start to fill for default 5s than optimize all data in SLOG for write with minimal IOPS weight on usually mechanical drives and write to zpool.
For unifi devices with L3 routing, if you have the L3 switch do the routing for the vlans in question, not the udm, you no longer get a penalty hit on your speed. There are probably downsides I am not aware of.
So in theory, the best of both worlds for ARC/ZIL sync vs. asnyc is having a dedicated "VM" dataset with sync enabled, a dedicated "sensitive / has to always exist dataset" with sync enabled, and a dedicated "arbitrary data that I don't care if it always exists or not dataset" with sync disabled? I'm asking because I think that if you have sync disabled for VMs, VMs could misbehave (I guess). Otherwise I'd make the "VM" dataset async as well.
Hi Tim,
I need your help.
I own
TrueNAS Core 13.0-U6.1
16GB RAM
2 Pools with 6Nos 4TB each HDD (WD Red Pro)
Since I built 5 years ago, I see its dead slow.
I even can't create a new folder and hit enter.
It will say no such path. So I have to hit enter after some delay.
I ran my life with this for 5 years assuming the FreeNAS/TrueNAS are like this only.
My other Synology NAS works like a charm with ~100Mbps read/write speed.
Where as this TrueNAS ~10Mbps aprox.
If I start copying a big folder from NAS to local drive for backup.
Then I can't even work with NAS at same time.
It will lose the network.
But my network intel 1Gig card is very good
Same issue with onboard LAN too.
I use best LAN Cat-6 cable too.
I did several updates too from FreeNAS to TrueNAS code latest.
(Though all latest TrueNAS Core 13.0-XXX shows incorrect drive size on windows 10)
Where could I be wrong ?
Can you or some one help me online or email chart ?
It might be easier to replace or add 2 new disks, but if you would use RaidZ1 with 6 drives you have much more space and therefore the might be no need to expand for a far longer time than with mirrors, it always depends a lot on what you want to do and what data you have, there is no rule of thumb in many cases :P
Ever tried to use Optane drives as SLOG? they have TONS of IO and I think they're great if you need that (for example VMs, Databases), combined with pretty large ARC it can be a great solution for storage.
I'm using one with a VDEV RAIDZ2 of 8x18TB and another with a VDEV RAIDZ1 of 3x18TB and 256GB of RAM in a DL380 gen9 with 2 xeon E5-2640 v4. Works fine for me.
For the L2Arc and Slog do you use consumer SSD?
For the SLOG you need a small capacity (less than 32GB is the recommanded size for a 10Gb network) and very high endurance/low latency, so usual consumer SSD is not recommanded. All the data written to the ZPOOL is also written to the SLOG so a lot of writes!
For the L2Arc it's less critical and a good SSD should be OK.
I'm currently running a pool of 8x8TB HDDs with 256GB RAM and 2x1.6TB NVMe for L2ARC and SLOG. It's running at it's limits, with 400 IOPs constantly, iirc, and 57 VMs. The other day I had 3 VMs uncompressing and copying files, and the performance tanked, it was impossible to work, and other VMs were complaining. I'll be upgrading next week to 2x(8x7.6TB RAIDZ2) All-flash SAS3 enterprise SSDs. The only issue I have is that I don't know much about performance tunning with all-flash storage on TrueNAS. I'll probably disable ARC and L2ARC, 'cause reads should be almost as fast. So, 256GB of RAM makes no sense anymore. I use always sync on iSCSI, but I don't think I need SLOG anymore either. I guess I'll see how it works once I finish migrating all the data.
Why aggregation instead of SMB multi-channel?
@13:13, yes those 10 gig pipes😂
hand gestures don't always turn out the way you think they will....
I hope the special vdev is mentioned in this video..
I guess a follow up video for the "special vdev" wendell (l1t) had some neat thoughts on using it to accelerate a pool of rust.
Coming to UPS. Why the hell am I loosing data from my ZIL/SLOG during a power outage? Actually the ZIL or SLOG ist living on persistent media. Shouldn't it pickup normal operations after a reboot after the power outage. Thats the part of the concept I did not understand. The ZIL/SLOG could even make sure that files that are only partly written to the pool because of power failures could be resumed in total after the power outage - I think, but for sure I did not write the algos so probably their creators did think different. So in my eyes, the only data that could get lost due to a power failure on a Nas should be that in RAM - and even there only the part that is not yet written to a disk/SSD. Am I wrong?
My understanding of SLOG is not the same. Am I wrong? :
The SLOG/ZIL is read only when ZFS start. That's when ZFS check that all the async writes are written on the ZPOOL and if there's some missing they can be read from the ZIL/SLOG and write to the ZPOOL so that there's no data loss. It's a log, not a cache. When a client write data to a zpool there's no read from the ZIL/SLOG, the data is in RAM, why read from a slower disc?
So mirroring a SLOG is good of course but the risk of data loss is only if the SLOG drive fail and the server crash SIMULTANEOUSLY. If the SLOG drive fail during normal usage of the server, ZFS just put it offline and use the ZIL instead and there's no data loss, only a drop in performance.
If you look the drive usage of a SLOG you will see only write, no read.
2:55 _"But the more mirrored VDEVs you have, the less likely this is to happen"_ Uhhhh no. When dealing with a striped mirror (which is how ZFS handles multiple mirrored VDEVs in a single zpool) adding more mirrored VDEVs will be adding more failure points, without increasing your redundancy so the likelihood of total data loss goes _up._ If you increase your mirror "depth" and go with 3-drive mirrored VDEVs, you're adding failure points, but you're also adding redundancy, so in that case you're likelihood of data loss goes _down._
I think I get what you were trying to say in that with a single mirrored VDEV, if you lose two drives, then you've lost it all , but as you increase the number of VDEVs and assume the loss of two drives, the likelihood of having the two failures occur on the same VDEV goes _down._ (But again, this is offset, by the greater likelihood of having multiple drive failures because of the added drives)
In short, all other things being equal, the more VDEVs you have the greater your chance of data loss.
Sorry, I thought that's exactly what I said, "the more mirrored VDEVs you have, the less likely that (2 drives in a VDEV dying) is to happen." Along with the illustration I drew I thought that was clear, I guess not. Thank you!
@@TechnoTim Yeah, sorry to be nit-picky, but I get that way about data loss. :) What you said was technically correct, but I think it conveyed the wrong message because you isolated a very specific scenario. In the very specific scenario you described (losing exactly two drives) the probability of losing two drives in the same VDEV does indeed go down. But the likelihood of having 2 drives fail also goes _up_ in that scenario, in fact more so than the corresponding drop in having them be in the same VDEV so the overall likelihood of data loss goes _up_ with the addition of more VDEVs. They way you described it was a little ambiguous and could have been interpreted the opposite way.
Again, not trying to be overly critical, I just like to be very clear on matters of data integrity. As always, appreciate your content and thanks for all of the work you put into your videos.
@@ajhieb Hey! No offense taken! I want be sure that the information is right, even if that means I am wrong, so I really appreciate the feedback! I don't think you're being picky at all, it means you are detail oriented, something that's appreciated from the tech community!
@@TechnoTim Tim, your math makes incorrect assumptions.
Your math only works when considering larger/smaller arrays with exactly the same number of failures.
Sure, it is less likely that 2 drives failing will be from the same mirror when you have more mirrors.
But any given drive has the same likelihood of failure as the others.
More drives = more failures = greater likelihood that both drives in a mirror will die.
i believe that you do not get improved performance with mirroring, the controller needs to check that the data are the same ...it is not reading half the data from each disks ...(like in raid0)
Nice video since the info you get from forums on truenas is not particularly helpful. For anyone new, dont add any of the extra features, like slog and l2arc, until you test with a basic configuration and determine where you need more performance. It will save you money. I thought I needed all those things, but after building 2 systems, mostly bulk storage, I just run with a boot drive and my storage drives. I get about 500MB/s on a 10Gb link, which is plenty and dont have to worry about the additional complexity of cost of adding the extras. It should have also mentioned you can only use about 85% of your drive space before Truenas runs like crap. So a mirror loses 65% of the storage.
Guys, correct me if I'm wrong, but either ZIL or SLOG will increase your speed ONLY when you do sequential writes. In any other cases just use fast enough disks.
SLOG allow to NOT wait until the data is written on the ZPOOL. A SLOG disc need a very low capacity (16GB recommended or more but with over provisioning) and low latency (like optane SSD). Even in non sequential writes, optane SSD will be faster than a mecanical drive... Of course if you can afford optane high capacity SSD in your ZPOOL you don't need the SLOG, better you should not use SLOG because it will slow you down but that's a lot of money...
👍🏻👍🏻👍🏻👍🏻👍🏻
I didn't really like TrueNAS personally. Way more than I need. I only have one HDD and just wanted a smb share. A little lxc file server works just great. That said of I had multiple HDDs and dedicated machine or tons of ram and CPU I'd go with TrueNAS scale.
12 Disk
Stripe over 2 vDev
Both vdev raidz2 (raid6)
So its rais 0 on two raidz2(raid6) devs.
Double performance, since 2 datasets, and 4 times fault tolerance overall, and 2 disks fault protection each vdev.
Striping ARC being nvme seems not necessary. At least from perf perspective.
It's not likely to be a bottleneck in most setups, but if you're repurposing some old drives, (as so many TrueNAS users seem to do, myself included) you're better off striping them than mirroring (or RAIDzn) as the extra capacity can make a difference, but the redundancy is pretty much pointless.
SCALE 24.04 will no longer have the 50% memory restriction by default!
I just saw this and updated my documentation site!
I use unraid but watched this because it seemed interesting. Thanks for explaining it all so thoroughly.
The ARC also compress your data in memory. If you have 16 GB RAM, 50% is used for the ARC. So, you would cache more than 8 GB. The ZIL or SLOG is only used when your server had a crash. When the system reboot, it checks if every commit was written to disk. If not, it writes the commits from the ZIL or SLOG to disk. This only happens when the filesystem was configured in "sync" mode. Always activate this when using nfs.
50% is a Linux issue that they are working on fixing but can be manually tuned, on FreeBSD and opensolaris ZFS will use more.
@@nadtz I wonder why Linux only uses 50% of the memory for zfs? Is it because Linux uses ext4 or xfs for its / filesystem and also needs cache memory for this? Or that Linux needs more memory than the BSD family?
@@yerrysherry135 I forget the specifics but it was a work around to prevent another issue that has to do with memory allocation in Linux. I also don't currently use Scale, I believe that was supposed to have been fixed in Cobia which I believe is now out so it might be fixed.
It's coming in Dragonfish later this year!
I'm looking for a confirmation: if the drive of the SLOG fail, ZFS put it offline and use the ZIL instead so there's is no data loss???
tell me where is the NextCloud folder? TrueNas Scale
One mistake though:
You mixed-up iops and throughout. ;-)
Holy shit your pool config is exactly like mine, mirrors basically. And what's more uncanny is that I also have 14tb hard drives :D
Still need to learn a bit more:
If you do not have a full copy of your precious data, in a whole different device/server, you do not really own such data.
I have offsite file level backups, but that's for DR. Would like another local copy on prem
why RSYNC vs Replication?
Based