Video

You can either use LVM, thin LVM, ZFS, or thin ZFS as backing storage for LINSTOR's storage-pools. LVM (and ZFS) provide the logical volume management tools needed to pool and partition the physical storage, while also adding a bunch of functionality (like dedupe, compression, striping, caching, etc). There are no plans for LINSTOR to support raw partitions at this time.

@@mattkereczman_lb Okay, thank you for your quick response. Could you tell me how LINSTOR relates to Vates XOSTOR?

I cannot speak to GlusterFS stability. DRBD is a block device, where GlusterFS is a filesystem. In the context of Proxmox, the DRBD devices aren't really mounted by the hosts, they're directly attached to the virtual machines as block devices. The virtual machine can then do whatever they'd like with the DRBD device, which is most likely format it as their root filesystem. The virtual machines can only run on one host in the Proxmox VE cluster at a time, and the DRBD device cannot be accessed from anywhere else in the cluster while the virtual machine is running. Technically, with the virtual machine stopped, you could use something like kpartx to scan the DRBD device to find and map the partitions the virtual machine created on the DRBD device and mount those on a host, but I would think you would do something like that for disaster recovery purposes only.

@@mattkereczman_lb Thank you for your time. I guess i didn't ask the question correctly. What i meant to ask was: can one mount a virtual drbd block device from a machine that is not a drbd host itself? And yes, I'm aware of the difference between drbd block devices and glusterfs file system. The reason i asked was that gluster is the only option for a 2 node with arbiter distributed system supported by proxmox. The requirement for ceph are quite staggering for a small setup. Linstor appears to be a very good alternative, but have learned from my time with gluster that you really don't want to have to give deep into the software to find out about that one particular bug that is keeping your cluster from running stable.

@@nalixl Aha, I understand. No, you cannot access the DRBD device from a host that isn't a part of the LINSTOR/DRBD cluster. However, because of how DRBD works internally, the DRBD devices are backed by simple LVM (or ZFS) volumes that can be accessed from any of the hosts that had a replica of the DRBD device, even if you completely destroy the LINSTOR database and DRBD device metadata. You can think of DRBD as RAID 1 mirroring between hosts, so you have a full copy of the block device on each peer, and there is no "distribution algorithm" or "data striping" to consider when you need to recover from catastrophic failures.

Databases typically have transactional replication capabilities "baked in". DRBD is a block replication tool, so it replicates the underlying storage as the blocks are written to by the database or filesystem. DRBD has no concept of what a "database transaction" would be. DRBD is active/passive in nature, so you can only have one active node accessing the block device at a time, so only one instance of the database would be running in the cluster. DRBD is often used to replicate a database within an appliance or application stack where many services and filesystems needs to be replicated. It's simpler to use DRBD to replicate everything, rather than using DRBD for some replication and SQL replication for the database.

Sorry for the late reply. Everything you need is right here - github.com/kermat/linstor-jenkins-eks-assets

You will still need the KVM hosts to have the LINSTOR Satellite software and DRBD installed on them, and need to be added to the LINSTOR cluster. The LINSTOR satellites that are also KVM hosts do not need to have storage in them that LINSTOR is managing, and will use DRBD's "diskless" attachment (or client mode) to read and write to a "diskful" replica on one of the LINSTOR satellites that has storage.

As a company that develops open source software, paying customers keep the company and software development going. For this reason, customers enjoy advantages like expert support, access to prebuilt packages, and the portal script that you mentioned for registering nodes and providing access to customer-only package repositories. Our technical guides and user's guides do reference the node registration script as the easiest way to install LINBIT software, that is, from our prebuilt packages. You can do this either as an existing customer or by contacting us about free trial access. That said, almost all the software LINBIT develops is open source: DRBD, LINSTOR, DRBD Reactor, and others. You can freely install this software from the source code in their respective GitHub repositories: github.com/linbit Once built from source and installed, you can use our technical guides and user's guides as if you had installed from prebuilt packages. If you have issues installing from source, you are welcome to ask for help from the community of users at our forum: forums.linbit.com/, or through our community mailing list: lists.linbit.com/listinfo/drbd-user Also, if you just want to try our software out, we have a PPA with DEB packages here: launchpad.net/~linbit/+archive/ubuntu/linbit-drbd9-stack/+packages. We don't officially support the packages in this repository but we provide them for convenience and for testing. Thanks for your feedback. We will review our user's guides and improve them so that they can better serve all of our users.

Hi, best place to ask technical questions is forums.linbit.com/ where our community and developers meet up

We can safely say, that SATELLITE vs COMBINED is definitely not the fault. They are basically the same - on that node, a controller could be running also.

and while installing "apt install linstor-controller linstor-client" I get this error and not completing the process with success ------------------------------ Deleting module version: 9.2.9~rc.1-1ppa1~focal1 completely from the DKMS tree. ------------------------------ Done. Loading new drbd-9.2.9~rc.1-1ppa1~focal1 DKMS files... Building for 5.4.0-177-generic Building initial module for 5.4.0-177-generic ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/drbd-dkms.0.crash' Error! Build of drbd.ko failed for: 5.4.0-177-generic (x86_64) Consult the make.log in the build directory /var/lib/dkms/drbd/9.2.9~rc.1-1ppa1~focal1/build/ for more information. dpkg: error processing package drbd-dkms (--configure): installed drbd-dkms package post-installation script subprocess returned error exit status 7 Setting up python-linstor (1.22.0-1ppa1~focal1) ... Setting up linstor-client (1.22.0-1ppa1~focal1) ... Setting up libjs-jquery (3.3.1~dfsg-3) ... Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... Setting up python-natsort-doc (7.0.1-1) ... Processing triggers for man-db (2.9.1-1) ... Errors were encountered while processing: drbd-dkms E: Sub-process /usr/bin/dpkg returned an error code (1)

@@mpbraj It is a RC version so bugs and errors are expected. We have already released 9.2.9 on April 30, and will add it to PPA soon, to replace 9.2.9-rc1. Will be fixed with the full release I believe!

​@@linbit Thanks for the update...what would be the work around if I have to implement it now...I am working on Virtualization HCI solution where I have to provide HA in regards to the Storage...on Ubuntu 22 or 24

​@@mpbraj We released v9.2.9 for 22.04 (Focal) a few days ago - it should work now. I hope that helps, but if you need more feedback from our devs then I recommend heading to the LINBIT forum. It's better suited to help with technical matters forums.linbit.com/

One of our customers opened a ticket for this issue, because Proxmox suddenly upgraded the kernel version from 6.5 to 6.8, and only DRBD v9.2.9-rc supports v6.8 kernel at the moment. So there are three options right now: 1. Use v6.5 kernel instead of v6.8 2. Use v6.8 + DRBD v9.2.9-rc 3. Wait for a few days, we will release DRBD v9.2.9 next Monday (planned) Hope that helps!

Thanks for your suggestion - will pass to the team 🤔

ABSOLUTELY - far less CPU intensive compared to Ceph!

Technically speaking, yes. One would most likely need to use our asynchronous replication mode for real time replication between remote clusters. A better solution might involve using LINSTOR's backup shipping feature, allowing you to push snapshots from one cluster to another on a scheduled basis such as every hour, etc.

You can use ZFS zPools instead of LVM volume groups for configuring LINSTOR's storage pools. In this configuration, ZFS zVols become the backing storage for replicated disk images instead of LVM logical volumes.

It sounds like you most likely need to make the LINSTOR Controller highly available. If you're losing the control plane when your storage node fails, Proxmox can no longer tell LINSTOR what to do with the storage when you perform new actions such as start a VM, or allocate new storage, etc. 1) Make the LINSTOR Controller highly available. 2) Add each possible controller IP address (should be your two storage nodes) to the PVE storage configuration for LINSTOR. I would link the url's for sections of our LINSTOR User's Guide, but I think my previous comment got flagged for doing do. A quick google search for the LINSTOR User's Guide should get you to the information needed to do steps 1 & 2 above.

@@rycodge Thank you very much, but that's not how it works. I have two storage nodes and a diskless node. I installed the controller on the diskless node. When I shut down a storage node, the pve storage and vm became unavailable. I don't know what I did wrong. Are there parameters that need to be specially adjusted?

@@jmhcxh Hmm, do you actually have a "three node" cluster configured in Proxmox? For HA you'll need to setup a qdevice on the quorum node. If you power down one storage host, do VMs (backed by LINSTOR) continue to run on the other node?

​@@rycodgeIt's because linstor lost a storage node and the remaining storage cannot be read or written. It's not a problem with the pve platform.

@@jmhcxh 'linstor resource list' and 'drbdadm status' (run from each node) will inform you of the current status of each replicated resource in the cluster. The status of the resources would point you in the direction to take action. There's a number of things that could be causing this, for example a quorum node that cannot reach one of the storage nodes (in the replication/DRBD network for quorum), that would cause the remaining LINSTOR node to lose quorum for any resources that were active on the node that was down, and refuse to do anything until the other "diskfull" node is powered back up.

It's already hard to get Enterprise customers to adopt Proxmox. Ceph is a tech that has higher market recognition amongst traditionalists, so it's an easier sell. Plus Proxmox would likely need a relationship with another company (Linbit) in order to be able to offer it as an enterprise license, they don't really need to do that for Ceph. A formal partnership between Linbit and Proxmox would be a good idea, and it could be offered as part of the installation wizard.

This should help answer your question - czcams.com/video/M5VD1xXCrh0/video.html

Also, Proxmox does need three nodes to have a true cluster with HA capabilities and we need three nodes to leverage quorum. But yes, technically speaking you can install LINSTOR, even on one node only (wouldn't make sense because you can't use replication with one node), but two nodes also works, just less robust and less functionality overall in the Proxmox cluster.

Use Ceph

To start, there will always be an interruption during a failover. In the best case (theoretical), this will be analogous to the same time it takes to hard reboot a VM. Proxmox has its own timeouts for determining if a host node is down and when to react accordingly and migrate VMs. This is not unique to LINSTOR, and we suggest looking into documentation from Proxmox on how to decrease the time it takes for VMs to failover between hosts.

@@tariq4846 That wouldn't change anything. For that to be the case, it would mean that a virtual machines memory content was being persisted to disk, which would be very slow.

Yes, of course. LINSTOR can support 4, 5, or many more nodes. A three-node cluster is simply the minimum size for a proper Proxmox cluster and the ability to enable Proxmox's high availability features (which pair nicely with LINSTOR). Avoiding diskless operation is as simple as making sure each node has a storage pool available backed by physical storage where the VMs are intended to run. Alternatively, selected VMs can be restricted to only run on certain hosts containing physical backing storage in environments that have VM hosts in a mixed envronment (with and without backing storage).

@@linbit I know this, for example, I have 5 pve storage nodes and 3 replicas. I can only know where the vm is stored in the linstor controller. But the pve gui does not know this, and diskless situations cannot be avoided during live migration. I want all vms to run on nodes with storage replicas, or can linstor automatically perform storage migration? I have no idea

​@@jmhcxh When configuring the LINSTOR plug-in in '/etc/pve/storage.cfg' one can use 'preferlocal' as mentioned in our UG here: linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-proxmox-ls-configuration. This will steer Proxmox to initially run VMs where there backing storage. I'd also recommend looking into setting "Auto-Diskfull" (and related options) on the LINSTOR resource-group used by Proxmox as described here: linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-linstor-auto-diskful

While LINSTOR's setup process is more involved, the better write performance of LINSTOR could be why someone would choose LINSTOR over Ceph - this blog post should help you understand too: linbit.com/blog/how-does-linstor-compare-to-ceph/

@@linbit Three things from the blog post: 1) "The downside to CRUSH is a complete loss of all data when Ceph isn’t operating normally." Is this actually true??? There's been times when I've rebooted my entire cluster, and as the nodes comes back online, it will initially report that Ceph ISN'T operating normally, but my VMs and containers are able to launch just fine. I would challenge you to present data that supports this statement. (N.B.: I'm using erasure coded Ceph pools rather than replicated Ceph pools. But in the case of replicated Ceph pools, that statement would likely to be LESS true, because it is replicated.) 2) "If replication overhead is a bigger concern than the system’s complexity, consider Ceph" The blog post talks almost exclusive from the perspective of a replicated pool. Does LINSTOR have and support an erasure coded pool? 3) "If the complexity of the system makes you feel uneasy about recovering from failure scenarios, consider LINSTOR." Ceph CAN be (quite) complicated as there are concepts that are introduced that users, operators, and admins will need to learn. But conceptually, the concepts that are introduced by Ceph makes decent sense given what they're trying to accomplish (and how). I haven't deliberately tried to break my Ceph setup, but I have shut down nodes, one at a time, which, conventional theory would suggest that that should take down the Ceph erasure coded pool. But once the nodes come back up and online (from a reboot and/or shutdown/cold start) -- Ceph checks the pools for consistency and after about 10 minutes or so (the nodes are connected to each other only via GbE and I am only using Intel N95 Processors, so they're not the fastest processors around) -- Ceph reports the pools as being healthy. So, in a case like that, it seems to handle it just fine. But like I said, I haven't actually tried breaking Ceph by doing something that I wasn't really supposed to.

Also DBRD allows for a 2 node setup@@linbit

@@nexus1972 right. But in a scenario where there are two nodes, I think you would want a witness to prevent split brain issues

@@linbit Thank you -- I just heard of linstor and heard of ceph/guster before.. was wondering about a chart so thank you :)

No resources could be created/modified/deleted during the controller failure, all existing resources are still working as before. When the controller comes back online, everything is automatically operational.

However, the controller can be HA between the nodes. And a virtual IP will be flowing through the nodes for HA functionality.

So that would be fine for static resources like ISO files, etc. but would be problematic for VM disk files (esp for running VMs), is that correct?

Ah yes, HA controller betwen the nodes ("combined" type I presume) with virtual IP sounds great.

@ As far as "what happens to the storage pool" - Even if the LINSTOR Controller "goes down" the storage pools are still active, currently running VMs continue to run within the cluster. However, modifying or creating new resources, such as provisioning new storage for a VM, will not work until the LINSTOR Controller is available again.

From one of the team: "I'm currently running a cluster on the latest Proxmox (8.1.4). So yes, it works on the latest Proxmox releases. The CZcams video happened to launch right before Proxmox 8.0 came out, but not much has changed."

For VDO to deduplicate a write, each new write has to be compared against VDOs index to check if there is already a matching block. If there is a match, the index is written to, but the block is not rewritten. If there isn't a match, the index gets written to as well as the block. This means each write amplifies into many reads and writes (index and block). This amplification is more tolerable with fast SSDs or NVMe, but could be painful on HDDs. For VDO to compress data, each write requires some CPU time which comes at the cost of additional latency. Some CPUs can compress data faster than others, and some data can be compressed more heavily than other data. If the dataset can be deduped at a good ratio, the saved space and "block writes" are probably worth the hit to performance, but if the data can not be deduped well, than it won't be worth the impact to performance. Same is true for compression. Testing with your specific hardware and your specific datasets are the only way to know how well it will perform and whether it's correct to use.

Thanks for the comment. We never parted ways with Proxmox, is there something that leads you to believe that's the case? Here is a recent blog/video we did on how to integrate our SDS solution with Proxmox linbit.com/blog/linstor-setup-proxmox-ve-volumes/

Linstor in proxmox COOL!!

Hello, I used source code to installed gateway, after created iscsi that cannot ping the virtual IP.

Short answer: DRBD Proxy linbit.com/drbd-proxy/

@user-nl6tu4kk3z I wrote about running LINBIT SDS (LINSTOR + DRBD) on this same Le Potato cluster on LINBIT's blogs: linbit.com/blog/kubernetes-at-the-edge-using-linbit-sds-for-persistent-storage/ I wrote about performance a bit there, and I can tell you I was up against the limits of this small cluster's resources while running LINBIT SDS (and it is pretty light on resources). I happen to have LINBIT SDS running along side Ceph in a virtualized k8s cluster, and can see that the rook-ceph namespace is using 10x the CPU and 2.4x the memory that the linbit-sds namespace is using, so I'm not confident rook-ceph would even deploy into my Le Potato cluster. Good luck with whichever storage solution you choose for your k8s cluster! Don't hesitate to contact LINBIT (linbit.com/contact-us/) to chat about your project :) - Matt

Not sure we can help with this question. Our guess is there is some context switching between kernel and userspace, where with Ganesha everything stays in user space.

Thanks for the feedback! Informed the team.