Lizardfs: Can't write data in ec & xor modes. Only empty files can be created

[r2r200]

Hello!

I've installed lizardfs 3.12 on VM with OS Ubuntu 16.04. Created 6 disks 1GB each with ext4 and set HDD_LEAVE_SPACE_DEFAULT = 100MiB. It works fine in standard goals of replication modes.
My mfsgoal.cfg:

root@mfs1:~# cat /etc/mfs/mfsgoals.cfg
18 first_ec : $ec(3,1)

But as soon as I make lizardfs setgoal -r first_es /mymountfolder I can't write data into file using echo "test" > /mymountfolder/test.log, can't write files using cp, dd, rsync. Only empty files can be created.
In syslog I can see the following errors:
mfsmount: write file error, inode: 43, index: 0 - error sent by master server (No chunk servers) (try counter: 1)

What can be a reason of this issue?

[blink69]

You don't have any chunkserver connected to your cluster.

[r2r200]

But if I execute lizardfs setgoal -r 1 /mymountfolder, it works.
And I can see all the disks in wui:

[blink69]

I will fix my reply
You don't have enough chunkservers connected to your cluster :)

[Zorlin]

You need to keep in mind that multiple bricks (or disks) on a single chunkserver still only count towards goals once, meaning that if you want to have more than one copy you need more than one chunkserver.

There are some hacky ways to run multiple chunkservers on one machine, which then can allow you to do what I think you're trying to do, but reduces the usefulness and resiliency of LizardFS.

[r2r200]

Thank you for your answer!
I've modified my test cluster. Now I have 3 VMs. Every VM has 6 disks of 1TB and each VM has 1 daemon of chunkserver. And $xor2 works fine on it.

[r2r200]

You need to keep in mind that multiple bricks (or disks) on a single chunkserver still only count towards goals once, meaning that if you want to have more than one copy you need more than one chunkserver.

There are some hacky ways to run multiple chunkservers on one machine, which then can allow you to do what I think you're trying to do, but reduces the usefulness and resiliency of LizardFS.

My main goal is to make a backup storage on 3 physical servers. Each server has 6 disks of 8TB. And I need to make $ec(4,2) because it's the only replication mode with required capacity.
Do I understand correctly the only way for me is to make 2 chunkservers on each physical server and to ditribute parity disks correctly (so they will not cross)?

[Zorlin]

@r2r200 Basically, yes. Otherwise you need 6 servers...

As for bricks (what I call disks), also yes... On each server, you would need to have half of your bricks assigned to one chunkserver and the other half to another.

In other words, a different mfshdd.cfg file for each...

[r2r200]

@r2r200 Basically, yes. Otherwise you need 6 servers...

As for bricks (what I call disks), also yes... On each server, you would need to have half of your bricks assigned to one chunkserver and the other half to another.

In other words, a different mfshdd.cfg file for each...

I counted I can loose only one server with ec(4,2). So It's better to take xor2.

Thank you!

[eleaner]

you could try to label the disk-chunkservers to make sure different machines are used to fulfill your goal

[eleaner]

Sorry @r2r200
I wasn't clear. One parity is a bad idea - unless your data is disposable. You probably know that.
ec(4,2) gives you parity of one on servers but parity of two in case your HDD fails.

[r2r200]

Yes, one parity can be enough in my goal (for backups). If storage would be alive after hypothetical network connectivity lost. I also can use standard goal with 2 copies for more important folders.

[jwest75674]

Is there a guide on the process of running multiple chunkservers on a single machine? In my case I have a very uneven distribution of disks across the machines in my cluster, and would like to split up the single machine with a peak in disk/chunk count.

(e.g:
Server 1 Disks: 4
Server 2 Disks: 4
Server 3 Disks: 4
Server 4 Disks: 4
Server 5 Disks: 4
Server 6 Disks: 16

-->

Server 6.1 Disks: 4
Server 6.2 Disks: 4
Server 6.3 Disks: 4
Server 6.4 Disks: 4
)

Would allow a switch from ec(4,2) to ec(6,2) correct?

Edit: (Sept 29, 2020)
Multiple chunkservers on a single machine is great to increase ec partitioning across drives, but does not help at all in the event of failure at the server. This compromise worked well for me, as a hobby user, not using lizardfs in production. I completed this goal via the creation of a [email protected] systemd service, and many duplicate /etc/lizardfs/mfschunkserver2.cfg + /etc/lizardfs/mfshdd2.cfg, etc.
Though, I do not think this approach is likely to be recommended in production environments, I couldn't be happier with the solution for my own use, a use which can bear a day or two of downtime without issue, but no data loss, should it come to that point.