Hello Friends, I am working on a storage system with some 12 year old servers. My goal is to get something working as a persistent file system while I am setting up our compute cluster. I had started to look at Ceph and BeeGFS. I have about 7 servers, and each have 8 cores and 2 hard drives bays for 3.5 inch SATA drives. Each has a 2TB hard drive now. If this works, I would like to order bigger drives, and later add more servers. What would people suggest I try with this system? I believe I have InfiniBand connections for each node, and a switch that will allow them to connect to the rest of the compute nodes, but nothing working right now. I have heard BeeGFS is easier, but Ceph handles node failure better. Please let me know your thoughts and where I might begin. I am only building the system and don’t have an established user base yet.
1 Like
From one of my friends in slack:
BeeGFS is easier, but it also has a SPoF with its metadata server. Something like scratch space would probably be OK with BeeGFS. But for keeping homedirs, etc., the options are really ceph, gluster, and commercial solutions (unless you want “one big box”, in which case you can add ZFS)
In light of this comment, where I wanted to keep home directories, I will likely try to build a Ceph system. I will try to take notes as I go.
Do have a look at MooseFS as well. It has a SPOF metadata server as well but is extremely reliable, has good performance, is more liberally licensed than BeeGFS and has a commercial version with metadata HA.