5 ms·

I built a very similar home NAS with the newer RockPro64 and a pair of 4TB HDDs in RIAD1, all on top of Debian[0]. I found OpenMediaVault to be overkill and kep

I built a very similar home NAS with the newer RockPro64 and a pair of 4TB HDDs in RIAD1, all on top of Debian[0]. I found OpenMediaVault to be overkill and kept me from really understanding what was going on. Plus there are a million guides to setting up SMB, rsync, Borg, etc. The RockPro64 hardware is great. Very performant, especially when using the PCIe to SATA card instead of USB like the OP did. [0] https://ameridroid.com/blogs/ameriblogs/how-to-build-your-own-nas-server-with-the-rockpro64 https://ameridroid.com/blogs/ameriblogs/how-to-build-your-ow...

Naac 7y ago
I asked this on another thread, but I don't understand why you would see real world difference between PCIe vs USB3. When using this setup as a NAS, isn't your bottleneck the gigabit network from machine to machine? You'll saturate the 1 Gigabit link ( or even 2 Gigabit duplex ) before you can reach 5 Gigabits ( USB3 ) or 6 Gigabits ( PCIe )
- walterbell 7y ago
  On a low-power CPU, overhead for USB protocol handling may be less efficient than PCIe DMA?
- mrb 7y ago
  PCie 3 x1 is 8 gigabit/s. But yes you are right, the gigabit network should be the bottleneck either way.
  - rbanffy 7y ago
    For transfers, yes, but for housekeeping activities such as scrubbing, defragmenting, indexing, and deduplicating, not at all.
- pnutjam 7y ago
  RAID seems very popular for 2 disk setups, but I don't like the cost benefit. With RAID under about 5 disks, you aren't getting much speedup, under 4 disks you can't do a proper fail-out and rebuild of a larger multi-disk volume. I use 2 x 4TB disks in my homeserver, but I keep one online and have a script that brings up the other one periodically and rsyncs everything. This gives me a local backup, something RAID lacks, so I'm protected from fat fingering or accidentally deleting stuff. I also have very minimal downtime, because I can mount that drive in the place of the primary drive in just afew seconds. I run xfs on the primary drive, and btrfs on the mirror, so I can take snapshots after I rsync and maintain differentials easily. My point is, you should consider getting rid of RAID and just use the bare drive or LVS Volume.
  - effie 7y ago
    Agreed, RAID makes little sense if you want a quiet home storage that is used only occasionaly. Additional disk is better used for regular backups. If you need to minimize downtime and have the storage accessible 24/7, then RAID makes sense.
- foresto 7y ago
  IIRC, SATA over USB3 can offer competitive performance for sequential I/O, but PCIe will give better random I/O performance and lower CPU usage. tkaiser of the armbian project has a lot to say about this stuff. I find his posts in the cnx-software.com comments and armbian forums very helpful. https://forum.armbian.com/topic/8097-nanopi-m4-performance-and-consumption-review/?do=findComment&comment=61783 https://forum.armbian.com/topic/8097-nanopi-m4-performance-a... https://github.com/ThomasKaiser/Knowledge https://github.com/ThomasKaiser/Knowledge
djsumdog 7y ago
Are you using ZFS by any chance? I was considering a Helios, by the ram requirements of ZFS turned me away from their offering.
- barrkel 7y ago
  FWIW, I run 10 x 4TB raidz2 NAS with 4GB using ZFS on Linux. ZFS on Linux isn't as memory-efficient as it would be on FreeBSD, which has a much closer file system cache / memory manager architecture to Solaris, but it works fine. ZFS memory consumption really rockets when you want to use dedupe, which basically wants to store a hash of every block in memory. The sweet spot for its use is in things like multiple VM images - where there's a lot of duplication inside large files that are otherwise different. But there's often ways to structure things to gain back the same space, e.g. with stackable file systems. Without dedupe, and for a NAS scenario where there isn't going to be a massive working set (typically source or sink for backups, streaming video, etc.), 4G has been more than sufficient for me for years.
- pnutjam 7y ago
  Have you considered Btrfs? I'm using it on my primary OS drive and my backup data drive. I keep my primary data on xfs to give me some cross platform resiliency. Btrfs allows me to take snapshots and uses space very efficiently. I haven't tried the deduplication features.

Naac 7y ago

I asked this on another thread, but I don't understand why you would see real world difference between PCIe vs USB3. When using this setup as a NAS, isn't your bottleneck the gigabit network from machine to machine? You'll saturate the 1 Gigabit link ( or even 2 Gigabit duplex ) before you can reach 5 Gigabits ( USB3 ) or 6 Gigabits ( PCIe )

walterbell 7y ago
On a low-power CPU, overhead for USB protocol handling may be less efficient than PCIe DMA?
mrb 7y ago
PCie 3 x1 is 8 gigabit/s. But yes you are right, the gigabit network should be the bottleneck either way.
- rbanffy 7y ago
  For transfers, yes, but for housekeeping activities such as scrubbing, defragmenting, indexing, and deduplicating, not at all.
pnutjam 7y ago
RAID seems very popular for 2 disk setups, but I don't like the cost benefit. With RAID under about 5 disks, you aren't getting much speedup, under 4 disks you can't do a proper fail-out and rebuild of a larger multi-disk volume. I use 2 x 4TB disks in my homeserver, but I keep one online and have a script that brings up the other one periodically and rsyncs everything. This gives me a local backup, something RAID lacks, so I'm protected from fat fingering or accidentally deleting stuff. I also have very minimal downtime, because I can mount that drive in the place of the primary drive in just afew seconds. I run xfs on the primary drive, and btrfs on the mirror, so I can take snapshots after I rsync and maintain differentials easily. My point is, you should consider getting rid of RAID and just use the bare drive or LVS Volume.
- effie 7y ago
  Agreed, RAID makes little sense if you want a quiet home storage that is used only occasionaly. Additional disk is better used for regular backups. If you need to minimize downtime and have the storage accessible 24/7, then RAID makes sense.
foresto 7y ago
IIRC, SATA over USB3 can offer competitive performance for sequential I/O, but PCIe will give better random I/O performance and lower CPU usage. tkaiser of the armbian project has a lot to say about this stuff. I find his posts in the cnx-software.com comments and armbian forums very helpful. https://forum.armbian.com/topic/8097-nanopi-m4-performance-and-consumption-review/?do=findComment&comment=61783 https://forum.armbian.com/topic/8097-nanopi-m4-performance-a... https://github.com/ThomasKaiser/Knowledge https://github.com/ThomasKaiser/Knowledge

djsumdog 7y ago

Are you using ZFS by any chance? I was considering a Helios, by the ram requirements of ZFS turned me away from their offering.

barrkel 7y ago
FWIW, I run 10 x 4TB raidz2 NAS with 4GB using ZFS on Linux. ZFS on Linux isn't as memory-efficient as it would be on FreeBSD, which has a much closer file system cache / memory manager architecture to Solaris, but it works fine. ZFS memory consumption really rockets when you want to use dedupe, which basically wants to store a hash of every block in memory. The sweet spot for its use is in things like multiple VM images - where there's a lot of duplication inside large files that are otherwise different. But there's often ways to structure things to gain back the same space, e.g. with stackable file systems. Without dedupe, and for a NAS scenario where there isn't going to be a massive working set (typically source or sink for backups, streaming video, etc.), 4G has been more than sufficient for me for years.
pnutjam 7y ago
Have you considered Btrfs? I'm using it on my primary OS drive and my backup data drive. I keep my primary data on xfs to give me some cross platform resiliency. Btrfs allows me to take snapshots and uses space very efficiently. I haven't tried the deduplication features.