Deduplication for HDD

[Awesomefreeman]

Does it make sense to use deduplication for HDDs? In the manual, it's marked "dangerous". Synology also does not allow it to be used for hard drives. What are the dangers of using this script for HDDs?

[007revad]

Without knowing why Synology decided not allow it to be used for hard drives I decided to mark running deduplication on HDDs as dangerous. e.g. At your own risk, and don't blame me if something goes wrong.

Maybe Synology have not tested it or did test it and decided not to implement it for some reason. The reason could be they thought it takes too long.

I've tested it on a HDD and it worked as it should. People do use this script to deduplicate HDDs and none have reported anything bad happening.

[ThomasGoering]

One more reason for Synology not allowing it for HDDs could be:

Because data deduplication removes duplicate data blocks, it can make some data less contiguous and affect the read-write performance.

Accessing fragmented data affects the performance of HDDs more than SSDs. And this is something I can live with for my HDD volumes that are used for archiving and backups.

[MaximumFish]

Just looking to enable dedupe myself and came across this (awesome) script, and wondered the same thing. Performance limitations are a concern in busy environments, but I'd imagine it becomes far less of an issue with an SSD based read/write cache attached to the volume. Even less so in typical home environments.

That said, the amount of random access reads and writes that happens in order to do the initial dedupe could be cause for concern if the health of the drive(s) are in any doubt. Drives die during RAID rebuilds and I'd assume it's a similar amount of stress during a large dedupe job.

[007revad]

RAID rebuilds create a lot of reads and writes, and file hashing.

Deduplication just does a lot reads, but very little writes and no file hashing. For any file blocks that are the same between 2 files it just changes the duplicate block's reflink to point to the identical block, then marks the duplicate block as unused.

[MaximumFish]

Fair enough, I stand corrected. Though from experience (albeit not on a Syno NAS), the amount of time it takes to do the initial run still seems like it would be somewhat stressful for the drives.

Either way, I'm still going to use it myself! The amount of space saved can be quite incredible.

[cb12tre]

I would like to try enabling deduplication on two HDDs in SHR mounted on a 920+. If I experience slowdowns and later decide to disable deduplication using the command 'syno_enable_dedupe.sh --restore,' will the already deduplicated data remain accessible and usable as normal?

Is it possible to reverse the deduplication process? I’m not sure if there's a specific term for this, but I essentially mean duplicating the deduplicated data to return it to its original state

[007revad]

You can disable deduplication by unticking it in the volume's options.

The script's --restore option only restores the changed DSM settings, so all the already deduplicated data will remain accessible and usable as normal.

It is not possible to reverse the deduplication process.

[nowherenearaninja]

Depending on how much data it is and your other hardware, you could copy the data off of your NAS and then copy it back again to reverse deduplication

[Black6spdZ]

ran the analyzer against 6.7TB of "office" data on a test box.. estimated 1.4TB in savings.. pretty substantial and it's 99% cold data anyway. I really would like to know if Syno disabled HDD volume dedupe only because of performance concerns.. which would be mitigated by a volume defrag afterwards correct? Has anyone actually had a hard disk volume crash or lost data?

[MaximumFish]

I imagine that's all it is. You can't really defrag deduped data, because where do you put it to be contiguous with the rest of the data when multiple files call on that same block? By only allowing it on SSDs by design they're saving themselves the headache of having to deal with performance complaints, especially since there's no way to "inflate" the data once it's been deduped other than to copy it wholesale to another volume and back again.

[Black6spdZ]

makes sense.. I was thinking the defrag would at least consolidate the data to the beginning of the volume space? does btrfs defrag even do this?

[ChengYen-Tang]

In fact, regarding this problem, I turned on Synology technical support a long time ago. The following is the reply from Synology service.

The requirement to use an all-SSD setup for the deduplication feature is due to performance considerations. After further discussion with the engineers, it was confirmed that deleting duplicate blocks can cause files to become more fragmented, resulting in decreased access performance in an all-HDD storage setup. Therefore, an all-SSD storage setup is necessary.

ABB deduplication feature primarily takes place at the backup destination and does not alter the source data, meaning the source data will not be affected. Regarding the performance at the backup destination, it largely depends on the compression rate of the files and the arrangement of the actual files. In some cases, the perceived performance may not be impacted, as the fragmented parts caused by compression might not be encountered during file retrieval, etc.

Of course, we will forward your product suggestions to see if future improvements can address the performance limitations, expand the feature to more models, and reduce restrictions. However, at present, the DS1522+ does not support the deduplication feature, which is currently only available on select models.

[007revad]

I've updated the script's help and the readme to say:

      --hdd             Enable data deduplication for HDDs.
                          Can cause files to become more fragmented,
                          resulting in decreased access performance.

[Black6spdZ]

a decent size SSD cache should mitigate the fragmented performance degradation

[ChengYen-Tang]

The lifespan of an SSD will decrease rapidly