hashget

Deduplication tool for archiving data with extremely high ratio

Hashget is network deduplication tool working together with usual compression utilities (such as tar/gz/xz). While usual compression tools uses mathematical algorithms for compressing data, hashget finds which files could be downloaded from public (e.g. WordPress or Debian servers) or private (e.g. your company internal website) resources and excludes it from archive (leaving only very short meta-information about it). Upon decompressing, hashget downloads these files, verifies hashsum and places it on target system with same permissions, ownership, atime and mtime. Hashget compression is lossless, based on cryptographically strong SHA256 hashsum.

homepage ↗ github: yaroslaff/hashget pypi: hashget