肯定有一种简单的方法来做到这一点!
我已经尝试过Linux命令行应用程序,例如sha1sum
和md5sum
,但它们似乎只能计算单个文件的哈希值,并输出每个文件的一个哈希值列表。
我需要为整个文件夹的所有内容(而不仅仅是文件名)生成一个单一的哈希值。
我想做类似于以下的操作:
sha1sum /folder/of/stuff > singlehashvalue
编辑:为了澄清,我的文件分布在目录树的多个层级中,它们并不都位于同一根文件夹中。
肯定有一种简单的方法来做到这一点!
我已经尝试过Linux命令行应用程序,例如sha1sum
和md5sum
,但它们似乎只能计算单个文件的哈希值,并输出每个文件的一个哈希值列表。
我需要为整个文件夹的所有内容(而不仅仅是文件名)生成一个单一的哈希值。
我想做类似于以下的操作:
sha1sum /folder/of/stuff > singlehashvalue
编辑:为了澄清,我的文件分布在目录树的多个层级中,它们并不都位于同一根文件夹中。
hashdir /folder/of/stuff
它有几个有用的标志,允许你指定散列算法、打印所有子项的散列值,以及保存和验证散列值。hashdir:
A command-line utility to checksum directories and files.
Usage:
hashdir [options] [<item>...] [command]
Arguments:
<item> Directory or file to hash/check
Options:
-t, --tree Print directory tree
-s, --save Save the checksum to a file
-i, --include-hidden-files Include hidden files
-e, --skip-empty-dir Skip empty directories
-a, --algorithm <md5|sha1|sha256|sha384|sha512> The hash function to use [default: sha1]
--version Show version information
-?, -h, --help Show help and usage information
Commands:
check <item> Verify that the specified hash file is valid.
# 1. How to get a sha256 hash over all file contents in a folder, including
# hashing over the relative file paths within that folder to check the
# filenames themselves (get this bash function below).
sha256sum_dir "path/to/folder"
# 2. How to quickly compare two folders (get the `diff_dir` bash function below)
diff_dir "path/to/folder1" "path/to/folder2"
# OR:
diff -r -q "path/to/folder1" "path/to/folder2"
请使用以下这个命令,而不是主答案,以获取整个文件夹中所有非目录文件内容的单个哈希值,无论该文件夹位于何处:
这是一个“一行代码”的命令。复制并粘贴整个命令以一次性运行它:
# This one works, but don't use it, because its hash output does NOT
# match that of my `sha256sum_dir` function. I recommend you use
# the "1-liner" just below, therefore, instead.
time ( \
starting_dir="$(pwd)" \
&& target_dir="path/to/folder" \
&& cd "$target_dir" \
&& find . -not -type d -print0 | sort -zV \
| xargs -0 sha256sum | sha256sum; \
cd "$starting_dir"
)
sha256sum_dir
bash函数生成的哈希值略有不同。因此,如果要使输出哈希值与我的sha256sum_dir
函数的输出完全匹配,请执行以下操作:# Use this one, as its output matches that of my `sha256sum_dir`
# function exactly.
all_hashes_str="$( \
starting_dir="$(pwd)" \
&& target_dir="path/to/folder" \
&& cd "$target_dir" \
&& find . -not -type d -print0 | sort -zV | xargs -0 sha256sum \
)"; \
cd "$starting_dir"; \
printf "%s" "$all_hashes_str" | sha256sum
要了解为什么主要答案在不同位置的相同文件夹中不会产生相同的哈希,请参见下文。
sha256sum_dir
和 diff_dir
将以下函数放入您的 ~/.bashrc
文件或您的 ~/.bash_aliases
文件中,假设您的 ~/.bashrc
文件像这样引用 ~/.bash_aliases
文件:
if [ -f ~/.bash_aliases ]; then
. ~/.bash_aliases
fi
~/.bash_aliases
文件中找到下面的两个函数,它们都在我的 eRCaGuy_dotfiles 存储库中。sha256sum_dir
函数,它获取目录中所有文件的总哈希值:# Take the sha256sum of all files in an entire dir, and then sha256sum that
# entire output to obtain a _single_ sha256sum which represents the _entire_
# dir.
# See:
# 1. [my answer] https://dev59.com/kXM_5IYBdhLWcg3wn0lO#72070772
sha256sum_dir() {
return_code="$RETURN_CODE_SUCCESS"
if [ "$#" -eq 0 ]; then
echo "ERROR: too few arguments."
return_code="$RETURN_CODE_ERROR"
fi
# Print help string if requested
if [ "$#" -eq 0 ] || [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
# Help string
echo "Obtain a sha256sum of all files in a directory."
echo "Usage: ${FUNCNAME[0]} [-h|--help] <dir>"
return "$return_code"
fi
starting_dir="$(pwd)"
target_dir="$1"
cd "$target_dir"
# See my answer: https://dev59.com/kXM_5IYBdhLWcg3wn0lO#72070772
filenames="$(find . -not -type d | sort -V)"
IFS=$'\n' read -r -d '' -a filenames_array <<< "$filenames"
time all_hashes_str="$(sha256sum "${filenames_array[@]}")"
cd "$starting_dir"
echo ""
echo "Note: you may now call:"
echo "1. 'printf \"%s\n\" \"\$all_hashes_str\"' to view the individual" \
"hashes of each file in the dir. Or:"
echo "2. 'printf \"%s\" \"\$all_hashes_str\" | sha256sum' to see that" \
"the hash of that output is what we are using as the final hash" \
"for the entire dir."
echo ""
printf "%s" "$all_hashes_str" | sha256sum | awk '{ print $1 }'
return "$?"
}
# Note: I prefix this with my initials to find my custom functions easier
alias gs_sha256sum_dir="sha256sum_dir"
diff -r -q "dir1" "dir2"
命令,我将其封装在这个 diff_dir
命令中。我从这里了解到使用 diff
命令比较整个文件夹的方法:如何在Linux中检查两个文件夹是否相同。# Compare dir1 against dir2 to see if they are equal or if they differ.
# See:
# 1. How to `diff` two dirs: https://dev59.com/HHRB5IYBdhLWcg3w9L3n#16404554
diff_dir() {
return_code="$RETURN_CODE_SUCCESS"
if [ "$#" -eq 0 ]; then
echo "ERROR: too few arguments."
return_code="$RETURN_CODE_ERROR"
fi
# Print help string if requested
if [ "$#" -eq 0 ] || [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
echo "Compare (diff) two directories to see if dir1 contains the same" \
"content as dir2."
echo "NB: the output will be **empty** if both directories match!"
echo "Usage: ${FUNCNAME[0]} [-h|--help] <dir1> <dir2>"
return "$return_code"
fi
dir1="$1"
dir2="$2"
time diff -r -q "$dir1" "$dir2"
return_code="$?"
if [ "$return_code" -eq 0 ]; then
echo -e "\nDirectories match!"
fi
# echo "$return_code"
return "$return_code"
}
# Note: I prefix this with my initials to find my custom functions easier
alias gs_diff_dir="diff_dir"
~/temp2
目录上运行sha256sum_dir
命令的输出结果(我会在下面描述这个目录,以便您可以重现并测试此命令)。您可以看到,在这种情况下,整个文件夹的哈希值为b86c66bcf2b033f65451e8c225425f315e618be961351992b7c7681c3822f6a3
:$ gs_sha256sum_dir ~/temp2
real 0m0.007s
user 0m0.000s
sys 0m0.007s
Note: you may now call:
1. 'printf "%s\n" "$all_hashes_str"' to view the individual hashes of each
file in the dir. Or:
2. 'printf "%s" "$all_hashes_str" | sha256sum' to see that the hash of that
output is what we are using as the final hash for the entire dir.
b86c66bcf2b033f65451e8c225425f315e618be961351992b7c7681c3822f6a3
diff_dir
命令和输出,用于比较两个目录是否相等。这是检查刚刚将整个目录复制到我的 SD 卡是否正确工作的过程。当目录匹配时,输出会显示 Directories match!
。$ gs_diff_dir "path/to/sd/card/tempdir" "/home/gabriel/tempdir"
real 0m0.113s
user 0m0.037s
sys 0m0.077s
Directories match!
我尝试了这里最受欢迎的答案,但它并不能完全正确地工作。它需要一些微调。它之所以不能正常工作是因为哈希值基于感兴趣的文件夹的基本路径而改变!这意味着即使两个文件夹是完美匹配并且包含完全相同的内容,某个文件夹的完全复制品将具有与从中复制的文件夹不同的哈希值!这种情况有点让人失望,因为如果两个相同文件夹的哈希值不同,那么对该文件夹进行哈希处理就没有意义了!让我解释一下:
假设我有一个名为temp2
的文件夹,在~/temp2
下。它包含file1.txt
、file2.txt
和file3.txt
。 file1.txt
包含字母a
和一个回车,file2.txt
包含字母b
和一个回车,file3.txt
包含字母c
和一个回车。find /home/gabriel/temp2
,我会得到:$ find /home/gabriel/temp2
/home/gabriel/temp2
/home/gabriel/temp2/file3.txt
/home/gabriel/temp2/file1.txt
/home/gabriel/temp2/file2.txt
sha256sum
(而不是sha1sum
),我会得到以下结果。请注意,它在每个哈希后面都有完整路径,这不是我们想要的:$ find /home/gabriel/temp2 -type f -print0 | sort -z | xargs -0 sha256sum
87428fc522803d31065e7bce3cf03fe475096631e5e07bbd7a0fde60c4cf25c7 /home/gabriel/temp2/file1.txt
0263829989b6fd954f72baaf2fc64bc2e2f01d692d4de72986ea808f6e99813f /home/gabriel/temp2/file2.txt
a3a5e715f0cc574a73c3f9bebb6bc24f32ffd5b67b387244c2c909da779a1478 /home/gabriel/temp2/file3.txt
sha256sum
中,它会使用文件哈希值的完整文件路径进行哈希,这不是我们想要的!文件哈希值可能在一个文件夹和该文件夹的副本中完全匹配,但是绝对路径并不完全匹配,因此它们将产生不同的最终哈希值,因为我们正在哈希完整文件路径作为我们单个最终哈希的一部分! cd
进入您感兴趣的文件夹,然后再运行其中所有文件的哈希命令,如下所示:cd "/home/gabriel/temp2" && find . -type f -print0 | sort -z | xargs -0 sha256sum
$ cd "/home/gabriel/temp2" && find . -type f -print0 | sort -z | xargs -0 sha256sum
87428fc522803d31065e7bce3cf03fe475096631e5e07bbd7a0fde60c4cf25c7 ./file1.txt
0263829989b6fd954f72baaf2fc64bc2e2f01d692d4de72986ea808f6e99813f ./file2.txt
a3a5e715f0cc574a73c3f9bebb6bc24f32ffd5b67b387244c2c909da779a1478 ./file3.txt
下面是一个简单的Python 3变体,适用于小型文件(例如源码树或者其他每个文件容易放入内存中的地方),忽略空目录,基于其他解决方案的想法。
import os, hashlib
def hash_for_directory(path, hashfunc=hashlib.sha1):
filenames = sorted(os.path.join(dp, fn) for dp, _, fns in os.walk(path) for fn in fns)
index = '\n'.join('{}={}'.format(os.path.relpath(fn, path), hashfunc(open(fn, 'rb').read()).hexdigest()) for fn in filenames)
return hashfunc(index.encode('utf-8')).hexdigest()
md5sum * | md5sum | cut -d' ' -f1
它通过文件生成哈希列表,然后将这些哈希连接成一个。
这比tar方法更快。
为了在我们的哈希中获得更强的隐私,我们可以在相同的配方上使用sha512sum。
sha512sum * | sha512sum | cut -d' ' -f1
使用 sha512sum,哈希值也是相同的,但目前没有已知的方法可以反向解密。
sha256sum /tmp/thd-agent/* | sort
是我尝试获得可靠排序的方法,然后只需对其进行哈希处理即可。 - thinkttls -r | sha256sum
是什么意思? - NVRM我会将每个文件的结果通过 sort
(以防止仅仅重新排序文件就改变哈希值)传输到 md5sum
或 sha1sum
,具体使用哪一个取决于您的选择。
尝试分两步完成:
就像这样:
# for FILE in `find /folder/of/stuff -type f | sort`; do sha1sum $FILE >> hashes; done
# sha1sum hashes
或者一次性完成:
# cat `find /folder/of/stuff -type f | sort` | sha1sum
for F in 'find ...' ...
就无法正常工作。 - mivk
import java.security.MessageDigest
public static String generateDigest(File file, String digest, int paddedLength){
MessageDigest md = MessageDigest.getInstance(digest)
md.reset()
def files = []
def directories = []
if(file.isDirectory()){
file.eachFileRecurse(){sf ->
if(sf.isFile()){
files.add(sf)
}
else{
directories.add(file.toURI().relativize(sf.toURI()).toString())
}
}
}
else if(file.isFile()){
files.add(file)
}
files.sort({a, b -> return a.getAbsolutePath() <=> b.getAbsolutePath()})
directories.sort()
files.each(){f ->
println file.toURI().relativize(f.toURI()).toString()
f.withInputStream(){is ->
byte[] buffer = new byte[8192]
int read = 0
while((read = is.read(buffer)) > 0){
md.update(buffer, 0, read)
}
}
}
directories.each(){d ->
println d
md.update(d.getBytes())
}
byte[] digestBytes = md.digest()
BigInteger bigInt = new BigInteger(1, digestBytes)
return bigInt.toString(16).padLeft(paddedLength, '0')
}
println "\n${generateDigest(new File(args[0]), 'SHA-256', 64)}"
gary-macbook:Scripts garypaduana$ groovy dirHash.groovy /Users/garypaduana/.config
.DS_Store
configstore/bower-github.yml
configstore/insight-bower.json
configstore/update-notifier-bower.json
filezilla/filezilla.xml
filezilla/layout.xml
filezilla/lockfile
filezilla/queue.sqlite3
filezilla/recentservers.xml
filezilla/sitemanager.xml
gtk-2.0/gtkfilechooser.ini
a/
configstore/
filezilla/
gtk-2.0/
lftp/
menus/
menus/applications-merged/
79de5e583734ca40ff651a3d9a54d106b52e94f1f8c2cd7133ca3bbddc0c6758
sha1sum
生成哈希值列表,然后再次对该列表进行sha1sum
,具体取决于你想要实现什么。假设您正在尝试比较一个文件夹及其所有内容,以确保它从一台计算机正确地复制到另一台计算机,例如,您可以按照以下方式执行。 假设文件夹命名为mydir
,在计算机1上的路径为/home/gabriel/mydir
,在计算机2上的路径为/home/gabriel/dev/repos/mydir
。
# 1. First, cd to the dir in which the dir of interest is found. This is
# important! If you don't do this, then the paths output by find will differ
# between the two computers since the absolute paths to `mydir` differ. We are
# going to hash the paths too, not just the file contents, so this matters.
cd /home/gabriel # on computer 1
cd /home/gabriel/dev/repos # on computer 2
# 2. hash all files inside `mydir`, then hash the list of all hashes and their
# respective file paths. This obtains one single final hash. Sorting is
# necessary by piping to `sort` to ensure we get a consistent file order in
# order to ensure a consistent final hash result.
find mydir -type f -exec sha256sum {} + | sort | sha256sum
# Optionally pipe that output to awk to filter in on just the hash (first field
# in the output)
find mydir -type f -exec sha256sum {} + | sort | sha256sum | awk '{print $1}'
就是这样!
为了学习的缘故,要查看文件哈希值的中间列表,只需运行此命令:
find mydir -type f -exec sha256sum {} + | sort
eclipse-workspace
文件夹从一台电脑正确地复制到另一台电脑。正如您所看到的,time
命令告诉我它用了11.790秒:$ time find eclipse-workspace -type f -exec sha256sum {} + | sort | sha256sum
8f493478e7bb77f1d025cba31068c1f1c8e1eab436f8a3cf79d6e60abe2cd2e4 -
real 0m11.790s
user 0m11.372s
sys 0m0.432s
我关心的哈希值是:8f493478e7bb77f1d025cba31068c1f1c8e1eab436f8a3cf79d6e60abe2cd2e4
如果将其传输到awk
并排除time
,我会得到:
$ find eclipse-workspace -type f -exec sha256sum {} + | sort | sha256sum | awk '{print $1}'
8f493478e7bb77f1d025cba31068c1f1c8e1eab436f8a3cf79d6e60abe2cd2e4
请确保检查打印的stderr
输出中的find
错误,因为即使find
失败,也会生成哈希。
考虑到eclipse-workspace
目录包含6480个文件,仅用12秒对其进行哈希处理是令人印象深刻的,如下所示:
find eclipse-workspace -type f | wc -l
...并且文件大小为3.6 GB,如下所示:
du -sh eclipse-workspace
其他致谢:我曾与ChatGPT交流,学习了上述部分内容。然而,所有的工作和文本都是我编写、测试和验证的。