警告:这个
git clean
行为在 Git 2.14 (Q3 2017) 中会略微改变。
"
git clean -d
" 以前会清除包含被忽略文件的目录,即使没有 "
-x
" 参数也不会丢失被忽略的文件。
"
git status --ignored
" 没有 "
-uall
" 参数时不会列出被忽略和未跟踪的文件。
请参见提交 6b1db43(2017年5月23日),以及提交 bbf504a, 提交 fb89888, 提交 df5bcdf, 提交 0a81d4a, 提交 b3487cc(2017年5月18日),作者为Samuel Lijin (sxlijin
)。
(由Junio C Hamano -- gitster
--在提交 f4fd99b中合并,2017年6月2日)
clean
: 教clean -d
保留被忽略的路径
有一个假设,即仅包含未跟踪和被忽略路径的目录本身应被视为未跟踪。这在我们询问是否应将目录添加到git数据库的用例中是有意义的,但不适用于我们询问是否可以安全地从工作树中删除目录的情况;因此,clean -d
会认为包含被忽略路径的“未跟踪”目录可以被删除,即使这样做也会删除被忽略的路径。
为了解决这个问题,我们教clean -d
收集被忽略的路径,并跳过包含被忽略路径的未跟踪目录,而只删除其中未跟踪的内容。
为实现这一点,cmd_clean()
必须收集所有未跟踪目录的未跟踪内容,以及所有被忽略的路径,以确定哪些未跟踪目录必须被跳过(因为它们包含被忽略的路径),哪些未跟踪目录不应该被跳过。
但是...自2017年以来,这个变化意味着git status --ignored
会无限期挂起!
正如Martin Melka在这个帖子中所报道的,并由SZEDER Gábor分析:
如果有120个目录,完成需要超过6*10^23年的深度。
这种减速是由提交df5bcdf引起的,它是修复'git clean -d
'删除未跟踪目录的一系列补丁之一,即使它们包含被忽略的文件。
所以...正在进行修复,将在2020年晚些时候发布。
Git 2.24 (2019年第四季度)展示了git clean
行为的变化,引入了一个回归问题。
请参见提交502c386(2019年8月25日)由SZEDER Gábor (szeder
)撰写。
(由Junio C Hamano -- gitster
--于提交026428c中合并,2019年9月30日)
t7300-clean
:演示删除嵌套仓库时忽略文件损坏
'git clean -fd
'不能删除属于另一个Git仓库或工作树的未跟踪目录。
不幸的是,如果外部仓库中的'.gitignore
'规则恰好匹配嵌套仓库或工作树中的文件,则会出现一些问题,'git clean -fd
'会删除嵌套仓库工作树中除被忽略的文件之外的所有内容,可能导致数据丢失。
添加一个测试到't7300-clean.sh
'以演示此故障。
这个问题是在6b1db43引入的(clean
:教clean -d
保留被忽略的路径,2017-05-23,Git v2.13.2)。
Git 2.24 进一步澄清了 git clean -d
命令:
请参见提交 69f272b(2019年10月1日),以及提交 902b90c,提交 ca8b539,提交 09487f2,提交 e86bbcf,提交 3aca580,提交 29b577b,提交 89a1f4a,提交 a3d89d8,提交 404ebce,提交 a5e916c,提交 bbbb6b0,提交 7541cc5(2019年9月17日),作者为Elijah Newren (newren
)。
(由Junio C Hamano -- gitster
--于提交 aafb754合并,2019年10月11日)
t7300
: add testcases showing failure to clean specified pathspecs
Someone brought me a testcase where multiple git-clean
invocations were
required to clean out unwanted files:
mkdir d{1,2}
touch d{1,2}/ut
touch d1/t && git add d1/t
With this setup, the user would need to run
git clean -ffd */ut
twice to delete both ut
files.
A little testing showed some interesting variants:
- If only one of those two ut files existed (either one), then only one clean command would be necessary.
- If both directories had tracked files, then only one git clean would be necessary to clean both files.
- If both directories had no tracked files then the clean command above would never clean either of the untracked files despite the pathspec explicitly calling both of them out.
A bisect showed that the failure to clean out the files started with commit cf424f5 ("clean
: respect pathspecs with "-d
", 2014-03-10, Git v1.9.1).
However, that pointed to a separate issue: while the "-d
" flag was used by the original user who showed me this problem, that flag should have been irrelevant to this problem.
Testing again without the "-d
" flag showed that the same buggy behavior exists without using that flag, and has in fact existed since before cf424f5.
所以:
clean
: 通过 "-d
" 尊重路径规范
git-clean
使用 read_directory
来填充一个 struct dir
,其中可能包含匹配项。但是,read_directory
实际上并不检查我们的路径规范。它使用了一个简化版本,可能会出现误报。因此,我们需要检查任何匹配项是否符合我们的路径规范。
对于非目录,我们可以可靠地进行检查。
对于目录,如果没有给出 "-d",我们将检查路径规范是否完全匹配(即,我们更加严格,要求显式 "git clean foo
" 来清除 "foo/
")。但是,如果给出了 "-d
",而不是放宽精确匹配以允许递归匹配,我们根本不检查路径规范。
这个回归是在113f10f中引入的(使 git-clean 成为内置命令,2007-11-11,Git v1.5.4-rc0)。
dir
: 如果我们的路径规范可能匹配到目录下的文件,则递归进入该目录
对于 git clean
,如果一个目录完全未跟踪且用户没有指定 -d
(对应于 DIR_SHOW_IGNORED_TOO
),那么我们通常不希望删除该目录,因此不会递归进入该目录。
但是,如果用户在该目录下的某个地方手动指定了特定的(甚至是通配符)路径以便删除,则我们需要递归进入该目录,以确保按照用户请求删除该目录下的相关路径。
请注意,这并不意味着递归进入的目录将被添加到 dir->entries
中以供稍后删除;作为本系列中几个提交之前的另一个更严格的匹配检查,在从递归进入的目录返回后运行,然后再决定是否将其添加到条目列表中。
因此,这只会导致与路径规范匹配的给定目录下的文件被添加到条目列表中。
并且:
dir
: also check directories for matching pathspecs
Even if a directory doesn't match a pathspec, it is possible, depending on the precise pathspecs, that some file underneath it might.
So we special case and recurse into the directory for such situations.
However, we previously always added any untracked directory that we recursed into to the list of untracked paths, regardless of whether the directory itself matched the pathspec.
For the case of git-clean
and a set of pathspecs of "dir/file
" and "more
",
this caused a problem because we'd end up with dir entries for both of:
"dir"
"dir/file"
Then correct_untracked_entries()
would try to helpfully prune duplicates for us by removing "dir/file
" since it's under "dir
", leaving us with
"dir"
Since the original pathspec only had "dir/file
", the only entry left doesn't match and leaves nothing to be removed.
(Note that if only one pathspec was specified, e.g. only "dir/file
", then the common_prefix_len
optimizations in fill_directory
would cause us to bypass this problem, making it appear in simple tests that we could correctly remove manually specified pathspecs.)
Fix this by actually checking whether the directory we are about to add to the list of dir entries actually matches the pathspec; only do this matching check after we have already returned from recursing into the directory.
那将导致:
clean
: disambiguate the definition of -d
The -d
flag pre-dated git-clean
's ability to have paths specified.
As such, the default for git-clean
was to only remove untracked files in
the current directory, and -d
existed to allow it to recurse into
subdirectories.
The interaction of paths and the -d
option appears to not have been carefully considered, as evidenced by numerous bugs and a dearth of tests covering such pairings in the testsuite.
The definition turns out to be important, so let's look at some of the various ways one could interpret the -d
option:
A) Without -d
, only look in subdirectories which contain tracked files under them; with -d
, also look in subdirectories which are untracked for files to clean.
B) Without specified paths from the user for us to delete, we need to have some kind of default, so...without -d
, only look in subdirectories which contain tracked files under them; with -d
, also look in subdirectories which are untracked for files to clean.
The important distinction here is that choice B says that the presence or absence of '-d
' is irrelevant if paths are specified.
The logic behind option B is that if a user explicitly asked us to clean a
specified pathspec, then we should clean anything that matches that
pathspec.
Some examples may clarify.
Should:
git clean -f untracked_dir/file
remove untracked_dir/file or not?
It seems crazy not to, but a strict reading of option A says it shouldn't be removed.
How about:
git clean -f untracked_dir/file1 tracked_dir/file2
or
git clean -f untracked_dir_1/file1 untracked_dir_2/file2
?
Should it remove either or both of these files?
Should it require multiple runs to remove both the files listed? (If this sounds like a crazy question to even ask, see the commit message of "t7300: Add some
testcases showing failure to clean specified pathspecs" added earlier in
this patch series.)
What if -ffd
were used instead of -f
-- should that allow these to be removed? Should it take multiple invocations with -ffd
?
What if a glob (such as 'tracked') were used instead of spelling out the directory names?
What if the filenames involved globs, such as
git clean -f '*.o'
or
git clean -f '*/*.o'
?
The current documentation actually suggests a definition that is
slightly different than choice A
, and the implementation prior to this
series provided something radically different than either choices A or B.
(The implementation, though, was clearly just buggy).
There may be other choices as well.
However, for almost any given choice of definition for -d
that I can think of, some of the examples above will appear buggy to the user.
The only case that doesn't have negative surprises is choice B: treat a user-specified path as a request to clean all untracked files which match that path specification, including recursing into any untracked directories.
Change the documentation and basic implementation to use this definition.
There were two regression tests that indirectly depended on the current
implementation, but neither was about subdirectory handling.
These two tests were introduced in commit 5b7570c ("git-clean
: add tests for relative path", 2008-03-07, Git v1.5.5-rc0) which was solely created to add coverage for the changes in commit fb328947c8e ("git-clean: correct printing relative path", 2008-03-07).
Both tests specified a directory that happened to have an untracked subdirectory, but both were only checking that the resulting printout of a file that was removed was shown with a relative path.
Update these tests appropriately.
最后,查看 "
Git clean排除嵌套子目录"。
警告:目录遍历代码存在冗余的递归调用,使其性能特征随着树的深度呈指数增长,这已在 Git 2.27(2020年第二季度)中得到修正。
而且这也影响到了 `git clean`。
请查看提交记录c0af173, 提交记录95c11ec, 提交记录7f45ab2, 提交记录1684644, 提交记录8d92fb2, 提交记录2df179d, 提交记录0126d14, 提交记录cd129ee, 提交记录446f46d, 提交记录7260c7b, 提交记录ce5c61a (2020年4月1日),作者为Elijah Newren (newren
)。
请查看提交记录0bbd0e8 (2020年4月1日),作者为Derrick Stolee (derrickstolee
)。
(由Junio C Hamano -- gitster
--在提交记录6eacc39中合并,2020年4月29日)
dir
: replace exponential algorithm with a linear one
Signed-off-by: Elijah Newren
dir's read_directory_recursive()
naturally operates recursively in order to walk the directory tree.
Treating of directories is sometimes weird because there are so many different permutations about how to handle directories.
Some examples:
- '
git ls-files -o --directory
' only needs to know that a directory itself is untracked; it doesn't need to recurse into it to see what is underneath.
- '
git status
' needs to recurse into an untracked directory, but only to determine whether or not it is empty.
If there are no files underneath, the directory itself will be omitted from the output.
If it is not empty, only the directory will be listed.
- '
git status --ignored
' needs to recurse into untracked directories and report all the ignored entries and then report the directory as untracked -- UNLESS all the entries under the directory are ignored, in which case we don't print any of the entries under the directory and just report the directory itself as ignored.
(Note that although this forces us to walk all untracked files underneath the directory as well, we strip them from the output, except for users like 'git clean' who also set DIR_KEEP_TRACKED_CONTENTS
.)
- For '
git clean
', we may need to recurse into a directory that doesn't match any specified pathspecs, if it's possible that there is an entry underneath the directory that can match one of the pathspecs.
In such a case, we need to be careful to omit the directory itself from the list of paths (see commit 404ebceda01c ("dir
: also check directories for matching pathspecs", 2019-09-17, Git v2.24.0-rc0))
Part of the tension noted above is that the treatment of a directory can change based on the files within it, and based on the various settings in dir->flags
.
Trying to keep this in mind while reading over the code, it is easy to think in terms of "treat_directory()
tells us what to do with a directory, and read_directory_recursive()
is the thing that recurses".
Since we need to look into a directory to know how to treat it, though, it is quite easy to decide to (also) recurse into the directory from treat_directory()
by adding a read_directory_recursive()
call.
Adding such a call is actually fine, IF we make sure that read_directory_recursive()
does not also recurse into that same directory.
Unfortunately, commit df5bcdf83aeb ("dir
: recurse into untracked dirs for ignored files", 2017-05-18, Git v2.14.0-rc0 -- merge listed in batch #5), added exactly such a case to the code, meaning we'd have two calls to read_directory_recursive()
for an untracked directory.
So, if we had a file named
one/two/three/four/five/somefile.txt
and nothing in one/
was tracked, then 'git status --ignored
' would call read_directory_recursive()
twice on the directory 'one/
', and each of those would call read_directory_recursive()
twice on the directory 'one/two/
', and so on until read_directory_recursive()
was called 2^5
times for 'one/two/three/four/five/
'.
Avoid calling read_directory_recursive()
twice per level by moving a lot of the special logic into treat_directory()
.
Since dir.c
is somewhat complex, extra cruft built up around this over time.
While trying to unravel it, I noticed several instances where the first call to read_directory_recursive()
would return e.g.
path_untracked
for some directory and a later one would return e.g.
path_none,
despite the fact that the directory clearly should have been considered untracked.
The code happened to work due to the side-effect from the first invocation of adding untracked entries to dir->entries;
this allowed it to get the correct output despite the supposed override in return value by the later call.
I am somewhat concerned that there are still bugs and maybe even testcases with the wrong expectation.
I have tried to carefully document treat_directory()
since it becomes more complex after this change (though much of this complexity came from elsewhere that probably deserved better comments to begin with).
However, much of my work felt more like a game of whackamole while attempting to make the code match the existing regression tests than an attempt to create an implementation that matched some clear design.
That seems wrong to me, but the rules of existing behavior had so many special cases that I had a hard time coming up with some overarching rules about what correct behavior is for all cases, forcing me to hope that the regression tests are correct and sufficient.
Such a hope seems likely to be ill-founded, given my experience with dir.c
-related testcases in the last few months:
Examples where the documentation was hard to parse or even just wrong:
- 3aca58045f4f (
git-clean.txt
: do not claim we will delete files with -n
/--dry-run
, 2019-09-17, Git v2.24.0-rc0)
- 09487f2cbad3 (
clean
: avoid removing untracked files in a nested git
repository, 2019-09-17, v2.24.0-rc0)
- e86bbcf987fa (
clean
: disambiguate the definition of -d
, 2019-09-17)
Examples where testcases were declared wrong and changed:
- 09487f2cbad3 (
clean
: avoid removing untracked files in a nested git repository, 2019-09-17, Git v2.24.0-rc0)
- e86bbcf987fa (
clean
: disambiguate the definition of -d
, 2019-09-17, Git v2.24.0-rc0)
- a2b13367fe55 (Revert "
dir.c
: make 'git-status --ignored
' work within leading directories", 2019-12-10, Git v2.25.0-rc0)
Examples where testcases were clearly inadequate:
- 502c386ff944 (
t7300-clean
: demonstrate deleting nested repo with an ignored file breakage, 2019-08-25, Git v2.24.0-rc0)
- 7541cc530239 (
t7300
: add testcases showing failure to clean specified pathspecs, 2019-09-17, Git v2.24.0-rc0)
- a5e916c7453b (
dir
: fix off-by-one error in match_pathspec_item
, 2019-09-17, Git v2.24.0-rc0)
- 404ebceda01c (
dir
: also check directories for matching pathspecs, 2019-09-17, Git v2.24.0-rc0)
- 09487f2cbad3 (
clean
: avoid removing untracked files in a nested git repository, 2019-09-17, Git v2.24.0-rc0)
- e86bbcf987fa (
clean
: disambiguate the definition of -d
, 2019-09-17, Git v2.24.0-rc0)
- 452efd11fbf6 (
t3011
: demonstrate directory traversal failures, 2019-12-10, Git v2.25.0-rc0)
- b9670c1f5e6b (
dir
: fix checks on common prefix directory, 2019-12-19, Git v2.25.0-rc0)
Examples where "correct behavior" was unclear to everyone:
其他值得注意的提交:
- 902b90cf42bc (
clean
: 修复理论上可能出现的路径损坏,2019-09-17, Git v2.24.0-rc0)
However, on the positive side, it does make the code much faster.
For the following simple shell loop in an empty repository:
for depth in $(seq 10 25)
do
dirs=$(for i in $(seq 1 $depth) ; do printf 'dir/' ; done)
rm -rf dir
mkdir -p $dirs
$dirs/untracked-file
/usr/bin/time --format="$depth: %e" git status --ignored >/dev/null
done
I saw the following timings, in seconds (note that the numbers are a little noisy from run-to-run, but the trend is very clear with every run):
10: 0.03
11: 0.05
12: 0.08
13: 0.19
14: 0.29
15: 0.50
16: 1.05
17: 2.11
18: 4.11
19: 8.60
20: 17.55
21: 33.87
22: 68.71
23: 140.05
24: 274.45
25: 551.15
For the above run, using strace
I can look for the number of untracked directories opened and can verify that it matches the expected 2^($depth+1)-2
(the sum of 2^1 + 2^2 + 2^3 + ... + 2^$depth
).
After this fix, with strace
I can verify that the number of untracked directories that are opened drops to just $depth, and the timings all drop to 0.00.
In fact, it isn't until a depth of 190 nested directories that it sometimes starts reporting a time of 0.01 seconds and doesn't consistently report 0.01 seconds until there are 240 nested directories. The previous code would have taken
17.55 * 2^220 / (60*60*24*365) = 9.4 * 10^59 YEARS
to have completed the 240 nested directories case.
It's not often that you get to speed something up by a factor of 3*10^69.
sar
目录本身中放置一个.gitignore
来忽略*.sar
文件。这不仅使顶级.gitignore
更加清晰,保持了所需的忽略信息,而且还有一个额外的好处,就是像@mvp提到的那样保持该目录的活跃性。 - Shahbaz