logo标签列表

使用Commons Compress压缩目录为tar.gz文件

javacompressiontarapache-commonsapache-commons-compress
17

我在使用commons compress库创建目录的tar.gz文件时遇到了问题。我的目录结构如下。

parent/
    child/
        file1.raw
        fileN.raw

我正在使用以下代码进行压缩。它可以正常运行且没有异常。然而，当我尝试解压缩tar.gz时，我得到一个名为“childDirToCompress”的单个文件。它的大小是正确的，因此文件显然已经被附加到了打包过程中。期望的输出是一个目录。我无法弄清楚我做错了什么。能否请有智慧的压缩器指引我走向正确的道路？

CreateTarGZ() throws CompressorException, FileNotFoundException, ArchiveException, IOException {
            File f = new File("parent");
            File f2 = new File("parent/childDirToCompress");

            File outFile = new File(f2.getAbsolutePath() + ".tar.gz");
            if(!outFile.exists()){
                outFile.createNewFile();
            }
            FileOutputStream fos = new FileOutputStream(outFile);

            TarArchiveOutputStream taos = new TarArchiveOutputStream(new GZIPOutputStream(new BufferedOutputStream(fos)));
            taos.setBigNumberMode(TarArchiveOutputStream.BIGNUMBER_STAR); 
            taos.setLongFileMode(TarArchiveOutputStream.LONGFILE_GNU);
            addFilesToCompression(taos, f2, ".");
            taos.close();

        }

        private static void addFilesToCompression(TarArchiveOutputStream taos, File file, String dir) throws IOException{
            taos.putArchiveEntry(new TarArchiveEntry(file, dir));

            if (file.isFile()) {
                BufferedInputStream bis = new BufferedInputStream(new FileInputStream(file));
                IOUtils.copy(bis, taos);
                taos.closeArchiveEntry();
                bis.close();
            }

            else if(file.isDirectory()) {
                taos.closeArchiveEntry();
                for (File childFile : file.listFiles()) {
                    addFilesToCompression(taos, childFile, file.getName());

                }
            }
        }
我开发了这个小类TarGzFile，可以将一个或多个文件和目录递归压缩成一个*.tar.gz文件。请参见链接。 - spongebob
6个回答
14

我遵循了这个解决方案，并且在处理较小的文件集时它能够正常工作，但是在处理15000到16000个文件后会随机崩溃。下面这行代码泄露了文件处理器：

IOUtils.copy(new FileInputStream(f), tOut);

代码在操作系统层面上崩溃，并显示“打开文件过多”的错误。进行以下微小更改即可解决问题：

FileInputStream in = new FileInputStream(f);
IOUtils.copy(in, tOut);
in.close();
1
如果有人对使用此修复程序并完美运行的Kotlin解决方案感兴趣，可以在此处查看：https://gist.github.com/Jire/8caa8603ef4871c79bef387f667a509f - Jire
13

我还没有弄清楚出了什么问题，但在谷歌缓存中搜寻后，我找到了一个可行的示例。对于那些等待回复的人，很抱歉！

public void CreateTarGZ()
    throws FileNotFoundException, IOException
{
    try {
        System.out.println(new File(".").getAbsolutePath());
        dirPath = "parent/childDirToCompress/";
        tarGzPath = "archive.tar.gz";
        fOut = new FileOutputStream(new File(tarGzPath));
        bOut = new BufferedOutputStream(fOut);
        gzOut = new GzipCompressorOutputStream(bOut);
        tOut = new TarArchiveOutputStream(gzOut);
        addFileToTarGz(tOut, dirPath, "");
    } finally {
        tOut.finish();
        tOut.close();
        gzOut.close();
        bOut.close();
        fOut.close();
    }
}

private void addFileToTarGz(TarArchiveOutputStream tOut, String path, String base)
    throws IOException
{
    File f = new File(path);
    System.out.println(f.exists());
    String entryName = base + f.getName();
    TarArchiveEntry tarEntry = new TarArchiveEntry(f, entryName);
    tOut.putArchiveEntry(tarEntry);

    if (f.isFile()) {
        IOUtils.copy(new FileInputStream(f), tOut);
        tOut.closeArchiveEntry();
    } else {
        tOut.closeArchiveEntry();
        File[] children = f.listFiles();
        if (children != null) {
            for (File child : children) {
                System.out.println(child.getName());
                addFileToTarGz(tOut, child.getAbsolutePath(), entryName + "/");
            }
        }
    }
}
1
作为参考，您不需要关闭所有串联的流。仅需关闭最外层的流（在本例中为“tOut”）即可。 - drigoangelo
2
这段代码给我一个错误提示：“此存档包含未关闭的条目”。你有什么想法是什么原因呢？ - krackoder
@awfulHack 我也遇到了“此存档包含未关闭的条目”的问题。虽然不是在包含另一个目录的目录上出现的，而是在仅包含文件而没有子目录的目录上出现的。 - conteh
如果在运行此代码时出现未关闭条目错误，请确保检查您的导入。我错过了一个（IOUtils），finally块遮盖了这个问题。 - Ethan Shepherd
我遇到了 java.io.FileNotFoundException: XXX/XXX/XXX (Too many open files) 的错误。 - Logic
嗨，使用上面的解决方案/代码，我该如何将archive.tar.gz下载到我的本地机器？谢谢。 - user1971376
7

我最终做了以下事情：

public URL createTarGzip() throws IOException {
    Path inputDirectoryPath = ...
    File outputFile = new File("/path/to/filename.tar.gz");

    try (FileOutputStream fileOutputStream = new FileOutputStream(outputFile);
            BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(fileOutputStream);
            GzipCompressorOutputStream gzipOutputStream = new GzipCompressorOutputStream(bufferedOutputStream);
            TarArchiveOutputStream tarArchiveOutputStream = new TarArchiveOutputStream(gzipOutputStream)) {

        tarArchiveOutputStream.setBigNumberMode(TarArchiveOutputStream.BIGNUMBER_POSIX);
        tarArchiveOutputStream.setLongFileMode(TarArchiveOutputStream.LONGFILE_GNU);

        List<File> files = new ArrayList<>(FileUtils.listFiles(
                inputDirectoryPath,
                new RegexFileFilter("^(.*?)"),
                DirectoryFileFilter.DIRECTORY
        ));

        for (int i = 0; i < files.size(); i++) {
            File currentFile = files.get(i);

            String relativeFilePath = new File(inputDirectoryPath.toUri()).toURI().relativize(
                    new File(currentFile.getAbsolutePath()).toURI()).getPath();

            TarArchiveEntry tarEntry = new TarArchiveEntry(currentFile, relativeFilePath);
            tarEntry.setSize(currentFile.length());

            tarArchiveOutputStream.putArchiveEntry(tarEntry);
            tarArchiveOutputStream.write(IOUtils.toByteArray(new FileInputStream(currentFile)));
            tarArchiveOutputStream.closeArchiveEntry();
        }
        tarArchiveOutputStream.close();
        return outputFile.toURI().toURL();
    }
}

这解决了其他解决方案中出现的一些特殊情况。
1
你能分享导入或库名称吗？我有这些库（以Maven格式），但是FileUtils.listFiles()没有找到那个擦除。谢谢。org.apache.commons:commons-collections4:4.1，org.apache.commons:commons-compress:1.12，commons-io:commons-io:2.4 - ldmtwo
并且导入： import java.io.; import java.net.URL; import java.nio.file.Path; import java.util.; import org.apache.commons.compress.archivers.tar.; import org.apache.commons.compress.compressors.gzip.; import org.apache.commons.io.IOUtils; import org.apache.commons.io.filefilter.*; - ldmtwo
我也遇到了和@ldmtwo一样的问题，只需要将“inputDirectoryPath”更改为实际文件路径（并且你可以将引用设置为集合而不是新的arraylist）。 - stuart
@awfulHack 这个方法对我有用。这应该是解决方案。当前的解决方案在某些情况下会产生错误。 - conteh
2

我使用的东西（通过 Files.walk API），你可以链式调用 gzip(tar(youFile));

public static File gzip(File fileToCompress) throws IOException {

    final File gzipFile = new File(fileToCompress.toPath().getParent().toFile(),
            fileToCompress.getName() + ".gz");

    final byte[] buffer = new byte[1024];

    try (FileInputStream in = new FileInputStream(fileToCompress);
            GZIPOutputStream out = new GZIPOutputStream(
                    new FileOutputStream(gzipFile))) {

        int len;
        while ((len = in.read(buffer)) > 0) {
            out.write(buffer, 0, len);
        }
    }

    return gzipFile;
}

public static File tar(File folderToCompress) throws IOException, ArchiveException {

    final File tarFile = Files.createTempFile(null, ".tar").toFile();

    try (TarArchiveOutputStream out = (TarArchiveOutputStream) new ArchiveStreamFactory()
            .createArchiveOutputStream(ArchiveStreamFactory.TAR,
                    new FileOutputStream(tarFile))) {

        out.setLongFileMode(TarArchiveOutputStream.LONGFILE_GNU);

        Files.walk(folderToCompress.toPath()) //
                .forEach(source -> {

                    if (source.toFile().isFile()) {
                        final String relatifSourcePath = StringUtils.substringAfter(
                                source.toString(), folderToCompress.getPath());

                        final TarArchiveEntry entry = new TarArchiveEntry(
                                source.toFile(), relatifSourcePath);

                        try (InputStream in = new FileInputStream(source.toFile())){
                            out.putArchiveEntry(entry);

                            IOUtils.copy(in, out);

                            out.closeArchiveEntry();
                        }
                        catch (IOException e) {
                            // Handle this better than bellow...
                            throw new RuntimeException(e);
                        }
                    }
                });

    }

    return tarFile;
}
2

我不得不对@merrick的解决方案进行一些调整，以使其与路径相关的工作。也许是由于最新的Maven依赖关系。目前被接受的解决方案对我无效。

import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
import org.apache.commons.compress.compressors.gzip.GzipCompressorOutputStream;
import org.apache.commons.io.FileUtils;
import org.apache.commons.io.IOUtils;
import org.apache.commons.io.filefilter.DirectoryFileFilter;
import org.apache.commons.io.filefilter.RegexFileFilter;

public class TAR {

    public static void CreateTarGZ(String inputDirectoryPath, String outputPath) throws IOException {

        File inputFile = new File(inputDirectoryPath);
        File outputFile = new File(outputPath);

        try (FileOutputStream fileOutputStream = new FileOutputStream(outputFile);
                BufferedOutputStream bufferedOutputStream = new BufferedOutputStream(fileOutputStream);
                GzipCompressorOutputStream gzipOutputStream = new GzipCompressorOutputStream(bufferedOutputStream);
                TarArchiveOutputStream tarArchiveOutputStream = new TarArchiveOutputStream(gzipOutputStream)) {

            tarArchiveOutputStream.setBigNumberMode(TarArchiveOutputStream.BIGNUMBER_POSIX);
            tarArchiveOutputStream.setLongFileMode(TarArchiveOutputStream.LONGFILE_GNU);

            List<File> files = new ArrayList<>(FileUtils.listFiles(
                    inputFile,
                    new RegexFileFilter("^(.*?)"),
                    DirectoryFileFilter.DIRECTORY
            ));

            for (int i = 0; i < files.size(); i++) {
                File currentFile = files.get(i);

                String relativeFilePath = inputFile.toURI().relativize(
                        new File(currentFile.getAbsolutePath()).toURI()).getPath();

                TarArchiveEntry tarEntry = new TarArchiveEntry(currentFile, relativeFilePath);
                tarEntry.setSize(currentFile.length());

                tarArchiveOutputStream.putArchiveEntry(tarEntry);
                tarArchiveOutputStream.write(IOUtils.toByteArray(new FileInputStream(currentFile)));
                tarArchiveOutputStream.closeArchiveEntry();
            }
            tarArchiveOutputStream.close();
        }
    }
}

Maven

        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>2.6</version>
        </dependency>

        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-compress</artifactId>
            <version>1.18</version>
        </dependency>
2
请查看以下Apache commons-compress和文件遍历器示例。
此示例将一个目录打包成tar.gz文件。
public static void createTarGzipFolder(Path source) throws IOException {

        if (!Files.isDirectory(source)) {
            throw new IOException("Please provide a directory.");
        }

        // get folder name as zip file name
        String tarFileName = source.getFileName().toString() + ".tar.gz";

        try (OutputStream fOut = Files.newOutputStream(Paths.get(tarFileName));
             BufferedOutputStream buffOut = new BufferedOutputStream(fOut);
             GzipCompressorOutputStream gzOut = new GzipCompressorOutputStream(buffOut);
             TarArchiveOutputStream tOut = new TarArchiveOutputStream(gzOut)) {

            Files.walkFileTree(source, new SimpleFileVisitor<>() {

                @Override
                public FileVisitResult visitFile(Path file,
                                            BasicFileAttributes attributes) {

                    // only copy files, no symbolic links
                    if (attributes.isSymbolicLink()) {
                        return FileVisitResult.CONTINUE;
                    }

                    // get filename
                    Path targetFile = source.relativize(file);

                    try {
                        TarArchiveEntry tarEntry = new TarArchiveEntry(
                                file.toFile(), targetFile.toString());

                        tOut.putArchiveEntry(tarEntry);

                        Files.copy(file, tOut);

                        tOut.closeArchiveEntry();

                        System.out.printf("file : %s%n", file);

                    } catch (IOException e) {
                        System.err.printf("Unable to tar.gz : %s%n%s%n", file, e);
                    }

                    return FileVisitResult.CONTINUE;
                }

                @Override
                public FileVisitResult visitFileFailed(Path file, IOException exc) {
                    System.err.printf("Unable to tar.gz : %s%n%s%n", file, exc);
                    return FileVisitResult.CONTINUE;
                }

            });

            tOut.finish();
        }

    }

这个例子提取了一个tar.gz文件，并检查了zip slip攻击。
public static void decompressTarGzipFile(Path source, Path target)
        throws IOException {

        if (Files.notExists(source)) {
            throw new IOException("File doesn't exists!");
        }

        try (InputStream fi = Files.newInputStream(source);
             BufferedInputStream bi = new BufferedInputStream(fi);
             GzipCompressorInputStream gzi = new GzipCompressorInputStream(bi);
             TarArchiveInputStream ti = new TarArchiveInputStream(gzi)) {

            ArchiveEntry entry;
            while ((entry = ti.getNextEntry()) != null) {

                Path newPath = zipSlipProtect(entry, target);

                if (entry.isDirectory()) {
                    Files.createDirectories(newPath);
                } else {

                    // check parent folder again
                    Path parent = newPath.getParent();
                    if (parent != null) {
                        if (Files.notExists(parent)) {
                            Files.createDirectories(parent);
                        }
                    }

                    // copy TarArchiveInputStream to Path newPath
                    Files.copy(ti, newPath, StandardCopyOption.REPLACE_EXISTING);

                }
            }
        }
    }

    private static Path zipSlipProtect(ArchiveEntry entry, Path targetDir)
        throws IOException {

        Path targetDirResolved = targetDir.resolve(entry.getName());

        Path normalizePath = targetDirResolved.normalize();

        if (!normalizePath.startsWith(targetDir)) {
            throw new IOException("Bad entry: " + entry.getName());
        }

        return normalizePath;
    }

References

  1. https://mkyong.com/java/how-to-create-tar-gz-in-java/
  2. https://commons.apache.org/proper/commons-compress/examples.html
网页内容由stack overflow 提供, 点击上面的
可以查看英文原文，
原文链接