如何使用ScatterZipOutputStream实现支持Zip64的并行压缩？

Question

我想知道是否有人可以帮忙使用ScatterZipOutputStream实现并行压缩。我已经搜索了很多地方，但没有找到相应的示例。

https://commons.apache.org/proper/commons-compress/zip.html

我已经尝试使用ZipArchiveOutputStream制作Zip文件、压缩目录等。现在，我正在尝试并行处理。

public static void makeZip(String filename) throws IOException,
        ArchiveException {
    File sourceFile = new File(filename);

    final OutputStream out = new FileOutputStream(filename.substring(0, filename.lastIndexOf('.')) + ".zip");
    ZipArchiveOutputStream os = new ZipArchiveOutputStream(out);
    os.setUseZip64(Zip64Mode.AsNeeded);

    os.putArchiveEntry(new ZipArchiveEntry(sourceFile.getName()));
    IOUtils.copy(new FileInputStream(sourceFile), os);
    os.closeArchiveEntry();
    os.close();
}

它应该能够将单个文件作为线程处理，然后将其合并以编写结果压缩包。

- Sameer Pradhan

---

希望这可以帮到你：https://dev59.com/Np3ha4cB1Zd3GeqPZcRJ - dkb

Answer 1

以下是zip和unzip的工作代码:
1. 更改sourceFolder和zipFilePath的路径
2. 仅压缩*.text类型的文件，它可以是任何类型或所有文件
3. 解压缩文件在sourceFolder/unzip/中。

在build.gradle或pom.xml中导入以下依赖项。

implementation("org.apache.commons:commons-compress:1.18")
implementation("commons-io:commons-io:2.6")

Ref: https://mvnrepository.com/artifact/org.apache.commons/commons-compress/1.18 https://mvnrepository.com/artifact/commons-io/commons-io/2.6 //code

import org.apache.commons.compress.archivers.zip.*;
import org.apache.commons.compress.parallel.InputStreamSupplier;
import org.apache.commons.io.FileUtils;

import java.io.*;
import java.nio.file.Files;
import java.util.Iterator;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;

public class ZipMain {

static ParallelScatterZipCreator scatterZipCreator = new ParallelScatterZipCreator();
static ScatterZipOutputStream dirs;

static {
    try {
        dirs = ScatterZipOutputStream.fileBased(File.createTempFile("java-zip-dirs", "tmp"));
    } catch (IOException e) {
        e.printStackTrace();
    }
}

public static void main(String[] args) throws IOException {
    String sourceFolder = "/Users/<user>/Desktop/";
    String zipFilePath = "/Users/<user>/Desktop/Desk.zip";
    String fileTypesToBeAddedToZip = "txt";
    zip(sourceFolder, zipFilePath, fileTypesToBeAddedToZip);
    unzip(zipFilePath, sourceFolder + "/unzip/");
}

private static void zip(String sourceFolder, String zipFilePath, String fileTypesToBeAddedToZip) throws IOException {
    OutputStream outputStream = null;
    ZipArchiveOutputStream zipArchiveOutputStream = null;
    try {

        File srcFolder = new File(sourceFolder);
        if (srcFolder.isDirectory()) {
            // uncomment following code if you want to add all files under srcFolder
            //Iterator<File> fileIterator = Arrays.asList(srcFolder.listFiles()).iterator();
            Iterator<File> fileIterator = FileUtils.iterateFiles(srcFolder, new String[]{fileTypesToBeAddedToZip}, true);

            File zipFile = new File(zipFilePath);
            zipFile.delete();
            outputStream = new FileOutputStream(zipFile);

            zipArchiveOutputStream = new ZipArchiveOutputStream(outputStream);
            zipArchiveOutputStream.setUseZip64(Zip64Mode.AsNeeded);

            int srcFolderLength = srcFolder.getAbsolutePath().length() + 1;  // +1 to remove the last file separator

            while (fileIterator.hasNext()) {
                File file = fileIterator.next();

                // uncomment following code if you want to add all files under srcFolder
                //if (file.isDirectory()) {
                //        continue;
                //    }

                String relativePath = file.getAbsolutePath().substring(srcFolderLength);


                InputStreamSupplier streamSupplier = () -> {
                    InputStream is = null;
                    try {
                        is = Files.newInputStream(file.toPath());
                    } catch (IOException e) {
                        e.printStackTrace();
                    }
                    return is;
                };
                ZipArchiveEntry zipArchiveEntry = new ZipArchiveEntry(relativePath);
                zipArchiveEntry.setMethod(ZipEntry.DEFLATED);
                scatterZipCreator.addArchiveEntry(zipArchiveEntry, streamSupplier);
            }
            scatterZipCreator.writeTo(zipArchiveOutputStream);
            }
            if (zipArchiveOutputStream != null) {
                zipArchiveOutputStream.close();
            }
    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        if (outputStream != null) {
            outputStream.close();
        }
    }
}

private static void unzip(String zipFilePath, String destDir) {
    File dir = new File(destDir);
    // create output directory if it doesn't exist
    if (!dir.exists()) {
        dir.mkdirs();
    } else {
        dir.delete();
    }

    FileInputStream fis;
    //buffer for read and write data to file
    byte[] buffer = new byte[1024];
    try {
        fis = new FileInputStream(zipFilePath);
        ZipInputStream zis = new ZipInputStream(fis);
        ZipEntry ze = zis.getNextEntry();
        while (ze != null) {
            String fileName = ze.getName();

            File newFile = new File(destDir + File.separator + fileName);

            System.out.println("Unzipping to " + newFile.getAbsolutePath());

            //create directories for sub directories in zip
            String parentFolder = newFile.getParent();
            File folder = new File(parentFolder);
            folder.mkdirs();

            FileOutputStream fos = new FileOutputStream(newFile);
            int len;
            while ((len = zis.read(buffer)) > 0) {
                fos.write(buffer, 0, len);
            }
            fos.close();
            //close this ZipEntry
            zis.closeEntry();
            ze = zis.getNextEntry();
        }
        //close last ZipEntry
        zis.closeEntry();
        zis.close();
        fis.close();
    } catch (IOException e) {
        e.printStackTrace();
    }

}
}

参考：使用Java ParallelScatterZipCreator快速压缩文件夹

- dkb

---

嗨@dkb，没有lambda表达式的代码是什么？我尝试了这个但是失败了。InputStreamSupplier streamSupplier = Files.newInputStream(file.toPath()); - Sameer Pradhan

你需要在你的构建文件或 pom.xml 中添加依赖项，不需要强制转换任何类。 - dkb

如果你在上面的代码中运行了unzip方法，那么它就会为你解压缩。你试过了吗？ - dkb

尝试使用更新后的代码，应该能正常工作。添加了 if (zipArchiveOutputStream != null) { zipArchiveOutputStream.close(); }。 - dkb

为什么不创建多个线程并在其中添加zip逻辑，将不同的zip文件夹路径作为参数传递给每个线程。这就是全部内容了。如果您有问题，请发布另一个问题，我们会帮助您解决。或者您可以先尝试自己解决问题，然后再发布问题并说明您正在面临的问题。 - dkb