Java使用输入/输出流进行LZ4压缩

Question

我正在使用jpountz LZ4来尝试压缩文件，并且我想使用Java文件输入和输出流来读取和输出文件。我试图在网上找到解决方案，但是没有找到。我在之前的stackoverflow问题中发现了如何正确实现LZ4的问题，并尝试修改它以使用流，但我不确定这是否正确或者是否有效。

当压缩文本文件时，输出的文件会缺少某些字符或替换为符号。

ðHello world Heðo world Hello ðrld Hello worlðHello worl

但是当使用图像文件运行时，它会抛出一个越界错误。我还无法使解压缩工作，因为它抛出一个“解码输入缓冲区偏移3的错误”。以下是我的代码，任何帮助将不胜感激，谢谢。

public void LZ4Compress(InputStream in, OutputStream out){
    int noBytesRead = 0;        //number of bytes read from input
    int noBytesProcessed = 0;   //number of bytes processed
    try {
        while ((noBytesRead = in.read(inputBuffer)) >= 0) {
            noBytesProcessed = inputBuffer.length;
            decompressedLength = inputBuffer.length;
            outputBuffer = compress(inputBuffer, decompressedLength);
            out.write(outputBuffer, 0, noBytesRead);
        }
        out.flush();
        in.close();
        out.close();
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
}

public void LZ4decompress(InputStream in, OutputStream out){
    int noBytesRead = 0;        //number of bytes read from input
    try {
        while((noBytesRead = in.read(inputBuffer)) >= 0){
            noBytesProcessed = inputBuffer.length;
            outputBuffer = decompress(inputBuffer);
            out.write(outputBuffer, 0, noBytesRead);

        }
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
}

public static byte[] compress(byte[] src, int srcLen) {
    decompressedLength = srcLen;
    int maxCompressedLength = compressor.maxCompressedLength(decompressedLength);
    byte[] compressed = new byte[maxCompressedLength];
    int compressLen = compressor.compress(src, 0, decompressedLength, compressed, 0, maxCompressedLength);
    byte[] finalCompressedArray = Arrays.copyOf(compressed, compressLen);
    return finalCompressedArray;
}

private static LZ4SafeDecompressor decompressor = factory.safeDecompressor();

public static byte[] decompress(byte[] finalCompressedArray) {
    byte[] restored = new byte[finalCompressedArray.length];
    restored = decompressor.decompress(finalCompressedArray, finalCompressedArray.length);
    return restored;
}

- user3758298

---

@user3758298- 我有一个问题，如果我不知道解压后的数组长度，那么我该如何解压缩字节数组？我有一个压缩的字节数组，我想要解压缩它。 - ketan

noBytesProcessed = inputBuffer.length: 这里出了问题。应该是 noBytesProcessed = noBytesRead，类似地，decompressedLength 以及其他所有你假设 read() 填充缓冲区的地方也是如此。 - user207421

Answer 1

我通过使用LZ4块输入/输出流解决了我的问题。

public static void LZ4compress(String filename, String lz4file){
    byte[] buf = new byte[2048];
    try {
        String outFilename = lz4file;
        LZ4BlockOutputStream out = new LZ4BlockOutputStream(new FileOutputStream(outFilename), 32*1024*1024);
        FileInputStream in = new FileInputStream(filename);
        int len;
        while((len = in.read(buf)) > 0){
            out.write(buf, 0, len);
        }
        in.close();
        out.close();
    } catch (IOException e) {

    }
}

public static void LZ4Uncompress(String lz4file, String filename){
    byte[] buf = new byte[2048];
    try {
        String outFilename = filename;
        LZ4BlockInputStream in = new LZ4BlockInputStream(new FileInputStream(lz4file));
        FileOutputStream out = new FileOutputStream(outFilename);
        int len;
        while((len = in.read(buf)) > 0){
            out.write(buf, 0, len);
        }
        in.close();
        out.close();
    } catch (IOException e) {

    }
}

- user3758298

Answer 2

仅从代码上看，我会说你在这里走错了：

 outputBuffer = compress(inputBuffer, decompressedLength);
 out.write(outputBuffer, 0, noBytesRead);

在压缩中，您已经修剪了输出缓冲区。尝试：

out.write(outputBuffer);

- Teemu Ilmonen

---

这解决了我的越界错误，而且似乎可以处理所有文件类型，问题是输出文件比原始文件更大。 - user3758298

很可能压缩最终会分成小块进行，这比一次性压缩整个文件效果要差。 - Teemu Ilmonen

Answer 3

只适用于那些寻找使用lz4对文件/目录进行tar和压缩的Java实用程序代码的人。如果您发现任何问题，请告诉我。

import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.file.Path;
import java.nio.file.Paths;

import org.apache.commons.compress.archivers.tar.TarArchiveEntry;
import org.apache.commons.compress.archivers.tar.TarArchiveOutputStream;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import net.jpountz.lz4.LZ4FrameOutputStream;

public class TarLz4Util {
    private static final Logger logger = LoggerFactory.getLogger(TarLz4Util .class);

    private final static int BUFFER_SIZE = 1024 * 1024 * 10;

    public static void compress(String sourcePath, String targetPath) throws IOException {
        compress(Paths.get(sourcePath), Paths.get(sourcePath));
    }

    /**
     * compress with lz4
     *
     */
    private static void compress(Path sourcePath, Path targetPath) throws IOException {
        //Files ->  Tar  -> LZ4 
        try (LZ4FrameOutputStream lz4Out = new LZ4FrameOutputStream(
                new FileOutputStream(targetPath.toFile()))) {
            TarArchiveOutputStream tarOut = new TarArchiveOutputStream(lz4Out);
            tarOut.setLongFileMode(TarArchiveOutputStream.LONGFILE_GNU);
            tar(sourcePath.toFile(), tarOut);
        }
    }

    /**
     * tar your files/directories recursively
     *
     */
    private static void tar(File sourceFile, TarArchiveOutputStream tarOut) throws IOException {
        if (!sourceFile.exists()) {
            logger.warn("source file/directory does not exist!, path={}", sourceFile.getAbsolutePath());
            return;
        }

        TarArchiveEntry entry = new TarArchiveEntry(sourceFile);
        tarOut.putArchiveEntry(entry);
        if (sourceFile.isFile()) {
            writeFile(new BufferedInputStream(new FileInputStream(sourceFile)), tarOut);
            tarOut.closeArchiveEntry();
        } else {
            for (File file : sourceFile.listFiles()) {
                tar(file, tarOut);
            }
        }
    }

    /**
     * copy file
     *
     */
    private static long writeFile(InputStream in, OutputStream out) throws IOException {
        int read = 0;
        long count = 0;
        byte[] buf = new byte[BUFFER_SIZE];
        while ((read = in.read(buf)) != -1) {
            out.write(buf, 0, read);
            count += read;
        }
        return count;
    }

    // a demo
    public static void main(String[] args) throws Exception {
        String srcDir = "d:\\test";
        String tarFile = "test.tar.lz4";
       
        compress(Paths.get(srctDir), Paths.get(tarFile));
    }
}

- snowfox