如何在Spark中读取没有扩展名的压缩（gzip）文件

Question

我刚接触Spark，手头有一个有趣的任务，需要从S3读取一堆包含XML内容的文件。

这些文件是压缩的(Gzip)，但没有该扩展名。

我在这里看到了一些相关的问题，人们建议扩展Spark中的默认编解码器并强制使用不同的扩展名。

但在我的情况下，文件没有扩展名，并且以16位UUID格式命名，例如 2c7358ca472ad91057da84adfba。

- g.cyberian

---

相关：在Spark追踪器上有一个问题(https://issues.apache.org/jira/browse/SPARK-29280)，关于添加一种显式地指定压缩编解码器的方法，以便Spark不会从文件扩展名推断它。 - Nick Chammas

Answer 1

你可以使用newAPIHadoopFile（而不是textFile），并配合自定义/修改的TextInputFormat，强制使用GzipCodec。

而不是调用sparkContext.textFile，

// gzip compressed but no .gz extension:
sparkContext.textFile("s3://mybucket/uuid")

我们可以使用底层的sparkContext.newAPIHadoopFile函数，该函数允许我们指定如何读取输入内容。

import org.apache.hadoop.mapreduce.lib.input.GzipInputFormatWithoutExtention
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.io.{LongWritable, Text}

sparkContext
  .newAPIHadoopFile(
    "s3://mybucket/uuid",
    classOf[GzipInputFormatWithoutExtention], // This is our custom reader
    classOf[LongWritable],
    classOf[Text],
    new Configuration(sparkContext.hadoopConfiguration)
  )
  .map { case (_, text) => text.toString }

通常调用newAPIHadoopFile的方式是使用TextInputFormat。这部分包装了文件的读取方式以及根据文件扩展名选择压缩编解码器的部分。

我们将其称为GzipInputFormatWithoutExtention，并按照以下方式实现它作为TextInputFormat的扩展（这是一个Java文件，让我们将其放在src/main/java/org/apache/hadoop/mapreduce/lib/input包中）：

package org.apache.hadoop.mapreduce.lib.input;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.InputSplit;
import org.apache.hadoop.mapreduce.JobContext;
import org.apache.hadoop.mapreduce.RecordReader;
import org.apache.hadoop.mapreduce.TaskAttemptContext;

import com.google.common.base.Charsets;

public class GzipInputFormatWithoutExtention extends TextInputFormat {

    public RecordReader<LongWritable, Text> createRecordReader(
        InputSplit split,
        TaskAttemptContext context
    ) {

        String delimiter =
            context.getConfiguration().get("textinputformat.record.delimiter");

        byte[] recordDelimiterBytes = null;
        if (null != delimiter)
            recordDelimiterBytes = delimiter.getBytes(Charsets.UTF_8);

        // Here we use our custom `GzipWithoutExtentionLineRecordReader`
        // instead of `LineRecordReader`:
        return new GzipWithoutExtentionLineRecordReader(recordDelimiterBytes);
    }

    @Override
    protected boolean isSplitable(JobContext context, Path file) {
        return false; // gzip isn't a splittable codec (as opposed to bzip2)
    }
}

实际上，我们需要再深入一层，并将默认的 LineRecordReader（Java）替换为我们自己的（让我们称其为GzipWithoutExtentionLineRecordReader）。

由于从LineRecordReader继承相当困难，因此我们可以复制 LineRecordReader（在src/main/java/org/apache/hadoop/mapreduce/lib/input中），并稍微修改（和简化）initialize(InputSplit genericSplit, TaskAttemptContext context)方法来强制使用Gzip编解码器：

与原始 LineRecordReader 相比，唯一的更改是添加了注释以解释正在发生的事情。

package org.apache.hadoop.mapreduce.lib.input;

import java.io.IOException;

import org.apache.hadoop.io.compress.*;

import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.Seekable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.InputSplit;
import org.apache.hadoop.mapreduce.RecordReader;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

@InterfaceAudience.LimitedPrivate({"MapReduce", "Pig"})
@InterfaceStability.Evolving
public class GzipWithoutExtentionLineRecordReader extends RecordReader<LongWritable, Text> {
    private static final Logger LOG =
            LoggerFactory.getLogger(GzipWithoutExtentionLineRecordReader.class);
    public static final String MAX_LINE_LENGTH =
            "mapreduce.input.linerecordreader.line.maxlength";

    private long start;
    private long pos;
    private long end;
    private SplitLineReader in;
    private FSDataInputStream fileIn;
    private Seekable filePosition;
    private int maxLineLength;
    private LongWritable key;
    private Text value;
    private boolean isCompressedInput;
    private Decompressor decompressor;
    private byte[] recordDelimiterBytes;

    public GzipWithoutExtentionLineRecordReader(byte[] recordDelimiter) {
        this.recordDelimiterBytes = recordDelimiter;
    }

    public void initialize(
        InputSplit genericSplit,
        TaskAttemptContext context
    ) throws IOException {

        FileSplit split = (FileSplit) genericSplit;
        Configuration job = context.getConfiguration();
        this.maxLineLength = job.getInt(MAX_LINE_LENGTH, Integer.MAX_VALUE);
        start = split.getStart();
        end = start + split.getLength();
        final Path file = split.getPath();

        // open the file and seek to the start of the split
        final FileSystem fs = file.getFileSystem(job);
        fileIn = fs.open(file);

        // This line is modified to force the use of the GzipCodec:
        // CompressionCodec codec = new CompressionCodecFactory(job).getCodec(file);
        CompressionCodecFactory ccf = new CompressionCodecFactory(job);
        CompressionCodec codec = ccf.getCodecByClassName(GzipCodec.class.getName());

        // This part has been extremely simplified as we don't have to handle
        // all the different codecs:
        isCompressedInput = true;
        decompressor = CodecPool.getDecompressor(codec);
        if (start != 0) {
            throw new IOException(
                "Cannot seek in " + codec.getClass().getSimpleName() + " compressed stream"
            );
        }

        in = new SplitLineReader(
            codec.createInputStream(fileIn, decompressor), job, this.recordDelimiterBytes
        );
        filePosition = fileIn;
        if (start != 0) {
            start += in.readLine(new Text(), 0, maxBytesToConsume(start));
        }
        this.pos = start;
    }

    private int maxBytesToConsume(long pos) {
        return isCompressedInput
                ? Integer.MAX_VALUE
                : (int) Math.max(Math.min(Integer.MAX_VALUE, end - pos), maxLineLength);
    }

    private long getFilePosition() throws IOException {
        long retVal;
        if (isCompressedInput && null != filePosition) {
            retVal = filePosition.getPos();
        } else {
            retVal = pos;
        }
        return retVal;
    }

    private int skipUtfByteOrderMark() throws IOException {
        int newMaxLineLength = (int) Math.min(3L + (long) maxLineLength,
                Integer.MAX_VALUE);
        int newSize = in.readLine(value, newMaxLineLength, maxBytesToConsume(pos));
        pos += newSize;
        int textLength = value.getLength();
        byte[] textBytes = value.getBytes();
        if ((textLength >= 3) && (textBytes[0] == (byte)0xEF) &&
                (textBytes[1] == (byte)0xBB) && (textBytes[2] == (byte)0xBF)) {
            LOG.info("Found UTF-8 BOM and skipped it");
            textLength -= 3;
            newSize -= 3;
            if (textLength > 0) {
                textBytes = value.copyBytes();
                value.set(textBytes, 3, textLength);
            } else {
                value.clear();
            }
        }
        return newSize;
    }

    public boolean nextKeyValue() throws IOException {
        if (key == null) {
            key = new LongWritable();
        }
        key.set(pos);
        if (value == null) {
            value = new Text();
        }
        int newSize = 0;
        while (getFilePosition() <= end || in.needAdditionalRecordAfterSplit()) {
            if (pos == 0) {
                newSize = skipUtfByteOrderMark();
            } else {
                newSize = in.readLine(value, maxLineLength, maxBytesToConsume(pos));
                pos += newSize;
            }

            if ((newSize == 0) || (newSize < maxLineLength)) {
                break;
            }

            LOG.info("Skipped line of size " + newSize + " at pos " +
                    (pos - newSize));
        }
        if (newSize == 0) {
            key = null;
            value = null;
            return false;
        } else {
            return true;
        }
    }

    @Override
    public LongWritable getCurrentKey() {
        return key;
    }

    @Override
    public Text getCurrentValue() {
        return value;
    }

    public float getProgress() throws IOException {
        if (start == end) {
            return 0.0f;
        } else {
            return Math.min(1.0f, (getFilePosition() - start) / (float)(end - start));
        }
    }

    public synchronized void close() throws IOException {
        try {
            if (in != null) {
                in.close();
            }
        } finally {
            if (decompressor != null) {
                CodecPool.returnDecompressor(decompressor);
                decompressor = null;
            }
        }
    }
}

- Xavier Guihot