据说自 Spark 2.3.0 起,Apache Spark 支持 Facebook 的 Zstandard 压缩算法 (https://issues.apache.org/jira/browse/SPARK-19112),但我无法真正读取一个经过 Zstandard 压缩的文件:
$ spark-shell
...
// Short name throws an exception
scala> val events = spark.read.option("compression", "zstd").json("data.zst")
java.lang.IllegalArgumentException: Codec [zstd] is not available. Known codecs are bzip2, deflate, uncompressed, lz4, gzip, snappy, none.
// Codec class can be imported
scala> import org.apache.spark.io.ZStdCompressionCodec
import org.apache.spark.io.ZStdCompressionCodec
// Fully-qualified code class bypasses error, but results in corrupt records
scala> spark.read.option("compression", "org.apache.spark.io.ZStdCompressionCodec").json("data.zst")
res4: org.apache.spark.sql.DataFrame = [_corrupt_record: string]
我需要做什么才能读取这样的文件呢?
环境是 AWS EMR 5.14.0。
checknative -a
,看起来 zstd : false,也许 zstd 许可证不完全开放,Apache 团队决定在没有它的情况下构建? - Diego Scaravaggi2.10
上,当我添加了native库后,我得到了main org.apache.spark.sql.AnalysisException: java.lang.Uns atisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$POSIX.stat
,现在我将推迟nativa库,我将安排一个带有3.x的测试平台,并等待bigtop apache团队发布稳定版本1.6。 - Diego Scaravaggi