我正在尝试在Hive中执行select * from db.abc的操作,该Hive表是使用Spark加载的。
但是出现错误,显示如下:
Error: java.io.IOException: java.lang.IllegalArgumentException: bucketId out of range: -1 (state=,code=0)
当我使用以下属性时,我能够查询Hive:
set hive.mapred.mode=nonstrict;
set hive.optimize.ppd=true;
set hive.optimize.index.filter=true;
set hive.tez.bucket.pruning=true;
set hive.explain.user=false;
set hive.fetch.task.conversion=none;
现在当我尝试使用Spark读取相同的Hive表db.abc时,我会收到以下错误:
如果客户端具备以下能力的话,那么他们才能访问这个表:CONNECTORREAD、HIVEFULLACIDREAD、HIVEFULLACIDWRITE、HIVEMANAGESTATS、HIVECACHEINVALIDATE、CONNECTORWRITE。这个表可能是一个由Hive管理的ACID表,或者需要Spark目前不支持的其他能力;在org.apache.spark.sql.catalyst.catalog.CatalogUtils$ .throwIfNoAccess (ExternalCatalogUtils.scala:280)和org.apache.spark.sql.hive.HiveTranslationLayerCheck$$anonfun$apply$1.applyOrElse(HiveTranslationLayerStrategies.scala:105)等位置将抛出异常。请问我需要在spark-submit或shell中添加任何属性吗?还是有其他方法可以使用spark读取这个hive数据表?
hive表的样本格式:
CREATE TABLE `hive``( |
| `c_id` decimal(11,0),etc.........
ROW FORMAT SERDE |
| 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' |
| WITH SERDEPROPERTIES (
STORED AS INPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' |
| OUTPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' |
LOCATION |
| path= 'hdfs://gjuyada/bbts/scl/raw' |
| TBLPROPERTIES ( |
| 'bucketing_version'='2', |
| 'spark.sql.create.version'='2.3.2.3.1.0.0-78', |
| 'spark.sql.sources.provider'='orc', |
| 'spark.sql.sources.schema.numParts'='1', |
| 'spark.sql.sources.schema.part.0'='{"type":"struct","fields":
[{"name":"Czz_ID","type":"decimal(11,0)","nullable":true,"metadata":{}},
{"name":"DzzzC_CD","type":"string","nullable":true,"metadata":{}},
{"name":"C0000_S_N","type":"decimal(11,0)","nullable":true,"metadata":{}},
{"name":"P_ _NB","type":"decimal(11,0)","nullable":true,"metadata":{}},
{"name":"C_YYYY","type":"string","nullable":true,"metadata":{}},"type":"string","nullable":true,"metadata":{}},{"name":"Cv_ID","type":"string","nullable":true,"metadata":{}},
| 'transactional'='true', |
| 'transient_lastDdlTime'='1574817059')