尝试从BigQuery读取外部表,但出现错误
SCALA_VERSION="2.12"
SPARK_VERSION="3.1.2"
com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.0,
com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.24.2'
table = 'data-lake.dataset.member'
df = spark.read.format('bigquery').load(table)
df.printSchema()
结果:
root
|-- createdAtmetadata: date (nullable = true)
|-- eventName: string (nullable = true)
|-- producerName: string (nullable = true)
所以当我打印
df.createOrReplaceTempView("member")
spark.sql("select * from member limit 100").show()
我收到了以下错误信息:
INVALID_ARGUMENT: 请求失败:只有带有连接的外部表才能使用 Storage API 读取。