我正在使用
尝试 #2:
有人知道是否可以限制描述输出的内容吗?
DESCRIBE
关键字获取临时视图的列信息。这是一个有用的方法,但我有一个表,我只想描述其中的某些列。我尝试在DESCRIBE
与LIMIT
配合使用来实现这一点,但无法解决。
这里是一个玩具数据集(使用pyspark创建):
# make some test data
columns = ['id', 'dogs', 'cats', 'horses', 'people']
vals = [
(1, 2, 0, 4, 3),
(2, 0, 1, 2, 4)
]
# create DataFrame
df = spark.createDataFrame(vals, columns)
df.createOrReplaceTempView('df')
现在用SQL描述一下:
%%sql
DESCRIBE df
输出:
col_name data_type
id bigint
dogs bigint
cats bigint
horses bigint
people bigint
实际上我拥有的列比这个还要多,我想要做的是限制查询结果的输出。以下是我尝试过的一些方法:
尝试 #1:
DESCRIBE df
LIMIT 3
错误:
An error was encountered:
"\nextraneous input '3' expecting {<EOF>, '.'}(line 3, pos 6)\n\n== SQL ==\n\nDESCRIBE df\nLIMIT 3 \n------^^^\n"
Traceback (most recent call last):
File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 603, in sql
return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 73, in deco
raise ParseException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.ParseException: "\nextraneous input '3' expecting {<EOF>, '.'}(line 3, pos 6)\n\n== SQL ==\n\nDESCRIBE df\nLIMIT 3 \n------^^^\n"
尝试 #2:
SELECT a.*
FROM (
DESCRIBE df
) AS a
LIMIT 3
错误:
An error was encountered:
'Table or view not found: DESCRIBE; line 4 pos 4'
Traceback (most recent call last):
File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 603, in sql
return DataFrame(self._jsparkSession.sql(sqlQuery), self._wrapped)
File "/usr/lib/spark/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 69, in deco
raise AnalysisException(s.split(': ', 1)[1], stackTrace)
pyspark.sql.utils.AnalysisException: 'Table or view not found: DESCRIBE; line 4 pos 4'
有人知道是否可以限制描述输出的内容吗?