以下是针对Spark 2.x数据集和数据框的推荐版本:
scala> val ds1 = spark.range(10)
ds1: org.apache.spark.sql.Dataset[Long] = [id: bigint]
scala> ds1.cache.count
res1: Long = 10
scala> val ds2 = spark.range(10)
ds2: org.apache.spark.sql.Dataset[Long] = [id: bigint]
scala> ds2.cache.count
res2: Long = 10
scala> val crossDS1DS2 = ds1.crossJoin(ds2)
crossDS1DS2: org.apache.spark.sql.DataFrame = [id: bigint, id: bigint]
scala> crossDS1DS2.count
res3: Long = 100
或者使用传统的JOIN语法而不需要JOIN条件。使用这个配置选项可以避免后面出现的错误。
spark.conf.set("spark.sql.crossJoin.enabled", true)
当省略该配置时出现错误(特别是使用“join”语法时):
scala> val crossDS1DS2 = ds1.join(ds2)
crossDS1DS2: org.apache.spark.sql.DataFrame = [id: bigint, id: bigint]
scala> crossDS1DS2.count
org.apache.spark.sql.AnalysisException: Detected cartesian product for INNER join between logical plans
...
Join condition is missing or trivial.
Use the CROSS JOIN syntax to allow cartesian products between these relations.
相关:Spark 2.x的spark.sql.crossJoin.enabled
(该参数用于启用或禁用Spark SQL中的跨连接操作。)