我同意@lev的观点,我已经想了很长时间,所以我写了一个简单的小代码来确保它可以工作,请注意!!为了控制每个驱动程序的工人数量,您需要使用coalesce限制数据框/集合。
以下是示例代码:
import org.apache.spark.SparkConf
import org.apache.spark.sql.SparkSession
object SparkMultiThreadExample extends App{
val TOTAL_WORKERS = 10
val NUMBER_OF_WORKERS_PER_DRIVER = 2
val sparkConf = new SparkConf()
sparkConf.setMaster(s"local[${TOTAL_WORKERS}]")
val spark = SparkSession.builder().config(sparkConf).getOrCreate()
val list1 = (0 until 10).toList
import spark.implicits._
list1.par.foreach(t => {
spark.createDataset(list1).coalesce(NUMBER_OF_WORKERS_PER_DRIVER).foreach(i => {
println(s"${Thread.currentThread()}, Driver thread ${t}: This is inside worker ${i} " )
Thread.sleep(1000)
println(s"FINISH ${Thread.currentThread()} Driver thread ${t}: This is inside worker ${i} " )
})
}) }
输出:
Thread[Executor task launch worker for task 0,5,main], Driver thread 0: This is inside worker 0
Thread[Executor task launch worker for task 4,5,main], Driver thread 3: This is inside worker 0
Thread[Executor task launch worker for task 7,5,main], Driver thread 5: This is inside worker 5
Thread[Executor task launch worker for task 1,5,main], Driver thread 0: This is inside worker 5
Thread[Executor task launch worker for task 3,5,main], Driver thread 2: This is inside worker 5
Thread[Executor task launch worker for task 6,5,main], Driver thread 5: This is inside worker 0
Thread[Executor task launch worker for task 2,5,main], Driver thread 2: This is inside worker 0
Thread[Executor task launch worker for task 5,5,main], Driver thread 3: This is inside worker 5
Thread[Executor task launch worker for task 9,5,main], Driver thread 4: This is inside worker 5
Thread[Executor task launch worker for task 8,5,main], Driver thread 4: This is inside worker 0
FINISH Thread[Executor task launch worker for task 0,5,main] Driver thread 0: This is inside worker 0
FINISH Thread[Executor task launch worker for task 7,5,main] Driver thread 5: This is inside worker 5
FINISH Thread[Executor task launch worker for task 4,5,main] Driver thread 3: This is inside worker 0
FINISH Thread[Executor task launch worker for task 3,5,main] Driver thread 2: This is inside worker 5
FINISH Thread[Executor task launch worker for task 1,5,main] Driver thread 0: This is inside worker 5
Thread[Executor task launch worker for task 3,5,main], Driver thread 2: This is inside worker 6
Thread[Executor task launch worker for task 4,5,main], Driver thread 3: This is inside worker 1
Thread[Executor task launch worker for task 1,5,main], Driver thread 0: This is inside worker 6
Thread[Executor task launch worker for task 0,5,main], Driver thread 0: This is inside worker 1
Thread[Executor task launch worker for task 7,5,main], Driver thread 5: This is inside worker 6
FINISH Thread[Executor task launch worker for task 2,5,main] Driver thread 2: This is inside worker 0
FINISH Thread[Executor task launch worker for task 5,5,main] Driver thread 3: This is inside worker 5
Thread[Executor task launch worker for task 2,5,main], Driver thread 2: This is inside worker 1
FINISH Thread[Executor task launch worker for task 9,5,main] Driver thread 4: This is inside worker 5
FINISH Thread[Executor task launch worker for task 6,5,main] Driver thread 5: This is inside worker 0
Thread[Executor task launch worker for task 9,5,main], Driver thread 4: This is inside worker 6
Thread[Executor task launch worker for task 5,5,main], Driver thread 3: This is inside worker 6
FINISH Thread[Executor task launch worker for task 8,5,main] Driver thread 4: This is inside worker 0
Thread[Executor task launch worker for task 6,5,main], Driver thread 5: This is inside worker 1
Thread[Executor task launch worker for task 8,5,main], Driver thread 4: This is inside worker 1
FINISH Thread[Executor task launch worker for task 3,5,main] Driver thread 2: This is inside worker 6
FINISH Thread[Executor task launch worker for task 4,5,main] Driver thread 3: This is inside worker 1
FINISH Thread[Executor task launch worker for task 1,5,main] Driver thread 0: This is inside worker 6
Thread[Executor task launch worker for task 4,5,main], Driver thread 3: This is inside worker 2
Thread[Executor task launch worker for task 3,5,main], Driver thread 2: This is inside worker 7
FINISH Thread[Executor task launch worker for task 0,5,main] Driver thread 0: This is inside worker 1
Thread[Executor task launch worker for task 1,5,main], Driver thread 0: This is inside worker 7
FINISH Thread[Executor task launch worker for task 7,5,main] Driver thread 5: This is inside worker 6
Thread[Executor task launch worker for task 0,5,main], Driver thread 0: This is inside worker 2
Thread[Executor task launch worker for task 7,5,main], Driver thread 5: This is inside worker 7
FINISH Thread[Executor task launch worker for task 2,5,main] Driver thread 2: This is inside worker 1
Thread[Executor task launch worker for task 2,5,main], Driver thread 2: This is inside worker 2
FINISH Thread[Executor task launch worker for task 9,5,main] Driver thread 4: This is inside worker 6
Thread[Executor task launch worker for task 9,5,main], Driver thread 4: This is inside worker 7
FINISH Thread[Executor task launch worker for task 5,5,main] Driver thread 3: This is inside worker 6
Thread[Executor task launch worker for task 5,5,main], Driver thread 3: This is inside worker 7
FINISH Thread[Executor task launch worker for task 6,5,main] Driver thread 5: This is inside worker 1
Thread[Executor task launch worker for task 6,5,main], Driver thread 5: This is inside worker 2
FINISH Thread[Executor task launch worker for task 8,5,main] Driver thread 4: This is inside worker 1
Thread[Executor task launch worker for task 8,5,main], Driver thread 4: This is inside worker 2
FINISH Thread[Executor task launch worker for task 4,5,main] Driver thread 3: This is inside worker 2
FINISH Thread[Executor task launch worker for task 7,5,main] Driver thread 5: This is inside worker 7
FINISH Thread[Executor task launch worker for task 0,5,main] Driver thread 0: This is inside worker 2
FINISH Thread[Executor task launch worker for task 1,5,main] Driver thread 0: This is inside worker 7
FINISH Thread[Executor task launch worker for task 3,5,main] Driver thread 2: This is inside worker 7
Thread[Executor task launch worker for task 7,5,main], Driver thread 5: This is inside worker 8
Thread[Executor task launch worker for task 4,5,main], Driver thread 3: This is inside worker 3
Thread[Executor task launch worker for task 3,5,main], Driver thread 2: This is inside worker 8
Thread[Executor task launch worker for task 0,5,main], Driver thread 0: This is inside worker 3
Thread[Executor task launch worker for task 1,5,main], Driver thread 0: This is inside worker 8
FINISH Thread[Executor task launch worker for task 2,5,main] Driver thread 2: This is inside worker 2
Thread[Executor task launch worker for task 2,5,main], Driver thread 2: This is inside worker 3
FINISH Thread[Executor task launch worker for task 9,5,main] Driver thread 4: This is inside worker 7
FINISH Thread[Executor task launch worker for task 5,5,main] Driver thread 3: This is inside worker 7
Thread[Executor task launch worker for task 9,5,main], Driver thread 4: This is inside worker 8
Thread[Executor task launch worker for task 5,5,main], Driver thread 3: This is inside worker 8
FINISH Thread[Executor task launch worker for task 6,5,main] Driver thread 5: This is inside worker 2
FINISH Thread[Executor task launch worker for task 8,5,main] Driver thread 4: This is inside worker 2
Thread[Executor task launch worker for task 6,5,main], Driver thread 5: This is inside worker 3
Thread[Executor task launch worker for task 8,5,main], Driver thread 4: This is inside worker 3
FINISH Thread[Executor task launch worker for task 7,5,main] Driver thread 5: This is inside worker 8
FINISH Thread[Executor task launch worker for task 4,5,main] Driver thread 3: This is inside worker 3
FINISH Thread[Executor task launch worker for task 0,5,main] Driver thread 0: This is inside worker 3
FINISH Thread[Executor task launch worker for task 3,5,main] Driver thread 2: This is inside worker 8
Thread[Executor task launch worker for task 0,5,main], Driver thread 0: This is inside worker 4
Thread[Executor task launch worker for task 3,5,main], Driver thread 2: This is inside worker 9
Thread[Executor task launch worker for task 4,5,main], Driver thread 3: This is inside worker 4
Thread[Executor task launch worker for task 7,5,main], Driver thread 5: This is inside worker 9
FINISH Thread[Executor task launch worker for task 1,5,main] Driver thread 0: This is inside worker 8
Thread[Executor task launch worker for task 1,5,main], Driver thread 0: This is inside worker 9
FINISH Thread[Executor task launch worker for task 2,5,main] Driver thread 2: This is inside worker 3
Thread[Executor task launch worker for task 2,5,main], Driver thread 2: This is inside worker 4
FINISH Thread[Executor task launch worker for task 9,5,main] Driver thread 4: This is inside worker 8
FINISH Thread[Executor task launch worker for task 5,5,main] Driver thread 3: This is inside worker 8
Thread[Executor task launch worker for task 9,5,main], Driver thread 4: This is inside worker 9
FINISH Thread[Executor task launch worker for task 6,5,main] Driver thread 5: This is inside worker 3
FINISH Thread[Executor task launch worker for task 8,5,main] Driver thread 4: This is inside worker 3
Thread[Executor task launch worker for task 5,5,main], Driver thread 3: This is inside worker 9
Thread[Executor task launch worker for task 8,5,main], Driver thread 4: This is inside worker 4
Thread[Executor task launch worker for task 6,5,main], Driver thread 5: This is inside worker 4
FINISH Thread[Executor task launch worker for task 0,5,main] Driver thread 0: This is inside worker 4
FINISH Thread[Executor task launch worker for task 4,5,main] Driver thread 3: This is inside worker 4
FINISH Thread[Executor task launch worker for task 3,5,main] Driver thread 2: This is inside worker 9
FINISH Thread[Executor task launch worker for task 7,5,main] Driver thread 5: This is inside worker 9
FINISH Thread[Executor task launch worker for task 1,5,main] Driver thread 0: This is inside worker 9
FINISH Thread[Executor task launch worker for task 2,5,main] Driver thread 2: This is inside worker 4
FINISH Thread[Executor task launch worker for task 9,5,main] Driver thread 4: This is inside worker 9
FINISH Thread[Executor task launch worker for task 5,5,main] Driver thread 3: This is inside worker 9
FINISH Thread[Executor task launch worker for task 6,5,main] Driver thread 5: This is inside worker 4
FINISH Thread[Executor task launch worker for task 8,5,main] Driver thread 4: This is inside worker 4
Thread[Executor task launch worker for task 11,5,main], Driver thread 7: This is inside worker 5
Thread[Executor task launch worker for task 10,5,main], Driver thread 7: This is inside worker 0
Thread[Executor task launch worker for task 12,5,main], Driver thread 6: This is inside worker 0
Thread[Executor task launch worker for task 13,5,main], Driver thread 6: This is inside worker 5
Thread[Executor task launch worker for task 14,5,main], Driver thread 1: This is inside worker 0
Thread[Executor task launch worker for task 15,5,main], Driver thread 1: This is inside worker 5
Thread[Executor task launch worker for task 16,5,main], Driver thread 8: This is inside worker 0
Thread[Executor task launch worker for task 17,5,main], Driver thread 8: This is inside worker 5
Thread[Executor task launch worker for task 19,5,main], Driver thread 9: This is inside worker 5
Thread[Executor task launch worker for task 18,5,main], Driver thread 9: This is inside worker 0
FINISH Thread[Executor task launch worker for task 11,5,main] Driver thread 7: This is inside worker 5
Thread[Executor task launch worker for task 11,5,main], Driver thread 7: This is inside worker 6
FINISH Thread[Executor task launch worker for task 10,5,main] Driver thread 7: This is inside worker 0
Thread[Executor task launch worker for task 10,5,main], Driver thread 7: This is inside worker 1
FINISH Thread[Executor task launch worker for task 12,5,main] Driver thread 6: This is inside worker 0
Thread[Executor task launch worker for task 12,5,main], Driver thread 6: This is inside worker 1
FINISH Thread[Executor task launch worker for task 13,5,main] Driver thread 6: This is inside worker 5
Thread[Executor task launch worker for task 13,5,main], Driver thread 6: This is inside worker 6
FINISH Thread[Executor task launch worker for task 14,5,main] Driver thread 1: This is inside worker 0
Thread[Executor task launch worker for task 14,5,main], Driver thread 1: This is inside worker 1
FINISH Thread[Executor task launch worker for task 15,5,main] Driver thread 1: This is inside worker 5
Thread[Executor task launch worker for task 15,5,main], Driver thread 1: This is inside worker 6
FINISH Thread[Executor task launch worker for task 16,5,main] Driver thread 8: This is inside worker 0
Thread[Executor task launch worker for task 16,5,main], Driver thread 8: This is inside worker 1
FINISH Thread[Executor task launch worker for task 17,5,main] Driver thread 8: This is inside worker 5
Thread[Executor task launch worker for task 17,5,main], Driver thread 8: This is inside worker 6
FINISH Thread[Executor task launch worker for task 19,5,main] Driver thread 9: This is inside worker 5
Thread[Executor task launch worker for task 19,5,main], Driver thread 9: This is inside worker 6
FINISH Thread[Executor task launch worker for task 18,5,main] Driver thread 9: This is inside worker 0
Thread[Executor task launch worker for task 18,5,main], Driver thread 9: This is inside worker 1
FINISH Thread[Executor task launch worker for task 11,5,main] Driver thread 7: This is inside worker 6
Thread[Executor task launch worker for task 11,5,main], Driver thread 7: This is inside worker 7
FINISH Thread[Executor task launch worker for task 10,5,main] Driver thread 7: This is inside worker 1
Thread[Executor task launch worker for task 10,5,main], Driver thread 7: This is inside worker 2
FINISH Thread[Executor task launch worker for task 12,5,main] Driver thread 6: This is inside worker 1
Thread[Executor task launch worker for task 12,5,main], Driver thread 6: This is inside worker 2
FINISH Thread[Executor task launch worker for task 13,5,main] Driver thread 6: This is inside worker 6
Thread[Executor task launch worker for task 13,5,main], Driver thread 6: This is inside worker 7
FINISH Thread[Executor task launch worker for task 14,5,main] Driver thread 1: This is inside worker 1
Thread[Executor task launch worker for task 14,5,main], Driver thread 1: This is inside worker 2
FINISH Thread[Executor task launch worker for task 15,5,main] Driver thread 1: This is inside worker 6
Thread[Executor task launch worker for task 15,5,main], Driver thread 1: This is inside worker 7
FINISH Thread[Executor task launch worker for task 16,5,main] Driver thread 8: This is inside worker 1
Thread[Executor task launch worker for task 16,5,main], Driver thread 8: This is inside worker 2
FINISH Thread[Executor task launch worker for task 17,5,main] Driver thread 8: This is inside worker 6
Thread[Executor task launch worker for task 17,5,main], Driver thread 8: This is inside worker 7
FINISH Thread[Executor task launch worker for task 19,5,main] Driver thread 9: This is inside worker 6
Thread[Executor task launch worker for task 19,5,main], Driver thread 9: This is inside worker 7
FINISH Thread[Executor task launch worker for task 18,5,main] Driver thread 9: This is inside worker 1
Thread[Executor task launch worker for task 18,5,main], Driver thread 9: This is inside worker 2
FINISH Thread[Executor task launch worker for task 11,5,main] Driver thread 7: This is inside worker 7
Thread[Executor task launch worker for task 11,5,main], Driver thread 7: This is inside worker 8
FINISH Thread[Executor task launch worker for task 10,5,main] Driver thread 7: This is inside worker 2
Thread[Executor task launch worker for task 10,5,main], Driver thread 7: This is inside worker 3
FINISH Thread[Executor task launch worker for task 12,5,main] Driver thread 6: This is inside worker 2
Thread[Executor task launch worker for task 12,5,main], Driver thread 6: This is inside worker 3
FINISH Thread[Executor task launch worker for task 13,5,main] Driver thread 6: This is inside worker 7
Thread[Executor task launch worker for task 13,5,main], Driver thread 6: This is inside worker 8
FINISH Thread[Executor task launch worker for task 14,5,main] Driver thread 1: This is inside worker 2
Thread[Executor task launch worker for task 14,5,main], Driver thread 1: This is inside worker 3
FINISH Thread[Executor task launch worker for task 15,5,main] Driver thread 1: This is inside worker 7
Thread[Executor task launch worker for task 15,5,main], Driver thread 1: This is inside worker 8
FINISH Thread[Executor task launch worker for task 16,5,main] Driver thread 8: This is inside worker 2
Thread[Executor task launch worker for task 16,5,main], Driver thread 8: This is inside worker 3
FINISH Thread[Executor task launch worker for task 17,5,main] Driver thread 8: This is inside worker 7
Thread[Executor task launch worker for task 17,5,main], Driver thread 8: This is inside worker 8
FINISH Thread[Executor task launch worker for task 19,5,main] Driver thread 9: This is inside worker 7
Thread[Executor task launch worker for task 19,5,main], Driver thread 9: This is inside worker 8
FINISH Thread[Executor task launch worker for task 18,5,main] Driver thread 9: This is inside worker 2
Thread[Executor task launch worker for task 18,5,main], Driver thread 9: This is inside worker 3
FINISH Thread[Executor task launch worker for task 11,5,main] Driver thread 7: This is inside worker 8
Thread[Executor task launch worker for task 11,5,main], Driver thread 7: This is inside worker 9
FINISH Thread[Executor task launch worker for task 10,5,main] Driver thread 7: This is inside worker 3
Thread[Executor task launch worker for task 10,5,main], Driver thread 7: This is inside worker 4
FINISH Thread[Executor task launch worker for task 12,5,main] Driver thread 6: This is inside worker 3
Thread[Executor task launch worker for task 12,5,main], Driver thread 6: This is inside worker 4
FINISH Thread[Executor task launch worker for task 13,5,main] Driver thread 6: This is inside worker 8
Thread[Executor task launch worker for task 13,5,main], Driver thread 6: This is inside worker 9
FINISH Thread[Executor task launch worker for task 14,5,main] Driver thread 1: This is inside worker 3
Thread[Executor task launch worker for task 14,5,main], Driver thread 1: This is inside worker 4
FINISH Thread[Executor task launch worker for task 15,5,main] Driver thread 1: This is inside worker 8
Thread[Executor task launch worker for task 15,5,main], Driver thread 1: This is inside worker 9
FINISH Thread[Executor task launch worker for task 16,5,main] Driver thread 8: This is inside worker 3
Thread[Executor task launch worker for task 16,5,main], Driver thread 8: This is inside worker 4
FINISH Thread[Executor task launch worker for task 17,5,main] Driver thread 8: This is inside worker 8
Thread[Executor task launch worker for task 17,5,main], Driver thread 8: This is inside worker 9
FINISH Thread[Executor task launch worker for task 19,5,main] Driver thread 9: This is inside worker 8
Thread[Executor task launch worker for task 19,5,main], Driver thread 9: This is inside worker 9
FINISH Thread[Executor task launch worker for task 18,5,main] Driver thread 9: This is inside worker 3
Thread[Executor task launch worker for task 18,5,main], Driver thread 9: This is inside worker 4
FINISH Thread[Executor task launch worker for task 11,5,main] Driver thread 7: This is inside worker 9
FINISH Thread[Executor task launch worker for task 10,5,main] Driver thread 7: This is inside worker 4
FINISH Thread[Executor task launch worker for task 12,5,main] Driver thread 6: This is inside worker 4
FINISH Thread[Executor task launch worker for task 13,5,main] Driver thread 6: This is inside worker 9
FINISH Thread[Executor task launch worker for task 14,5,main] Driver thread 1: This is inside worker 4
FINISH Thread[Executor task launch worker for task 15,5,main] Driver thread 1: This is inside worker 9
FINISH Thread[Executor task launch worker for task 16,5,main] Driver thread 8: This is inside worker 4
FINISH Thread[Executor task launch worker for task 17,5,main] Driver thread 8: This is inside worker 9
FINISH Thread[Executor task launch worker for task 19,5,main] Driver thread 9: This is inside worker 9
FINISH Thread[Executor task launch worker for task 18,5,main] Driver thread 9: This is inside worker 4