pyspark.SparkContext.setLocalProperty

SparkContext.setLocalProperty(key, value)

Set a local property that affects jobs submitted from this thread, such as the Spark fair scheduler pool.

Notes

Currently, setting a local property with multiple threads does not properly work. Internally threads on PVM and JVM are not synced, and JVM thread can be reused for multiple threads on PVM, which fails to isolate local properties for each thread on PVM.

To avoid this, enable the pinned thread mode by setting PYSPARK_PIN_THREAD environment variable to true and uses pyspark.InheritableThread.

pyspark.SparkContext.setJobGroup pyspark.SparkContext.setLogLevel