pyspark.sql.functions.
spark_partition_id
A column for partition ID.
New in version 1.6.0.
Notes
This is non deterministic because it depends on data partitioning and task scheduling.
Examples
>>> df.repartition(1).select(spark_partition_id().alias("pid")).collect() [Row(pid=0), Row(pid=0)]