DataFrame.
sampleBy
Returns a stratified sample without replacement based on the fraction given on each stratum.
New in version 1.5.0.
Column
column that defines strata
Changed in version 3.0: Added sampling by a column of Column
sampling fraction for each stratum. If a stratum is not specified, we treat its fraction as zero.
random seed
DataFrame
Examples
>>> from pyspark.sql.functions import col >>> dataset = sqlContext.range(0, 100).select((col("id") % 3).alias("key")) >>> sampled = dataset.sampleBy("key", fractions={0: 0.1, 1: 0.2}, seed=0) >>> sampled.groupBy("key").count().orderBy("key").show() +---+-----+ |key|count| +---+-----+ | 0| 3| | 1| 6| +---+-----+ >>> dataset.sampleBy(col("key"), fractions={2: 1.0}, seed=0).count() 33