DataFrameWriter.bucketBy(numBuckets: int, col: Union[str, List[str], Tuple[str, …]], *cols: Optional[str]) → pyspark.sql.readwriter.DataFrameWriter[source]

Buckets the output by the given columns. If specified, the output is laid out on the file system similar to Hive’s bucketing scheme, but with a different bucket hash function and is not compatible with Hive’s bucketing.

New in version 2.3.0.

Changed in version 3.4.0: Supports Spark Connect.


the number of buckets to save

colstr, list or tuple

a name of a column, or a list of names.


additional names (optional). If col is a list it should be empty.


Applicable for file-based data sources in combination with DataFrameWriter.saveAsTable().


Write a DataFrame into a Parquet file in a buckted manner, and read it back.

>>> from pyspark.sql.functions import input_file_name
>>> # Write a DataFrame into a Parquet file in a bucketed manner.
... _ = spark.sql("DROP TABLE IF EXISTS bucketed_table")
>>> spark.createDataFrame([
...     (100, "Hyukjin Kwon"), (120, "Hyukjin Kwon"), (140, "Haejoon Lee")],
...     schema=["age", "name"]
... ).write.bucketBy(2, "name").mode("overwrite").saveAsTable("bucketed_table")
>>> # Read the Parquet file as a DataFrame.
|age|        name|
|100|Hyukjin Kwon|
|120|Hyukjin Kwon|
|140| Haejoon Lee|
>>> _ = spark.sql("DROP TABLE bucketed_table")