pyspark.testing.assertDataFrameEqual¶

pyspark.testing.assertDataFrameEqual(actual: Union[pyspark.sql.dataframe.DataFrame, pandas.DataFrame, pyspark.pandas.DataFrame, List[pyspark.sql.types.Row]], expected: Union[pyspark.sql.dataframe.DataFrame, pandas.DataFrame, pyspark.pandas.DataFrame, List[pyspark.sql.types.Row]], checkRowOrder: bool = False, rtol: float = 1e-05, atol: float = 1e-08)[source]¶

A util function to assert equality between actual and expected (DataFrames or lists of Rows), with optional parameters checkRowOrder, rtol, and atol.

Supports Spark, Spark Connect, pandas, and pandas-on-Spark DataFrames. For more information about pandas-on-Spark DataFrame equality, see the docs for assertPandasOnSparkEqual.

New in version 3.5.0.

Parameters

actualDataFrame (Spark, Spark Connect, pandas, or pandas-on-Spark) or list of Rows: The DataFrame that is being compared or tested.
expectedDataFrame (Spark, Spark Connect, pandas, or pandas-on-Spark) or list of Rows: The expected result of the operation, for comparison with the actual result.
checkRowOrderbool, optional: A flag indicating whether the order of rows should be considered in the comparison. If set to False (default), the row order is not taken into account. If set to True, the order of rows is important and will be checked during comparison. (See Notes)
rtolfloat, optional: The relative tolerance, used in asserting approximate equality for float values in actual and expected. Set to 1e-5 by default. (See Notes)
atolfloat, optional: The absolute tolerance, used in asserting approximate equality for float values in actual and expected. Set to 1e-8 by default. (See Notes)

Notes

When assertDataFrameEqual fails, the error message uses the Python difflib library to display a diff log of each row that differs in actual and expected.

For checkRowOrder, note that PySpark DataFrame ordering is non-deterministic, unless explicitly sorted.

Note that schema equality is checked only when expected is a DataFrame (not a list of Rows).

For DataFrames with float values, assertDataFrame asserts approximate equality. Two float values a and b are approximately equal if the following equation is True:

absolute(a - b) <= (atol + rtol * absolute(b)).

Examples

>>> df1 = spark.createDataFrame(data=[("1", 1000), ("2", 3000)], schema=["id", "amount"])
>>> df2 = spark.createDataFrame(data=[("1", 1000), ("2", 3000)], schema=["id", "amount"])
>>> assertDataFrameEqual(df1, df2)  # pass, DataFrames are identical

>>> df1 = spark.createDataFrame(data=[("1", 0.1), ("2", 3.23)], schema=["id", "amount"])
>>> df2 = spark.createDataFrame(data=[("1", 0.109), ("2", 3.23)], schema=["id", "amount"])
>>> assertDataFrameEqual(df1, df2, rtol=1e-1)  # pass, DataFrames are approx equal by rtol

>>> df1 = spark.createDataFrame(data=[(1, 1000), (2, 3000)], schema=["id", "amount"])
>>> list_of_rows = [Row(1, 1000), Row(2, 3000)]
>>> assertDataFrameEqual(df1, list_of_rows)  # pass, actual and expected data are equal

>>> import pyspark.pandas as ps
>>> df1 = ps.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9]})
>>> df2 = ps.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9]})
>>> assertDataFrameEqual(df1, df2)  # pass, pandas-on-Spark DataFrames are equal

>>> df1 = spark.createDataFrame(
...     data=[("1", 1000.00), ("2", 3000.00), ("3", 2000.00)], schema=["id", "amount"])
>>> df2 = spark.createDataFrame(
...     data=[("1", 1001.00), ("2", 3000.00), ("3", 2003.00)], schema=["id", "amount"])
>>> assertDataFrameEqual(df1, df2)  
Traceback (most recent call last):
...
PySparkAssertionError: [DIFFERENT_ROWS] Results do not match: ( 66.66667 % )
*** actual ***
! Row(id='1', amount=1000.0)
Row(id='2', amount=3000.0)
! Row(id='3', amount=2000.0)
*** expected ***
! Row(id='1', amount=1001.0)
Row(id='2', amount=3000.0)
! Row(id='3', amount=2003.0)

Testing

pyspark.testing.assertPandasOnSparkEqual