Upgrading from PySpark 2.3.0 to 2.3.1 and aboveΒΆ

  • As of version 2.3.1 Arrow functionality, including pandas_udf and toPandas()/createDataFrame() with spark.sql.execution.arrow.enabled set to True, has been marked as experimental. These are still evolving and not currently recommended for use in production.