Spark Release 3.5.5

Spark 3.5.5 is the fourth maintenance release containing security and correctness fixes. This release is based on the branch-3.5 maintenance branch of Spark. We strongly recommend all 3.5 users to upgrade to this stable release.

Notable changes

  • [SPARK-45013]: Flaky Test with NPE: track allocated resources by taskId
  • [SPARK-48899]: Fix ENV key value format in K8s Dockerfiles
  • [SPARK-50091]: Query fails when aggregate expression is in left-hand operand of IN-subquery
  • [SPARK-50624]: Add TimestampNTZType to ColumnarRow/MutableColumnarRow
  • [SPARK-50769]: ClassCastException in HistogramNumeric
  • [SPARK-50784]: Fix lint-scala not to ignore scalastyle errors
  • [SPARK-50797]: Move HiveCharVarcharTestSuite from o/a/s/sql to o/a/s/sql/hive
  • [SPARK-50853]: Close temp shuffle file writable channel
  • [SPARK-50870]: DataSourceV2 relations may contain unresolved cast expressions after materializing current_timestamp()
  • [SPARK-50890]: Skip test_take in Spark Connect only build
  • [SPARK-50946]: When using Java version 17.0.14, the SPARK-35907: createDirectory test in UtilsSuite will fail.
  • [SPARK-50984]: Make ExpressionImplUtilsSuite robust by matching JDK msgs via regex
  • [SPARK-50987]: Make spark-connect-overview.md’s version strings up-to-date
  • [SPARK-50998]: Fix spark.kubernetes.configMap.maxSize default value in docs
  • [SPARK-51027]: Prevent HiveClient.runSqlHive invocation in non-testing environment
  • [SPARK-51033]: Fix CryptoStreamUtilsSuite to use sameElements for array comparision
  • [SPARK-51039]: Fix hive-llap-common dependency to use hive.llap.scope in root pom.xml
  • [SPARK-51042]: Read and write CalendarIntervals using one call to get/putLong consistently
  • [SPARK-51061]: Hide Jetty info in REST Submission API
  • [SPARK-51071]: Upgrade upload-artifact to v4 to recover branch-3.5 CI
  • [SPARK-51093]: Fix minor endianness issues in tests
  • [SPARK-51104]: Use self-host javascript and CSS in Spark website
  • [SPARK-51128]: Self host docsearch.min.css.map
  • [SPARK-51129]: Fix code tab switching in Spark Website
  • [SPARK-51157]: Add missing @varargs Scala annotation for scala functon APIs
  • [SPARK-51164]: Fix CallerContext test by enabling hadoop.caller.context.enabled
  • [SPARK-51173]: Add configName Scalastyle rule
  • [SPARK-51185]: Revert simplifications to PartitionedFileUtil API due to increased risk of OOM
  • [SPARK-51187]: Implement the graceful deprecation of incorrect config introduced in SPARK-49699
  • [SPARK-51219]: Catalog.listTables fails on temp table when non-V2SessionCatalog is in use
  • [SPARK-51244]: Upgrade left Github Action image from ubuntu-20.04 to ubuntu-22.04 and solved the TPCDSQueryBenchmark compatibility issue

Dependency changes

While being a maintenance release we did still upgrade some dependencies in this release they are:

You can consult JIRA for the detailed changes.

We would like to acknowledge all community members for contributing patches to this release.


Spark News Archive

Latest News

Archive