Spark Release 0.6.0

Spark 0.6.0 is a major release that brings several new features, architectural changes, and performance enhancements. The most visible additions are a standalone deploy mode, a Java API, and expanded documentation; but there are also numerous other changes under the hood, which improve performance in some cases by as much as 2x.

You can download this release as either a source package (2 MB tar.gz) or prebuilt package (48 MB tar.gz)

Simpler Deployment

In addition to running on Mesos, Spark now has a standalone deploy mode that lets you quickly launch a cluster without installing an external cluster manager. The standalone mode only needs Java installed on each machine, and Spark deployed to it.

In addition, there is experimental support for running on YARN (Hadoop NextGen), currently in a separate branch.

Java API

Java programmers can now use Spark through a new Java API layer. This layer makes available all of Spark’s features, including parallel transformations, distributed datasets, broadcast variables, and accumulators, in a Java-friendly manner.

Expanded Documentation

Spark’s documentation has been expanded with a new quick start guide, additional deployment instructions, configuration guide, tuning guide, and improved Scaladoc API documentation.

Engine Changes

Under the hood, Spark 0.6 has new, custom storage and communication layers brought in from the upcoming Spark Streaming project. These can improve performance over past versions by as much as 2x. Specifically:

• A new communication manager using asynchronous Java NIO lets shuffle operations run faster, especially when sending large amounts of data or when jobs have many tasks.
• A new storage manager supports per-dataset storage level settings (e.g. whether to keep the dataset in memory, deserialized, on disk, etc, or even replicated across nodes).
• Spark's scheduler and control plane have been optimized to better support ultra-low-latency jobs (under 500ms) and high-throughput scheduling decisions.

New APIs

• This release adds the ability to control caching strategies on a per-RDD level, so that different RDDs may be stored in memory, on disk, as serialized bytes, etc. You can choose your storage level using the persist() method on RDD.
• A new Accumulable class generalizes Accumulators for the case when the type being accumulated is not the same as the types of elements being added (e.g. you wish to accumulate a collection, such as a Set, by adding individual elements).
• You can now dynamically add files or JARs that should be shipped to your workers with SparkContext.addFile/Jar.
• More Spark operators (e.g. joins) support custom partitioners.

Enhanced Debugging

Spark’s log now prints which operation in your program each RDD and job described in your logs belongs to, making it easier to tie back to which parts of your code experience problems.

Maven Artifacts

Spark is now available in Maven Central, making it easier to link into your programs without having to build it as a JAR. Use the following Maven identifiers to add it to a project:

• groupId: org.spark-project
• artifactId: spark-core_2.9.2
• version: 0.6.0

Compatibility

This release is source-compatible with Spark 0.5 programs, but you will need to recompile them against 0.6. In addition, the configuration for caching has changed: instead of having a spark.cache.class parameter that sets one caching strategy for all RDDs, you can now set a per-RDD storage level. Spark will warn if you try to set spark.cache.class.

Credits

Spark 0.6 was the work of a large set of new contributors from Berkeley and outside.

• Tathagata Das contributed the new communication layer, and parts of the storage layer.
• Haoyuan Li contributed the new storage manager.
• Denny Britz contributed the YARN deploy mode, key aspects of the standalone one, and several other features.
• Andy Konwinski contributed the revamped documentation site, Maven publishing, and several API docs.
• Josh Rosen contributed the Java API, and several bug fixes.
• Patrick Wendell contributed the enhanced debugging feature and helped with testing and documentation.
• Reynold Xin contributed numerous bug and performance fixes.
• Imran Rashid contributed the new Accumulable class.
• Harvey Feng contributed improvements to shuffle operations.
• Shivaram Venkataraman improved Spark's memory estimation and wrote a memory tuning guide.
• Ravi Pandya contributed Spark run scripts for Windows.
• Mosharaf Chowdhury provided several fixes to broadcast.
• Henry Milner pointed out several bugs in sampling algorithms.
• Ray Racine provided improvements to the EC2 scripts.
• Paul Ruan and Bill Zhao helped with testing.

Thanks also to all the Spark users who have diligently suggested features or reported bugs.

Spark News Archive