org.apache.spark.rdd

Class RDDBarrier<T>

Object

org.apache.spark.rdd.RDDBarrier<T>

public class RDDBarrier<T>
extends Object

:: Experimental :: Wraps an RDD in a barrier stage, which forces Spark to launch tasks of this stage together. RDDBarrier instances are created by RDD.barrier().

Method Summary

Modifier and Type	Method and Description
<S> RDD<S>	mapPartitions(scala.Function1<scala.collection.Iterator<T>,scala.collection.Iterator<S>> f, boolean preservesPartitioning, scala.reflect.ClassTag<S> evidence$2) :: Experimental :: Returns a new RDD by applying a function to each partition of the wrapped RDD, where tasks are launched together in a barrier stage.
<S> RDD<S>	mapPartitionsWithIndex(scala.Function2<Object,scala.collection.Iterator<T>,scala.collection.Iterator<S>> f, boolean preservesPartitioning, scala.reflect.ClassTag<S> evidence$3) :: Experimental :: Returns a new RDD by applying a function to each partition of the wrapped RDD, while tracking the index of the original partition.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

mapPartitions

public <S> RDD<S> mapPartitions(scala.Function1<scala.collection.Iterator<T>,scala.collection.Iterator<S>> f,
                                boolean preservesPartitioning,
                                scala.reflect.ClassTag<S> evidence$2)

:: Experimental :: Returns a new RDD by applying a function to each partition of the wrapped RDD, where tasks are launched together in a barrier stage. The interface is the same as RDD.mapPartitions(scala.Function1<scala.collection.Iterator<T>, scala.collection.Iterator<U>>, boolean, scala.reflect.ClassTag<U>). Please see the API doc there.

Parameters:

f - (undocumented)

preservesPartitioning - (undocumented)

evidence$2 - (undocumented)

Returns:

(undocumented)

See Also:

BarrierTaskContext

mapPartitionsWithIndex

public <S> RDD<S> mapPartitionsWithIndex(scala.Function2<Object,scala.collection.Iterator<T>,scala.collection.Iterator<S>> f,
                                         boolean preservesPartitioning,
                                         scala.reflect.ClassTag<S> evidence$3)

:: Experimental :: Returns a new RDD by applying a function to each partition of the wrapped RDD, while tracking the index of the original partition. And all tasks are launched together in a barrier stage. The interface is the same as RDD.mapPartitionsWithIndex(scala.Function2<java.lang.Object, scala.collection.Iterator<T>, scala.collection.Iterator<U>>, boolean, scala.reflect.ClassTag<U>). Please see the API doc there.

Parameters:

f - (undocumented)

preservesPartitioning - (undocumented)

evidence$3 - (undocumented)

Returns:

(undocumented)

See Also:

BarrierTaskContext