org.apache.spark.streaming.api.java

Class JavaDStream<T>

Object

org.apache.spark.streaming.api.java.JavaDStream<T>

All Implemented Interfaces:

java.io.Serializable, JavaDStreamLike<T,JavaDStream<T>,JavaRDD<T>>

Direct Known Subclasses:

JavaInputDStream, JavaMapWithStateDStream

public class JavaDStream<T>
extends Object

A Java-friendly interface to DStream, the basic abstraction in Spark Streaming that represents a continuous stream of data. DStreams can either be created from live data (such as, data from TCP sockets, Kafka, etc.) or it can be generated by transforming existing DStreams using operations such as map, window. For operations applicable to key-value pair DStreams, see JavaPairDStream.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
JavaDStream(DStream<T> dstream, scala.reflect.ClassTag<T> classTag)

Method Summary

Modifier and Type	Method and Description
JavaDStream<T>	cache() Persist RDDs of this DStream with the default storage level (MEMORY_ONLY_SER)
scala.reflect.ClassTag<T>	classTag()
JavaRDD<T>	compute(Time validTime) Generate an RDD for the given duration
DStream<T>	dstream()
JavaDStream<T>	filter(Function<T,Boolean> f) Return a new DStream containing only the elements that satisfy a predicate.
static <T> JavaDStream<T>	fromDStream(DStream<T> dstream, scala.reflect.ClassTag<T> evidence$1) Convert a scala DStream to a Java-friendly JavaDStream.
JavaDStream<T>	persist() Persist RDDs of this DStream with the default storage level (MEMORY_ONLY_SER)
JavaDStream<T>	persist(StorageLevel storageLevel) Persist the RDDs of this DStream with the given storage level
JavaDStream<T>	repartition(int numPartitions) Return a new DStream with an increased or decreased level of parallelism.
JavaDStream<T>	union(JavaDStream<T> that) Return a new DStream by unifying data of another DStream with this DStream.
JavaDStream<T>	window(Duration windowDuration) Return a new DStream in which each RDD contains all the elements in seen in a sliding window of time over this DStream.
JavaDStream<T>	window(Duration windowDuration, Duration slideDuration) Return a new DStream in which each RDD contains all the elements in seen in a sliding window of time over this DStream.
JavaRDD<T>	wrapRDD(RDD<T> rdd)

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.streaming.api.java.JavaDStreamLike

checkpoint, context, count, countByValue, countByValue, countByValueAndWindow, countByValueAndWindow, countByWindow, flatMap, flatMapToPair, foreachRDD, foreachRDD, glom, map, mapPartitions, mapPartitionsToPair, mapToPair, print, print, reduce, reduceByWindow, reduceByWindow, scalaIntToJavaLong, slice, transform, transform, transformToPair, transformToPair, transformWith, transformWith, transformWithToPair, transformWithToPair

Constructor Detail

JavaDStream

public JavaDStream(DStream<T> dstream,
                   scala.reflect.ClassTag<T> classTag)

Method Detail

fromDStream

public static <T> JavaDStream<T> fromDStream(DStream<T> dstream,
                                             scala.reflect.ClassTag<T> evidence$1)

Convert a scala DStream to a Java-friendly JavaDStream.

Parameters:

dstream - (undocumented)

evidence$1 - (undocumented)

Returns:

(undocumented)

dstream

public DStream<T> dstream()

classTag

public scala.reflect.ClassTag<T> classTag()

wrapRDD

public JavaRDD<T> wrapRDD(RDD<T> rdd)

filter

public JavaDStream<T> filter(Function<T,Boolean> f)

Return a new DStream containing only the elements that satisfy a predicate.

cache

public JavaDStream<T> cache()

Persist RDDs of this DStream with the default storage level (MEMORY_ONLY_SER)

persist

public JavaDStream<T> persist()

Persist RDDs of this DStream with the default storage level (MEMORY_ONLY_SER)

persist

public JavaDStream<T> persist(StorageLevel storageLevel)

Persist the RDDs of this DStream with the given storage level

compute

public JavaRDD<T> compute(Time validTime)

Generate an RDD for the given duration

window

public JavaDStream<T> window(Duration windowDuration)

Return a new DStream in which each RDD contains all the elements in seen in a sliding window of time over this DStream. The new DStream generates RDDs with the same interval as this DStream.

Parameters:

windowDuration - width of the window; must be a multiple of this DStream's interval.

Returns:

(undocumented)

window

public JavaDStream<T> window(Duration windowDuration,
                             Duration slideDuration)

Return a new DStream in which each RDD contains all the elements in seen in a sliding window of time over this DStream.

Parameters:

windowDuration - width of the window; must be a multiple of this DStream's batching interval

slideDuration - sliding interval of the window (i.e., the interval after which the new DStream will generate RDDs); must be a multiple of this DStream's batching interval

Returns:

(undocumented)

union

public JavaDStream<T> union(JavaDStream<T> that)

Return a new DStream by unifying data of another DStream with this DStream.

Parameters:

that - Another DStream having the same interval (i.e., slideDuration) as this DStream.

Returns:

(undocumented)

repartition

public JavaDStream<T> repartition(int numPartitions)

Return a new DStream with an increased or decreased level of parallelism. Each RDD in the returned DStream has exactly numPartitions partitions.

Parameters:

numPartitions - (undocumented)

Returns:

(undocumented)