org.apache.spark.streaming.dstream

Class InputDStream<T>

Object

org.apache.spark.streaming.dstream.DStream<T>

org.apache.spark.streaming.dstream.InputDStream<T>

All Implemented Interfaces:

java.io.Serializable, org.apache.spark.internal.Logging

Direct Known Subclasses:

ConstantInputDStream, ReceiverInputDStream

public abstract class InputDStream<T>
extends DStream<T>

This is the abstract base class for all input streams. This class provides methods start() and stop() which are called by Spark Streaming system to start and stop receiving data, respectively. Input streams that can generate RDDs from new data by running a service/thread only on the driver node (that is, without running a receiver on worker nodes), can be implemented by directly inheriting this InputDStream. For example, FileInputDStream, a subclass of InputDStream, monitors an HDFS directory from the driver for new files and generates RDDs with the new files. For implementing input streams that requires running a receiver on the worker nodes, use ReceiverInputDStream as the parent class.

param: _ssc Streaming context that will execute this input stream

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

org.apache.spark.internal.Logging.SparkShellLoggingFilter

Constructor Summary

Constructors

Constructor and Description
InputDStream(StreamingContext _ssc, scala.reflect.ClassTag<T> evidence$1)

Method Summary

Modifier and Type	Method and Description
scala.collection.immutable.List<DStream<?>>	dependencies() List of parent DStreams on which this DStream depends on
int	id() This is a unique identifier for the input stream.
Duration	slideDuration() Time interval after which the DStream generates an RDD
abstract void	start() Method called to start receiving data.
abstract void	stop() Method called to stop receiving data.

Methods inherited from class org.apache.spark.streaming.dstream.DStream

cache, checkpoint, compute, context, count, countByValue, countByValueAndWindow, countByWindow, filter, flatMap, foreachRDD, foreachRDD, glom, map, mapPartitions, persist, persist, print, print, reduce, reduceByWindow, reduceByWindow, repartition, saveAsObjectFiles, saveAsTextFiles, slice, slice, toPairDStreamFunctions, transform, transform, transformWith, transformWith, union, window, window

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.internal.Logging

$init$, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, initLock, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, uninitialize

Constructor Detail

InputDStream

public InputDStream(StreamingContext _ssc,
                    scala.reflect.ClassTag<T> evidence$1)

Method Detail

dependencies

public scala.collection.immutable.List<DStream<?>> dependencies()

Description copied from class: DStream

List of parent DStreams on which this DStream depends on

Specified by:

dependencies in class DStream<T>

id

public int id()

This is a unique identifier for the input stream.

slideDuration

public Duration slideDuration()

Description copied from class: DStream

Time interval after which the DStream generates an RDD

Specified by:

slideDuration in class DStream<T>

start

public abstract void start()

Method called to start receiving data. Subclasses must implement this method.

stop

public abstract void stop()

Method called to stop receiving data. Subclasses must implement this method.