JavaSparkContext

Value Members

1. final def !=(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

2. final def !=(arg0: Any): Boolean

   Definition Classes

   Any

3. final def ##(): Int

   Definition Classes

   AnyRef → Any

4. final def ==(arg0: AnyRef): Boolean

   Definition Classes

   AnyRef

5. final def ==(arg0: Any): Boolean

   Definition Classes

   Any

6. def accumulator[T](initialValue: T, accumulatorParam: AccumulatorParam[T]): Accumulator[T]

   Create an Accumulator variable of a given type, which tasks can "add" values to using the += method.

   Create an Accumulator variable of a given type, which tasks can "add" values to using the += method. Only the master can access the accumulator's value.

7. final def asInstanceOf[T0]: T0

   Definition Classes

   Any

8. def broadcast[T](value: T): Broadcast[T]

   Broadcast a read-only variable to the cluster, returning a spark.Broadcast object for reading it in distributed functions.

   Broadcast a read-only variable to the cluster, returning a spark.Broadcast object for reading it in distributed functions. The variable will be sent to each cluster only once.

9. def clone(): AnyRef

   Attributes

   protected[lang]

   Definition Classes

   AnyRef

   Annotations

   @throws()

10. def doubleAccumulator(initialValue: Double): Accumulator[Double]

    Create an Accumulator double variable, which tasks can "add" values to using the += method.

    Create an Accumulator double variable, which tasks can "add" values to using the += method. Only the master can access the accumulator's value.

11. final def eq(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

12. def equals(arg0: Any): Boolean

    Definition Classes

    AnyRef → Any

13. def finalize(): Unit

    Attributes

    protected[lang]

    Definition Classes

    AnyRef

    Annotations

    @throws()

14. final def getClass(): java.lang.Class[_]

    Definition Classes

    AnyRef → Any

15. def getSparkHome(): Option[String]

    Get Spark's home location from either a value set through the constructor, or the spark.

    Get Spark's home location from either a value set through the constructor, or the spark.home Java property, or the SPARK_HOME environment variable (in that order of preference). If neither of these is set, return None.

16. def hadoopFile[K, V, F <: InputFormat[K, V]](path: String, inputFormatClass: Class[F], keyClass: Class[K], valueClass: Class[V]): JavaPairRDD[K, V]

    Get an RDD for a Hadoop file with an arbitrary InputFormat

17. def hadoopFile[K, V, F <: InputFormat[K, V]](path: String, inputFormatClass: Class[F], keyClass: Class[K], valueClass: Class[V], minSplits: Int): JavaPairRDD[K, V]

    Get an RDD for a Hadoop file with an arbitrary InputFormat

18. def hadoopRDD[K, V, F <: InputFormat[K, V]](conf: JobConf, inputFormatClass: Class[F], keyClass: Class[K], valueClass: Class[V]): JavaPairRDD[K, V]

    Get an RDD for a Hadoop-readable dataset from a Hadooop JobConf giving its InputFormat and any other necessary info (e.

    Get an RDD for a Hadoop-readable dataset from a Hadooop JobConf giving its InputFormat and any other necessary info (e.g. file name for a filesystem-based dataset, table name for HyperTable, etc).

19. def hadoopRDD[K, V, F <: InputFormat[K, V]](conf: JobConf, inputFormatClass: Class[F], keyClass: Class[K], valueClass: Class[V], minSplits: Int): JavaPairRDD[K, V]

    Get an RDD for a Hadoop-readable dataset from a Hadooop JobConf giving its InputFormat and any other necessary info (e.

    Get an RDD for a Hadoop-readable dataset from a Hadooop JobConf giving its InputFormat and any other necessary info (e.g. file name for a filesystem-based dataset, table name for HyperTable, etc).

20. def hashCode(): Int

    Definition Classes

    AnyRef → Any

21. def intAccumulator(initialValue: Int): Accumulator[Int]

    Create an Accumulator integer variable, which tasks can "add" values to using the += method.

    Create an Accumulator integer variable, which tasks can "add" values to using the += method. Only the master can access the accumulator's value.

22. final def isInstanceOf[T0]: Boolean

    Definition Classes

    Any

23. final def ne(arg0: AnyRef): Boolean

    Definition Classes

    AnyRef

24. def newAPIHadoopFile[K, V, F <: InputFormat[K, V]](path: String, fClass: Class[F], kClass: Class[K], vClass: Class[V], conf: Configuration): JavaPairRDD[K, V]

    Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.

25. def newAPIHadoopRDD[K, V, F <: InputFormat[K, V]](conf: Configuration, fClass: Class[F], kClass: Class[K], vClass: Class[V]): JavaPairRDD[K, V]

    Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.

26. final def notify(): Unit

    Definition Classes

    AnyRef

27. final def notifyAll(): Unit

    Definition Classes

    AnyRef

28. def objectFile[T](path: String): JavaRDD[T]

    Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition.

    Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition. This is still an experimental storage format and may not be supported exactly as is in future Spark releases. It will also be pretty slow if you use the default serializer (Java serialization), though the nice thing about it is that there's very little effort required to save arbitrary objects.

29. def objectFile[T](path: String, minSplits: Int): JavaRDD[T]

    Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition.

    Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition. This is still an experimental storage format and may not be supported exactly as is in future Spark releases. It will also be pretty slow if you use the default serializer (Java serialization), though the nice thing about it is that there's very little effort required to save arbitrary objects.

30. def parallelize[T](list: List[T]): JavaRDD[T]

    Distribute a local Scala collection to form an RDD.

31. def parallelize[T](list: List[T], numSlices: Int): JavaRDD[T]

    Distribute a local Scala collection to form an RDD.

32. def parallelizeDoubles(list: List[Double]): JavaDoubleRDD

    Distribute a local Scala collection to form an RDD.

33. def parallelizeDoubles(list: List[Double], numSlices: Int): JavaDoubleRDD

    Distribute a local Scala collection to form an RDD.

34. def parallelizePairs[K, V](list: List[(K, V)]): JavaPairRDD[K, V]

    Distribute a local Scala collection to form an RDD.

35. def parallelizePairs[K, V](list: List[(K, V)], numSlices: Int): JavaPairRDD[K, V]

    Distribute a local Scala collection to form an RDD.

36. val sc: SparkContext

37. def sequenceFile[K, V](path: String, keyClass: Class[K], valueClass: Class[V]): JavaPairRDD[K, V]

    Get an RDD for a Hadoop SequenceFile.

38. def sequenceFile[K, V](path: String, keyClass: Class[K], valueClass: Class[V], minSplits: Int): JavaPairRDD[K, V]

    Get an RDD for a Hadoop SequenceFile with given key and value types.

39. def stop(): Unit

    Shut down the SparkContext.

40. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes

    AnyRef

41. def textFile(path: String, minSplits: Int): JavaRDD[String]

    Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings.

42. def textFile(path: String): JavaRDD[String]

    Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings.

43. def toString(): String

    Definition Classes

    AnyRef → Any

44. def union(first: JavaDoubleRDD, rest: List[JavaDoubleRDD]): JavaDoubleRDD

    Build the union of two or more RDDs.

    Build the union of two or more RDDs.

    Definition Classes

    JavaSparkContext → JavaSparkContextVarargsWorkaround

45. def union[K, V](first: JavaPairRDD[K, V], rest: List[JavaPairRDD[K, V]]): JavaPairRDD[K, V]

    Build the union of two or more RDDs.

    Build the union of two or more RDDs.

    Definition Classes

    JavaSparkContext → JavaSparkContextVarargsWorkaround

46. def union[T](first: JavaRDD[T], rest: List[JavaRDD[T]]): JavaRDD[T]

    Build the union of two or more RDDs.

    Build the union of two or more RDDs.

    Definition Classes

    JavaSparkContext → JavaSparkContextVarargsWorkaround

47. def union[K, V](rdds: <repeated...>[JavaPairRDD[K, V]]): JavaPairRDD[K, V]

    Definition Classes

    JavaSparkContextVarargsWorkaround

48. def union(rdds: <repeated...>[JavaDoubleRDD]): JavaDoubleRDD

    Definition Classes

    JavaSparkContextVarargsWorkaround

49. def union[T](rdds: <repeated...>[JavaRDD[T]]): JavaRDD[T]

    Definition Classes

    JavaSparkContextVarargsWorkaround

50. final def wait(): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws()

51. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws()

52. final def wait(arg0: Long): Unit

    Definition Classes

    AnyRef

    Annotations

    @throws()