Class JavaRDD<T>

Object
org.apache.spark.api.java.JavaRDD<T>
All Implemented Interfaces:
Serializable, JavaRDDLike<T,JavaRDD<T>>, scala.Serializable

public class JavaRDD<T> extends Object
See Also:
  • Constructor Details

    • JavaRDD

      public JavaRDD(RDD<T> rdd, scala.reflect.ClassTag<T> classTag)
  • Method Details

    • fromRDD

      public static <T> JavaRDD<T> fromRDD(RDD<T> rdd, scala.reflect.ClassTag<T> evidence$1)
    • toRDD

      public static <T> RDD<T> toRDD(JavaRDD<T> rdd)
    • rdd

      public RDD<T> rdd()
    • classTag

      public scala.reflect.ClassTag<T> classTag()
    • wrapRDD

      public JavaRDD<T> wrapRDD(RDD<T> rdd)
    • cache

      public JavaRDD<T> cache()
      Persist this RDD with the default storage level (MEMORY_ONLY).
      Returns:
      (undocumented)
    • persist

      public JavaRDD<T> persist(StorageLevel newLevel)
      Set this RDD's storage level to persist its values across operations after the first time it is computed. This can only be used to assign a new storage level if the RDD does not have a storage level set yet..
      Parameters:
      newLevel - (undocumented)
      Returns:
      (undocumented)
    • withResources

      public JavaRDD<T> withResources(ResourceProfile rp)
      Specify a ResourceProfile to use when calculating this RDD. This is only supported on certain cluster managers and currently requires dynamic allocation to be enabled. It will result in new executors with the resources specified being acquired to calculate the RDD.
      Parameters:
      rp - (undocumented)
      Returns:
      (undocumented)
    • getResourceProfile

      public ResourceProfile getResourceProfile()
      Get the ResourceProfile specified with this RDD or None if it wasn't specified.
      Returns:
      the user specified ResourceProfile or null if none was specified
    • unpersist

      public JavaRDD<T> unpersist()
      Mark the RDD as non-persistent, and remove all blocks for it from memory and disk. This method blocks until all blocks are deleted.
      Returns:
      (undocumented)
    • unpersist

      public JavaRDD<T> unpersist(boolean blocking)
      Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.

      Parameters:
      blocking - Whether to block until all blocks are deleted.
      Returns:
      (undocumented)
    • distinct

      public JavaRDD<T> distinct()
      Return a new RDD containing the distinct elements in this RDD.
      Returns:
      (undocumented)
    • distinct

      public JavaRDD<T> distinct(int numPartitions)
      Return a new RDD containing the distinct elements in this RDD.
      Parameters:
      numPartitions - (undocumented)
      Returns:
      (undocumented)
    • filter

      public JavaRDD<T> filter(Function<T,Boolean> f)
      Return a new RDD containing only the elements that satisfy a predicate.
      Parameters:
      f - (undocumented)
      Returns:
      (undocumented)
    • coalesce

      public JavaRDD<T> coalesce(int numPartitions)
      Return a new RDD that is reduced into numPartitions partitions.
      Parameters:
      numPartitions - (undocumented)
      Returns:
      (undocumented)
    • coalesce

      public JavaRDD<T> coalesce(int numPartitions, boolean shuffle)
      Return a new RDD that is reduced into numPartitions partitions.
      Parameters:
      numPartitions - (undocumented)
      shuffle - (undocumented)
      Returns:
      (undocumented)
    • repartition

      public JavaRDD<T> repartition(int numPartitions)
      Return a new RDD that has exactly numPartitions partitions.

      Can increase or decrease the level of parallelism in this RDD. Internally, this uses a shuffle to redistribute data.

      If you are decreasing the number of partitions in this RDD, consider using coalesce, which can avoid performing a shuffle.

      Parameters:
      numPartitions - (undocumented)
      Returns:
      (undocumented)
    • sample

      public JavaRDD<T> sample(boolean withReplacement, double fraction)
      Return a sampled subset of this RDD with a random seed.

      Parameters:
      withReplacement - can elements be sampled multiple times (replaced when sampled out)
      fraction - expected size of the sample as a fraction of this RDD's size without replacement: probability that each element is chosen; fraction must be [0, 1] with replacement: expected number of times each element is chosen; fraction must be greater than or equal to 0

      Returns:
      (undocumented)
      Note:
      This is NOT guaranteed to provide exactly the fraction of the count of the given RDD.
    • sample

      public JavaRDD<T> sample(boolean withReplacement, double fraction, long seed)
      Return a sampled subset of this RDD, with a user-supplied seed.

      Parameters:
      withReplacement - can elements be sampled multiple times (replaced when sampled out)
      fraction - expected size of the sample as a fraction of this RDD's size without replacement: probability that each element is chosen; fraction must be [0, 1] with replacement: expected number of times each element is chosen; fraction must be greater than or equal to 0
      seed - seed for the random number generator

      Returns:
      (undocumented)
      Note:
      This is NOT guaranteed to provide exactly the fraction of the count of the given RDD.
    • randomSplit

      public JavaRDD<T>[] randomSplit(double[] weights)
      Randomly splits this RDD with the provided weights.

      Parameters:
      weights - weights for splits, will be normalized if they don't sum to 1

      Returns:
      split RDDs in an array
    • randomSplit

      public JavaRDD<T>[] randomSplit(double[] weights, long seed)
      Randomly splits this RDD with the provided weights.

      Parameters:
      weights - weights for splits, will be normalized if they don't sum to 1
      seed - random seed

      Returns:
      split RDDs in an array
    • union

      public JavaRDD<T> union(JavaRDD<T> other)
      Return the union of this RDD and another one. Any identical elements will appear multiple times (use .distinct() to eliminate them).
      Parameters:
      other - (undocumented)
      Returns:
      (undocumented)
    • intersection

      public JavaRDD<T> intersection(JavaRDD<T> other)
      Return the intersection of this RDD and another one. The output will not contain any duplicate elements, even if the input RDDs did.

      Parameters:
      other - (undocumented)
      Returns:
      (undocumented)
      Note:
      This method performs a shuffle internally.
    • subtract

      public JavaRDD<T> subtract(JavaRDD<T> other)
      Return an RDD with the elements from this that are not in other.

      Uses this partitioner/partition size, because even if other is huge, the resulting RDD will be less than or equal to us.

      Parameters:
      other - (undocumented)
      Returns:
      (undocumented)
    • subtract

      public JavaRDD<T> subtract(JavaRDD<T> other, int numPartitions)
      Return an RDD with the elements from this that are not in other.
      Parameters:
      other - (undocumented)
      numPartitions - (undocumented)
      Returns:
      (undocumented)
    • subtract

      public JavaRDD<T> subtract(JavaRDD<T> other, Partitioner p)
      Return an RDD with the elements from this that are not in other.
      Parameters:
      other - (undocumented)
      p - (undocumented)
      Returns:
      (undocumented)
    • toString

      public String toString()
      Overrides:
      toString in class Object
    • setName

      public JavaRDD<T> setName(String name)
      Assign a name to this RDD
    • sortBy

      public <S> JavaRDD<T> sortBy(Function<T,S> f, boolean ascending, int numPartitions)
      Return this RDD sorted by the given key function.
      Parameters:
      f - (undocumented)
      ascending - (undocumented)
      numPartitions - (undocumented)
      Returns:
      (undocumented)