Package org.apache.spark.streaming.util

Class RawTextHelper

Object
org.apache.spark.streaming.util.RawTextHelper
public class RawTextHelper extends Object

  • Constructor Summary

    Constructors
    Constructor
    Description
    RawTextHelper()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    static long
    add(long v1, long v2)
     
    static long
    max(long v1, long v2)
     
    static scala.collection.Iterator<scala.Tuple2<String,Object>>
    splitAndCountPartitions(scala.collection.Iterator<String> iter)
    Splits lines and counts the words.
    static long
    subtract(long v1, long v2)
     
    static scala.collection.Iterator<scala.Tuple2<String,Object>>
    topK(scala.collection.Iterator<scala.Tuple2<String,Object>> data, int k)
    Gets the top k words in terms of word counts.
    static void
    warmUp(SparkContext sc)
    Warms up the SparkContext in master and executor by running tasks to force JIT kick in before real workload starts.

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • RawTextHelper

      public RawTextHelper()

  • Method Details

    • splitAndCountPartitions

      public static scala.collection.Iterator<scala.Tuple2<String,Object>> splitAndCountPartitions(scala.collection.Iterator<String> iter)
      Splits lines and counts the words.
      Parameters:
      iter - (undocumented)
      Returns:
      (undocumented)

    • topK

      public static scala.collection.Iterator<scala.Tuple2<String,Object>> topK(scala.collection.Iterator<scala.Tuple2<String,Object>> data, int k)
      Gets the top k words in terms of word counts. Assumes that each word exists only once in the data iterator (that is, the counts have been reduced).
      Parameters:
      data - (undocumented)
      k - (undocumented)
      Returns:
      (undocumented)

    • warmUp

      public static void warmUp(SparkContext sc)
      Warms up the SparkContext in master and executor by running tasks to force JIT kick in before real workload starts.
      Parameters:
      sc - (undocumented)

    • add

      public static long add(long v1, long v2)

    • subtract

      public static long subtract(long v1, long v2)

    • max

      public static long max(long v1, long v2)