Class StatsReportListener

Object
org.apache.spark.scheduler.SparkListener
org.apache.spark.scheduler.StatsReportListener
All Implemented Interfaces:
org.apache.spark.internal.Logging, SparkListenerInterface

public class StatsReportListener extends SparkListener implements org.apache.spark.internal.Logging
:: DeveloperApi :: Simple SparkListener that logs a few summary statistics when each stage completes.
  • Constructor Details

    • StatsReportListener

      public StatsReportListener()
  • Method Details

    • percentiles

      public static int[] percentiles()
    • probabilities

      public static double[] probabilities()
    • percentilesHeader

      public static String percentilesHeader()
    • extractDoubleDistribution

      public static scala.Option<org.apache.spark.util.Distribution> extractDoubleDistribution(scala.collection.Seq<scala.Tuple2<TaskInfo,org.apache.spark.executor.TaskMetrics>> taskInfoMetrics, scala.Function2<TaskInfo,org.apache.spark.executor.TaskMetrics,Object> getMetric)
    • extractLongDistribution

      public static scala.Option<org.apache.spark.util.Distribution> extractLongDistribution(scala.collection.Seq<scala.Tuple2<TaskInfo,org.apache.spark.executor.TaskMetrics>> taskInfoMetrics, scala.Function2<TaskInfo,org.apache.spark.executor.TaskMetrics,Object> getMetric)
    • showDistribution

      public static void showDistribution(String heading, org.apache.spark.util.Distribution d, scala.Function1<Object,String> formatNumber)
    • showDistribution

      public static void showDistribution(String heading, scala.Option<org.apache.spark.util.Distribution> dOpt, scala.Function1<Object,String> formatNumber)
    • showDistribution

      public static void showDistribution(String heading, scala.Option<org.apache.spark.util.Distribution> dOpt, String format)
    • showDistribution

      public static void showDistribution(String heading, String format, scala.Function2<TaskInfo,org.apache.spark.executor.TaskMetrics,Object> getMetric, scala.collection.Seq<scala.Tuple2<TaskInfo,org.apache.spark.executor.TaskMetrics>> taskInfoMetrics)
    • showBytesDistribution

      public static void showBytesDistribution(String heading, scala.Function2<TaskInfo,org.apache.spark.executor.TaskMetrics,Object> getMetric, scala.collection.Seq<scala.Tuple2<TaskInfo,org.apache.spark.executor.TaskMetrics>> taskInfoMetrics)
    • showBytesDistribution

      public static void showBytesDistribution(String heading, scala.Option<org.apache.spark.util.Distribution> dOpt)
    • showBytesDistribution

      public static void showBytesDistribution(String heading, org.apache.spark.util.Distribution dist)
    • showMillisDistribution

      public static void showMillisDistribution(String heading, scala.Option<org.apache.spark.util.Distribution> dOpt)
    • showMillisDistribution

      public static void showMillisDistribution(String heading, scala.Function2<TaskInfo,org.apache.spark.executor.TaskMetrics,Object> getMetric, scala.collection.Seq<scala.Tuple2<TaskInfo,org.apache.spark.executor.TaskMetrics>> taskInfoMetrics)
    • seconds

      public static long seconds()
    • minutes

      public static long minutes()
    • hours

      public static long hours()
    • millisToString

      public static String millisToString(long ms)
      Reformat a time interval in milliseconds to a prettier format for output
      Parameters:
      ms - (undocumented)
      Returns:
      (undocumented)
    • org$apache$spark$internal$Logging$$log_

      public static org.slf4j.Logger org$apache$spark$internal$Logging$$log_()
    • org$apache$spark$internal$Logging$$log__$eq

      public static void org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)
    • onTaskEnd

      public void onTaskEnd(SparkListenerTaskEnd taskEnd)
      Description copied from interface: SparkListenerInterface
      Called when a task ends
      Specified by:
      onTaskEnd in interface SparkListenerInterface
      Overrides:
      onTaskEnd in class SparkListener
      Parameters:
      taskEnd - (undocumented)
    • onStageCompleted

      public void onStageCompleted(SparkListenerStageCompleted stageCompleted)
      Description copied from interface: SparkListenerInterface
      Called when a stage completes successfully or fails, with information on the completed stage.
      Specified by:
      onStageCompleted in interface SparkListenerInterface
      Overrides:
      onStageCompleted in class SparkListener
      Parameters:
      stageCompleted - (undocumented)