org.apache.spark.scheduler

Class StatsReportListener

Object

org.apache.spark.scheduler.SparkListener

org.apache.spark.scheduler.StatsReportListener

All Implemented Interfaces:

org.apache.spark.internal.Logging, SparkListenerInterface

public class StatsReportListener
extends SparkListener
implements org.apache.spark.internal.Logging

:: DeveloperApi :: Simple SparkListener that logs a few summary statistics when each stage completes.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

org.apache.spark.internal.Logging.SparkShellLoggingFilter

Constructor Summary

Constructors

Constructor and Description
StatsReportListener()

Method Summary

Modifier and Type	Method and Description
static scala.Option<org.apache.spark.util.Distribution>	extractDoubleDistribution(scala.collection.Seq<scala.Tuple2<TaskInfo,org.apache.spark.executor.TaskMetrics>> taskInfoMetrics, scala.Function2<TaskInfo,org.apache.spark.executor.TaskMetrics,Object> getMetric)
static scala.Option<org.apache.spark.util.Distribution>	extractLongDistribution(scala.collection.Seq<scala.Tuple2<TaskInfo,org.apache.spark.executor.TaskMetrics>> taskInfoMetrics, scala.Function2<TaskInfo,org.apache.spark.executor.TaskMetrics,Object> getMetric)
static long	hours()
static String	millisToString(long ms) Reformat a time interval in milliseconds to a prettier format for output
static long	minutes()
void	onStageCompleted(SparkListenerStageCompleted stageCompleted) Called when a stage completes successfully or fails, with information on the completed stage.
void	onTaskEnd(SparkListenerTaskEnd taskEnd) Called when a task ends
static void	org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)
static org.slf4j.Logger	org$apache$spark$internal$Logging$$log_()
static int[]	percentiles()
static String	percentilesHeader()
static double[]	probabilities()
static long	seconds()
static void	showBytesDistribution(String heading, scala.Function2<TaskInfo,org.apache.spark.executor.TaskMetrics,Object> getMetric, scala.collection.Seq<scala.Tuple2<TaskInfo,org.apache.spark.executor.TaskMetrics>> taskInfoMetrics)
static void	showBytesDistribution(String heading, scala.Option<org.apache.spark.util.Distribution> dOpt)
static void	showBytesDistribution(String heading, org.apache.spark.util.Distribution dist)
static void	showDistribution(String heading, scala.Option<org.apache.spark.util.Distribution> dOpt, scala.Function1<Object,String> formatNumber)
static void	showDistribution(String heading, scala.Option<org.apache.spark.util.Distribution> dOpt, String format)
static void	showDistribution(String heading, org.apache.spark.util.Distribution d, scala.Function1<Object,String> formatNumber)
static void	showDistribution(String heading, String format, scala.Function2<TaskInfo,org.apache.spark.executor.TaskMetrics,Object> getMetric, scala.collection.Seq<scala.Tuple2<TaskInfo,org.apache.spark.executor.TaskMetrics>> taskInfoMetrics)
static void	showMillisDistribution(String heading, scala.Function2<TaskInfo,org.apache.spark.executor.TaskMetrics,Object> getMetric, scala.collection.Seq<scala.Tuple2<TaskInfo,org.apache.spark.executor.TaskMetrics>> taskInfoMetrics)
static void	showMillisDistribution(String heading, scala.Option<org.apache.spark.util.Distribution> dOpt)

Methods inherited from class org.apache.spark.scheduler.SparkListener

onApplicationEnd, onApplicationStart, onBlockManagerAdded, onBlockManagerRemoved, onBlockUpdated, onEnvironmentUpdate, onExecutorAdded, onExecutorBlacklisted, onExecutorBlacklistedForStage, onExecutorExcluded, onExecutorExcludedForStage, onExecutorMetricsUpdate, onExecutorRemoved, onExecutorUnblacklisted, onExecutorUnexcluded, onJobEnd, onJobStart, onNodeBlacklisted, onNodeBlacklistedForStage, onNodeExcluded, onNodeExcludedForStage, onNodeUnblacklisted, onNodeUnexcluded, onOtherEvent, onResourceProfileAdded, onSpeculativeTaskSubmitted, onStageExecutorMetrics, onStageSubmitted, onTaskGettingResult, onTaskStart, onUnpersistRDD, onUnschedulableTaskSetAdded, onUnschedulableTaskSetRemoved

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.internal.Logging

$init$, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, initLock, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, uninitialize

Constructor Detail

StatsReportListener

public StatsReportListener()

Method Detail

percentiles

public static int[] percentiles()

probabilities

public static double[] probabilities()

percentilesHeader

public static String percentilesHeader()

extractDoubleDistribution

public static scala.Option<org.apache.spark.util.Distribution> extractDoubleDistribution(scala.collection.Seq<scala.Tuple2<TaskInfo,org.apache.spark.executor.TaskMetrics>> taskInfoMetrics,
                                                                                         scala.Function2<TaskInfo,org.apache.spark.executor.TaskMetrics,Object> getMetric)

extractLongDistribution

public static scala.Option<org.apache.spark.util.Distribution> extractLongDistribution(scala.collection.Seq<scala.Tuple2<TaskInfo,org.apache.spark.executor.TaskMetrics>> taskInfoMetrics,
                                                                                       scala.Function2<TaskInfo,org.apache.spark.executor.TaskMetrics,Object> getMetric)

showDistribution

public static void showDistribution(String heading,
                                    org.apache.spark.util.Distribution d,
                                    scala.Function1<Object,String> formatNumber)

showDistribution

public static void showDistribution(String heading,
                                    scala.Option<org.apache.spark.util.Distribution> dOpt,
                                    scala.Function1<Object,String> formatNumber)

showDistribution

public static void showDistribution(String heading,
                                    scala.Option<org.apache.spark.util.Distribution> dOpt,
                                    String format)

showDistribution

public static void showDistribution(String heading,
                                    String format,
                                    scala.Function2<TaskInfo,org.apache.spark.executor.TaskMetrics,Object> getMetric,
                                    scala.collection.Seq<scala.Tuple2<TaskInfo,org.apache.spark.executor.TaskMetrics>> taskInfoMetrics)

showBytesDistribution

public static void showBytesDistribution(String heading,
                                         scala.Function2<TaskInfo,org.apache.spark.executor.TaskMetrics,Object> getMetric,
                                         scala.collection.Seq<scala.Tuple2<TaskInfo,org.apache.spark.executor.TaskMetrics>> taskInfoMetrics)

showBytesDistribution

public static void showBytesDistribution(String heading,
                                         scala.Option<org.apache.spark.util.Distribution> dOpt)

showBytesDistribution

public static void showBytesDistribution(String heading,
                                         org.apache.spark.util.Distribution dist)

showMillisDistribution

public static void showMillisDistribution(String heading,
                                          scala.Option<org.apache.spark.util.Distribution> dOpt)

showMillisDistribution

public static void showMillisDistribution(String heading,
                                          scala.Function2<TaskInfo,org.apache.spark.executor.TaskMetrics,Object> getMetric,
                                          scala.collection.Seq<scala.Tuple2<TaskInfo,org.apache.spark.executor.TaskMetrics>> taskInfoMetrics)

seconds

public static long seconds()

minutes

public static long minutes()

hours

public static long hours()

millisToString

public static String millisToString(long ms)

Reformat a time interval in milliseconds to a prettier format for output

Parameters:

ms - (undocumented)

Returns:

(undocumented)

org$apache$spark$internal$Logging$$log_

public static org.slf4j.Logger org$apache$spark$internal$Logging$$log_()

org$apache$spark$internal$Logging$$log__$eq

public static void org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)

onTaskEnd

public void onTaskEnd(SparkListenerTaskEnd taskEnd)

Description copied from interface: SparkListenerInterface

Called when a task ends

Specified by:

onTaskEnd in interface SparkListenerInterface

Overrides:

onTaskEnd in class SparkListener

Parameters:

taskEnd - (undocumented)

onStageCompleted

public void onStageCompleted(SparkListenerStageCompleted stageCompleted)

Description copied from interface: SparkListenerInterface

Called when a stage completes successfully or fails, with information on the completed stage.

Specified by:

onStageCompleted in interface SparkListenerInterface

Overrides:

onStageCompleted in class SparkListener

Parameters:

stageCompleted - (undocumented)