org.apache.spark.scheduler

Class SparkListener

Object

org.apache.spark.scheduler.SparkListener

All Implemented Interfaces:

SparkListenerInterface

Direct Known Subclasses:

SpillListener, StatsReportListener

public abstract class SparkListener
extends Object
implements SparkListenerInterface

:: DeveloperApi :: A default implementation for SparkListenerInterface that has no-op implementations for all callbacks.

Note that this is an internal interface which might change in different Spark releases.

Constructor Summary

Constructors

Constructor and Description
SparkListener()

Method Summary

Modifier and Type	Method and Description
void	onApplicationEnd(SparkListenerApplicationEnd applicationEnd) Called when the application ends
void	onApplicationStart(SparkListenerApplicationStart applicationStart) Called when the application starts
void	onBlockManagerAdded(SparkListenerBlockManagerAdded blockManagerAdded) Called when a new block manager has joined
void	onBlockManagerRemoved(SparkListenerBlockManagerRemoved blockManagerRemoved) Called when an existing block manager has been removed
void	onBlockUpdated(SparkListenerBlockUpdated blockUpdated) Called when the driver receives a block update info.
void	onEnvironmentUpdate(SparkListenerEnvironmentUpdate environmentUpdate) Called when environment properties have been updated
void	onExecutorAdded(SparkListenerExecutorAdded executorAdded) Called when the driver registers a new executor.
void	onExecutorBlacklisted(SparkListenerExecutorBlacklisted executorBlacklisted) Called when the driver excludes an executor for a Spark application.
void	onExecutorBlacklistedForStage(SparkListenerExecutorBlacklistedForStage executorBlacklistedForStage) Called when the driver excludes an executor for a stage.
void	onExecutorExcluded(SparkListenerExecutorExcluded executorExcluded) Called when the driver excludes an executor for a Spark application.
void	onExecutorExcludedForStage(SparkListenerExecutorExcludedForStage executorExcludedForStage) Called when the driver excludes an executor for a stage.
void	onExecutorMetricsUpdate(SparkListenerExecutorMetricsUpdate executorMetricsUpdate) Called when the driver receives task metrics from an executor in a heartbeat.
void	onExecutorRemoved(SparkListenerExecutorRemoved executorRemoved) Called when the driver removes an executor.
void	onExecutorUnblacklisted(SparkListenerExecutorUnblacklisted executorUnblacklisted) Called when the driver re-enables a previously excluded executor.
void	onExecutorUnexcluded(SparkListenerExecutorUnexcluded executorUnexcluded) Called when the driver re-enables a previously excluded executor.
void	onJobEnd(SparkListenerJobEnd jobEnd) Called when a job ends
void	onJobStart(SparkListenerJobStart jobStart) Called when a job starts
void	onNodeBlacklisted(SparkListenerNodeBlacklisted nodeBlacklisted) Called when the driver excludes a node for a Spark application.
void	onNodeBlacklistedForStage(SparkListenerNodeBlacklistedForStage nodeBlacklistedForStage) Called when the driver excludes a node for a stage.
void	onNodeExcluded(SparkListenerNodeExcluded nodeExcluded) Called when the driver excludes a node for a Spark application.
void	onNodeExcludedForStage(SparkListenerNodeExcludedForStage nodeExcludedForStage) Called when the driver excludes a node for a stage.
void	onNodeUnblacklisted(SparkListenerNodeUnblacklisted nodeUnblacklisted) Called when the driver re-enables a previously excluded node.
void	onNodeUnexcluded(SparkListenerNodeUnexcluded nodeUnexcluded) Called when the driver re-enables a previously excluded node.
void	onOtherEvent(SparkListenerEvent event) Called when other events like SQL-specific events are posted.
void	onResourceProfileAdded(SparkListenerResourceProfileAdded event) Called when a Resource Profile is added to the manager.
void	onSpeculativeTaskSubmitted(SparkListenerSpeculativeTaskSubmitted speculativeTask) Called when a speculative task is submitted
void	onStageCompleted(SparkListenerStageCompleted stageCompleted) Called when a stage completes successfully or fails, with information on the completed stage.
void	onStageExecutorMetrics(SparkListenerStageExecutorMetrics executorMetrics) Called with the peak memory metrics for a given (executor, stage) combination.
void	onStageSubmitted(SparkListenerStageSubmitted stageSubmitted) Called when a stage is submitted
void	onTaskEnd(SparkListenerTaskEnd taskEnd) Called when a task ends
void	onTaskGettingResult(SparkListenerTaskGettingResult taskGettingResult) Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).
void	onTaskStart(SparkListenerTaskStart taskStart) Called when a task starts
void	onUnpersistRDD(SparkListenerUnpersistRDD unpersistRDD) Called when an RDD is manually unpersisted by the application
void	onUnschedulableTaskSetAdded(SparkListenerUnschedulableTaskSetAdded unschedulableTaskSetAdded) Called when a taskset becomes unschedulable due to exludeOnFailure and dynamic allocation is enabled.
void	onUnschedulableTaskSetRemoved(SparkListenerUnschedulableTaskSetRemoved unschedulableTaskSetRemoved) Called when an unschedulable taskset becomes schedulable and dynamic allocation is enabled.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SparkListener

public SparkListener()

Method Detail

onApplicationEnd

public void onApplicationEnd(SparkListenerApplicationEnd applicationEnd)

Description copied from interface: SparkListenerInterface

Called when the application ends

Specified by:

onApplicationEnd in interface SparkListenerInterface

Parameters:

applicationEnd - (undocumented)

onApplicationStart

public void onApplicationStart(SparkListenerApplicationStart applicationStart)

Description copied from interface: SparkListenerInterface

Called when the application starts

Specified by:

onApplicationStart in interface SparkListenerInterface

Parameters:

applicationStart - (undocumented)

onBlockManagerAdded

public void onBlockManagerAdded(SparkListenerBlockManagerAdded blockManagerAdded)

Description copied from interface: SparkListenerInterface

Called when a new block manager has joined

Specified by:

onBlockManagerAdded in interface SparkListenerInterface

Parameters:

blockManagerAdded - (undocumented)

onBlockManagerRemoved

public void onBlockManagerRemoved(SparkListenerBlockManagerRemoved blockManagerRemoved)

Description copied from interface: SparkListenerInterface

Called when an existing block manager has been removed

Specified by:

onBlockManagerRemoved in interface SparkListenerInterface

Parameters:

blockManagerRemoved - (undocumented)

onBlockUpdated

public void onBlockUpdated(SparkListenerBlockUpdated blockUpdated)

Description copied from interface: SparkListenerInterface

Called when the driver receives a block update info.

Specified by:

onBlockUpdated in interface SparkListenerInterface

Parameters:

blockUpdated - (undocumented)

onEnvironmentUpdate

public void onEnvironmentUpdate(SparkListenerEnvironmentUpdate environmentUpdate)

Description copied from interface: SparkListenerInterface

Called when environment properties have been updated

Specified by:

onEnvironmentUpdate in interface SparkListenerInterface

Parameters:

environmentUpdate - (undocumented)

onExecutorAdded

public void onExecutorAdded(SparkListenerExecutorAdded executorAdded)

Description copied from interface: SparkListenerInterface

Called when the driver registers a new executor.

Specified by:

onExecutorAdded in interface SparkListenerInterface

Parameters:

executorAdded - (undocumented)

onExecutorBlacklisted

public void onExecutorBlacklisted(SparkListenerExecutorBlacklisted executorBlacklisted)

Description copied from interface: SparkListenerInterface

Called when the driver excludes an executor for a Spark application.

Specified by:

onExecutorBlacklisted in interface SparkListenerInterface

Parameters:

executorBlacklisted - (undocumented)

onExecutorBlacklistedForStage

public void onExecutorBlacklistedForStage(SparkListenerExecutorBlacklistedForStage executorBlacklistedForStage)

Description copied from interface: SparkListenerInterface

Called when the driver excludes an executor for a stage.

Specified by:

onExecutorBlacklistedForStage in interface SparkListenerInterface

Parameters:

executorBlacklistedForStage - (undocumented)

onExecutorExcluded

public void onExecutorExcluded(SparkListenerExecutorExcluded executorExcluded)

Description copied from interface: SparkListenerInterface

Called when the driver excludes an executor for a Spark application.

Specified by:

onExecutorExcluded in interface SparkListenerInterface

Parameters:

executorExcluded - (undocumented)

onExecutorExcludedForStage

public void onExecutorExcludedForStage(SparkListenerExecutorExcludedForStage executorExcludedForStage)

Description copied from interface: SparkListenerInterface

Called when the driver excludes an executor for a stage.

Specified by:

onExecutorExcludedForStage in interface SparkListenerInterface

Parameters:

executorExcludedForStage - (undocumented)

onExecutorMetricsUpdate

public void onExecutorMetricsUpdate(SparkListenerExecutorMetricsUpdate executorMetricsUpdate)

Description copied from interface: SparkListenerInterface

Called when the driver receives task metrics from an executor in a heartbeat.

Specified by:

onExecutorMetricsUpdate in interface SparkListenerInterface

Parameters:

executorMetricsUpdate - (undocumented)

onExecutorRemoved

public void onExecutorRemoved(SparkListenerExecutorRemoved executorRemoved)

Description copied from interface: SparkListenerInterface

Called when the driver removes an executor.

Specified by:

onExecutorRemoved in interface SparkListenerInterface

Parameters:

executorRemoved - (undocumented)

onExecutorUnblacklisted

public void onExecutorUnblacklisted(SparkListenerExecutorUnblacklisted executorUnblacklisted)

Description copied from interface: SparkListenerInterface

Called when the driver re-enables a previously excluded executor.

Specified by:

onExecutorUnblacklisted in interface SparkListenerInterface

Parameters:

executorUnblacklisted - (undocumented)

onExecutorUnexcluded

public void onExecutorUnexcluded(SparkListenerExecutorUnexcluded executorUnexcluded)

Description copied from interface: SparkListenerInterface

Called when the driver re-enables a previously excluded executor.

Specified by:

onExecutorUnexcluded in interface SparkListenerInterface

Parameters:

executorUnexcluded - (undocumented)

onJobEnd

public void onJobEnd(SparkListenerJobEnd jobEnd)

Description copied from interface: SparkListenerInterface

Called when a job ends

Specified by:

onJobEnd in interface SparkListenerInterface

Parameters:

jobEnd - (undocumented)

onJobStart

public void onJobStart(SparkListenerJobStart jobStart)

Description copied from interface: SparkListenerInterface

Called when a job starts

Specified by:

onJobStart in interface SparkListenerInterface

Parameters:

jobStart - (undocumented)

onNodeBlacklisted

public void onNodeBlacklisted(SparkListenerNodeBlacklisted nodeBlacklisted)

Description copied from interface: SparkListenerInterface

Called when the driver excludes a node for a Spark application.

Specified by:

onNodeBlacklisted in interface SparkListenerInterface

Parameters:

nodeBlacklisted - (undocumented)

onNodeBlacklistedForStage

public void onNodeBlacklistedForStage(SparkListenerNodeBlacklistedForStage nodeBlacklistedForStage)

Description copied from interface: SparkListenerInterface

Called when the driver excludes a node for a stage.

Specified by:

onNodeBlacklistedForStage in interface SparkListenerInterface

Parameters:

nodeBlacklistedForStage - (undocumented)

onNodeExcluded

public void onNodeExcluded(SparkListenerNodeExcluded nodeExcluded)

Description copied from interface: SparkListenerInterface

Called when the driver excludes a node for a Spark application.

Specified by:

onNodeExcluded in interface SparkListenerInterface

Parameters:

nodeExcluded - (undocumented)

onNodeExcludedForStage

public void onNodeExcludedForStage(SparkListenerNodeExcludedForStage nodeExcludedForStage)

Description copied from interface: SparkListenerInterface

Called when the driver excludes a node for a stage.

Specified by:

onNodeExcludedForStage in interface SparkListenerInterface

Parameters:

nodeExcludedForStage - (undocumented)

onNodeUnblacklisted

public void onNodeUnblacklisted(SparkListenerNodeUnblacklisted nodeUnblacklisted)

Description copied from interface: SparkListenerInterface

Called when the driver re-enables a previously excluded node.

Specified by:

onNodeUnblacklisted in interface SparkListenerInterface

Parameters:

nodeUnblacklisted - (undocumented)

onNodeUnexcluded

public void onNodeUnexcluded(SparkListenerNodeUnexcluded nodeUnexcluded)

Description copied from interface: SparkListenerInterface

Called when the driver re-enables a previously excluded node.

Specified by:

onNodeUnexcluded in interface SparkListenerInterface

Parameters:

nodeUnexcluded - (undocumented)

onOtherEvent

public void onOtherEvent(SparkListenerEvent event)

Description copied from interface: SparkListenerInterface

Called when other events like SQL-specific events are posted.

Specified by:

onOtherEvent in interface SparkListenerInterface

Parameters:

event - (undocumented)

onResourceProfileAdded

public void onResourceProfileAdded(SparkListenerResourceProfileAdded event)

Description copied from interface: SparkListenerInterface

Called when a Resource Profile is added to the manager.

Specified by:

onResourceProfileAdded in interface SparkListenerInterface

Parameters:

event - (undocumented)

onSpeculativeTaskSubmitted

public void onSpeculativeTaskSubmitted(SparkListenerSpeculativeTaskSubmitted speculativeTask)

Description copied from interface: SparkListenerInterface

Called when a speculative task is submitted

Specified by:

onSpeculativeTaskSubmitted in interface SparkListenerInterface

Parameters:

speculativeTask - (undocumented)

onStageCompleted

public void onStageCompleted(SparkListenerStageCompleted stageCompleted)

Description copied from interface: SparkListenerInterface

Called when a stage completes successfully or fails, with information on the completed stage.

Specified by:

onStageCompleted in interface SparkListenerInterface

Parameters:

stageCompleted - (undocumented)

onStageExecutorMetrics

public void onStageExecutorMetrics(SparkListenerStageExecutorMetrics executorMetrics)

Description copied from interface: SparkListenerInterface

Called with the peak memory metrics for a given (executor, stage) combination. Note that this is only present when reading from the event log (as in the history server), and is never called in a live application.

Specified by:

onStageExecutorMetrics in interface SparkListenerInterface

Parameters:

executorMetrics - (undocumented)

onStageSubmitted

public void onStageSubmitted(SparkListenerStageSubmitted stageSubmitted)

Description copied from interface: SparkListenerInterface

Called when a stage is submitted

Specified by:

onStageSubmitted in interface SparkListenerInterface

Parameters:

stageSubmitted - (undocumented)

onTaskEnd

public void onTaskEnd(SparkListenerTaskEnd taskEnd)

Description copied from interface: SparkListenerInterface

Called when a task ends

Specified by:

onTaskEnd in interface SparkListenerInterface

Parameters:

taskEnd - (undocumented)

onTaskGettingResult

public void onTaskGettingResult(SparkListenerTaskGettingResult taskGettingResult)

Description copied from interface: SparkListenerInterface

Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).

Specified by:

onTaskGettingResult in interface SparkListenerInterface

Parameters:

taskGettingResult - (undocumented)

onTaskStart

public void onTaskStart(SparkListenerTaskStart taskStart)

Description copied from interface: SparkListenerInterface

Called when a task starts

Specified by:

onTaskStart in interface SparkListenerInterface

Parameters:

taskStart - (undocumented)

onUnpersistRDD

public void onUnpersistRDD(SparkListenerUnpersistRDD unpersistRDD)

Description copied from interface: SparkListenerInterface

Called when an RDD is manually unpersisted by the application

Specified by:

onUnpersistRDD in interface SparkListenerInterface

Parameters:

unpersistRDD - (undocumented)

onUnschedulableTaskSetAdded

public void onUnschedulableTaskSetAdded(SparkListenerUnschedulableTaskSetAdded unschedulableTaskSetAdded)

Description copied from interface: SparkListenerInterface

Called when a taskset becomes unschedulable due to exludeOnFailure and dynamic allocation is enabled.

Specified by:

onUnschedulableTaskSetAdded in interface SparkListenerInterface

Parameters:

unschedulableTaskSetAdded - (undocumented)

onUnschedulableTaskSetRemoved

public void onUnschedulableTaskSetRemoved(SparkListenerUnschedulableTaskSetRemoved unschedulableTaskSetRemoved)

Description copied from interface: SparkListenerInterface

Called when an unschedulable taskset becomes schedulable and dynamic allocation is enabled.

Specified by:

onUnschedulableTaskSetRemoved in interface SparkListenerInterface

Parameters:

unschedulableTaskSetRemoved - (undocumented)