Package org.apache.spark
Class SparkFirehoseListener
Object
org.apache.spark.SparkFirehoseListener
- All Implemented Interfaces:
SparkListenerInterface
Class that allows users to receive all SparkListener events.
Users should override the onEvent method.
This is a concrete Java class in order to ensure that we don't forget to update it when adding
new methods to SparkListener: forgetting to add a method will result in a compilation error (if
this was a concrete Scala class, default implementations of new event handlers would be inherited
from the SparkListener trait).
Please note until Spark 3.1.0 this was missing the DevelopApi annotation, this needs to be
taken into account if changing this API before a major release.
-
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionfinal voidonApplicationEnd(SparkListenerApplicationEnd applicationEnd) Called when the application endsfinal voidonApplicationStart(SparkListenerApplicationStart applicationStart) Called when the application startsfinal voidonBlockManagerAdded(SparkListenerBlockManagerAdded blockManagerAdded) Called when a new block manager has joinedfinal voidonBlockManagerRemoved(SparkListenerBlockManagerRemoved blockManagerRemoved) Called when an existing block manager has been removedvoidonBlockUpdated(SparkListenerBlockUpdated blockUpdated) Called when the driver receives a block update info.final voidonEnvironmentUpdate(SparkListenerEnvironmentUpdate environmentUpdate) Called when environment properties have been updatedvoidonEvent(SparkListenerEvent event) final voidonExecutorAdded(SparkListenerExecutorAdded executorAdded) Called when the driver registers a new executor.final voidonExecutorBlacklisted(SparkListenerExecutorBlacklisted executorBlacklisted) Called when the driver excludes an executor for a Spark application.voidonExecutorBlacklistedForStage(SparkListenerExecutorBlacklistedForStage executorBlacklistedForStage) Called when the driver excludes an executor for a stage.final voidonExecutorExcluded(SparkListenerExecutorExcluded executorExcluded) Called when the driver excludes an executor for a Spark application.voidonExecutorExcludedForStage(SparkListenerExecutorExcludedForStage executorExcludedForStage) Called when the driver excludes an executor for a stage.final voidonExecutorMetricsUpdate(SparkListenerExecutorMetricsUpdate executorMetricsUpdate) Called when the driver receives task metrics from an executor in a heartbeat.final voidonExecutorRemoved(SparkListenerExecutorRemoved executorRemoved) Called when the driver removes an executor.final voidonExecutorUnblacklisted(SparkListenerExecutorUnblacklisted executorUnblacklisted) Called when the driver re-enables a previously excluded executor.final voidonExecutorUnexcluded(SparkListenerExecutorUnexcluded executorUnexcluded) Called when the driver re-enables a previously excluded executor.final voidonJobEnd(SparkListenerJobEnd jobEnd) Called when a job endsfinal voidonJobStart(SparkListenerJobStart jobStart) Called when a job startsfinal voidonNodeBlacklisted(SparkListenerNodeBlacklisted nodeBlacklisted) Called when the driver excludes a node for a Spark application.voidonNodeBlacklistedForStage(SparkListenerNodeBlacklistedForStage nodeBlacklistedForStage) Called when the driver excludes a node for a stage.final voidonNodeExcluded(SparkListenerNodeExcluded nodeExcluded) Called when the driver excludes a node for a Spark application.voidonNodeExcludedForStage(SparkListenerNodeExcludedForStage nodeExcludedForStage) Called when the driver excludes a node for a stage.final voidonNodeUnblacklisted(SparkListenerNodeUnblacklisted nodeUnblacklisted) Called when the driver re-enables a previously excluded node.final voidonNodeUnexcluded(SparkListenerNodeUnexcluded nodeUnexcluded) Called when the driver re-enables a previously excluded node.voidonOtherEvent(SparkListenerEvent event) Called when other events like SQL-specific events are posted.voidCalled when a Resource Profile is added to the manager.voidonSpeculativeTaskSubmitted(SparkListenerSpeculativeTaskSubmitted speculativeTask) Called when a speculative task is submittedfinal voidonStageCompleted(SparkListenerStageCompleted stageCompleted) Called when a stage completes successfully or fails, with information on the completed stage.final voidonStageExecutorMetrics(SparkListenerStageExecutorMetrics executorMetrics) Called with the peak memory metrics for a given (executor, stage) combination.final voidonStageSubmitted(SparkListenerStageSubmitted stageSubmitted) Called when a stage is submittedfinal voidonTaskEnd(SparkListenerTaskEnd taskEnd) Called when a task endsfinal voidonTaskGettingResult(SparkListenerTaskGettingResult taskGettingResult) Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).final voidonTaskStart(SparkListenerTaskStart taskStart) Called when a task startsfinal voidonUnpersistRDD(SparkListenerUnpersistRDD unpersistRDD) Called when an RDD is manually unpersisted by the applicationvoidonUnschedulableTaskSetAdded(SparkListenerUnschedulableTaskSetAdded unschedulableTaskSetAdded) Called when a taskset becomes unschedulable due to exludeOnFailure and dynamic allocation is enabled.voidonUnschedulableTaskSetRemoved(SparkListenerUnschedulableTaskSetRemoved unschedulableTaskSetRemoved) Called when an unschedulable taskset becomes schedulable and dynamic allocation is enabled.
-
Constructor Details
-
SparkFirehoseListener
public SparkFirehoseListener()
-
-
Method Details
-
onEvent
-
onStageCompleted
Description copied from interface:SparkListenerInterfaceCalled when a stage completes successfully or fails, with information on the completed stage.- Specified by:
onStageCompletedin interfaceSparkListenerInterface- Parameters:
stageCompleted- (undocumented)
-
onStageSubmitted
Description copied from interface:SparkListenerInterfaceCalled when a stage is submitted- Specified by:
onStageSubmittedin interfaceSparkListenerInterface- Parameters:
stageSubmitted- (undocumented)
-
onTaskStart
Description copied from interface:SparkListenerInterfaceCalled when a task starts- Specified by:
onTaskStartin interfaceSparkListenerInterface- Parameters:
taskStart- (undocumented)
-
onTaskGettingResult
Description copied from interface:SparkListenerInterfaceCalled when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).- Specified by:
onTaskGettingResultin interfaceSparkListenerInterface- Parameters:
taskGettingResult- (undocumented)
-
onTaskEnd
Description copied from interface:SparkListenerInterfaceCalled when a task ends- Specified by:
onTaskEndin interfaceSparkListenerInterface- Parameters:
taskEnd- (undocumented)
-
onJobStart
Description copied from interface:SparkListenerInterfaceCalled when a job starts- Specified by:
onJobStartin interfaceSparkListenerInterface- Parameters:
jobStart- (undocumented)
-
onJobEnd
Description copied from interface:SparkListenerInterfaceCalled when a job ends- Specified by:
onJobEndin interfaceSparkListenerInterface- Parameters:
jobEnd- (undocumented)
-
onEnvironmentUpdate
Description copied from interface:SparkListenerInterfaceCalled when environment properties have been updated- Specified by:
onEnvironmentUpdatein interfaceSparkListenerInterface- Parameters:
environmentUpdate- (undocumented)
-
onBlockManagerAdded
Description copied from interface:SparkListenerInterfaceCalled when a new block manager has joined- Specified by:
onBlockManagerAddedin interfaceSparkListenerInterface- Parameters:
blockManagerAdded- (undocumented)
-
onBlockManagerRemoved
Description copied from interface:SparkListenerInterfaceCalled when an existing block manager has been removed- Specified by:
onBlockManagerRemovedin interfaceSparkListenerInterface- Parameters:
blockManagerRemoved- (undocumented)
-
onUnpersistRDD
Description copied from interface:SparkListenerInterfaceCalled when an RDD is manually unpersisted by the application- Specified by:
onUnpersistRDDin interfaceSparkListenerInterface- Parameters:
unpersistRDD- (undocumented)
-
onApplicationStart
Description copied from interface:SparkListenerInterfaceCalled when the application starts- Specified by:
onApplicationStartin interfaceSparkListenerInterface- Parameters:
applicationStart- (undocumented)
-
onApplicationEnd
Description copied from interface:SparkListenerInterfaceCalled when the application ends- Specified by:
onApplicationEndin interfaceSparkListenerInterface- Parameters:
applicationEnd- (undocumented)
-
onExecutorMetricsUpdate
Description copied from interface:SparkListenerInterfaceCalled when the driver receives task metrics from an executor in a heartbeat.- Specified by:
onExecutorMetricsUpdatein interfaceSparkListenerInterface- Parameters:
executorMetricsUpdate- (undocumented)
-
onStageExecutorMetrics
Description copied from interface:SparkListenerInterfaceCalled with the peak memory metrics for a given (executor, stage) combination. Note that this is only present when reading from the event log (as in the history server), and is never called in a live application.- Specified by:
onStageExecutorMetricsin interfaceSparkListenerInterface- Parameters:
executorMetrics- (undocumented)
-
onExecutorAdded
Description copied from interface:SparkListenerInterfaceCalled when the driver registers a new executor.- Specified by:
onExecutorAddedin interfaceSparkListenerInterface- Parameters:
executorAdded- (undocumented)
-
onExecutorRemoved
Description copied from interface:SparkListenerInterfaceCalled when the driver removes an executor.- Specified by:
onExecutorRemovedin interfaceSparkListenerInterface- Parameters:
executorRemoved- (undocumented)
-
onExecutorBlacklisted
Description copied from interface:SparkListenerInterfaceCalled when the driver excludes an executor for a Spark application.- Specified by:
onExecutorBlacklistedin interfaceSparkListenerInterface- Parameters:
executorBlacklisted- (undocumented)
-
onExecutorExcluded
Description copied from interface:SparkListenerInterfaceCalled when the driver excludes an executor for a Spark application.- Specified by:
onExecutorExcludedin interfaceSparkListenerInterface- Parameters:
executorExcluded- (undocumented)
-
onExecutorBlacklistedForStage
public void onExecutorBlacklistedForStage(SparkListenerExecutorBlacklistedForStage executorBlacklistedForStage) Description copied from interface:SparkListenerInterfaceCalled when the driver excludes an executor for a stage.- Specified by:
onExecutorBlacklistedForStagein interfaceSparkListenerInterface- Parameters:
executorBlacklistedForStage- (undocumented)
-
onExecutorExcludedForStage
public void onExecutorExcludedForStage(SparkListenerExecutorExcludedForStage executorExcludedForStage) Description copied from interface:SparkListenerInterfaceCalled when the driver excludes an executor for a stage.- Specified by:
onExecutorExcludedForStagein interfaceSparkListenerInterface- Parameters:
executorExcludedForStage- (undocumented)
-
onNodeBlacklistedForStage
Description copied from interface:SparkListenerInterfaceCalled when the driver excludes a node for a stage.- Specified by:
onNodeBlacklistedForStagein interfaceSparkListenerInterface- Parameters:
nodeBlacklistedForStage- (undocumented)
-
onNodeExcludedForStage
Description copied from interface:SparkListenerInterfaceCalled when the driver excludes a node for a stage.- Specified by:
onNodeExcludedForStagein interfaceSparkListenerInterface- Parameters:
nodeExcludedForStage- (undocumented)
-
onExecutorUnblacklisted
Description copied from interface:SparkListenerInterfaceCalled when the driver re-enables a previously excluded executor.- Specified by:
onExecutorUnblacklistedin interfaceSparkListenerInterface- Parameters:
executorUnblacklisted- (undocumented)
-
onExecutorUnexcluded
Description copied from interface:SparkListenerInterfaceCalled when the driver re-enables a previously excluded executor.- Specified by:
onExecutorUnexcludedin interfaceSparkListenerInterface- Parameters:
executorUnexcluded- (undocumented)
-
onNodeBlacklisted
Description copied from interface:SparkListenerInterfaceCalled when the driver excludes a node for a Spark application.- Specified by:
onNodeBlacklistedin interfaceSparkListenerInterface- Parameters:
nodeBlacklisted- (undocumented)
-
onNodeExcluded
Description copied from interface:SparkListenerInterfaceCalled when the driver excludes a node for a Spark application.- Specified by:
onNodeExcludedin interfaceSparkListenerInterface- Parameters:
nodeExcluded- (undocumented)
-
onNodeUnblacklisted
Description copied from interface:SparkListenerInterfaceCalled when the driver re-enables a previously excluded node.- Specified by:
onNodeUnblacklistedin interfaceSparkListenerInterface- Parameters:
nodeUnblacklisted- (undocumented)
-
onNodeUnexcluded
Description copied from interface:SparkListenerInterfaceCalled when the driver re-enables a previously excluded node.- Specified by:
onNodeUnexcludedin interfaceSparkListenerInterface- Parameters:
nodeUnexcluded- (undocumented)
-
onBlockUpdated
Description copied from interface:SparkListenerInterfaceCalled when the driver receives a block update info.- Specified by:
onBlockUpdatedin interfaceSparkListenerInterface- Parameters:
blockUpdated- (undocumented)
-
onSpeculativeTaskSubmitted
Description copied from interface:SparkListenerInterfaceCalled when a speculative task is submitted- Specified by:
onSpeculativeTaskSubmittedin interfaceSparkListenerInterface- Parameters:
speculativeTask- (undocumented)
-
onUnschedulableTaskSetAdded
public void onUnschedulableTaskSetAdded(SparkListenerUnschedulableTaskSetAdded unschedulableTaskSetAdded) Description copied from interface:SparkListenerInterfaceCalled when a taskset becomes unschedulable due to exludeOnFailure and dynamic allocation is enabled.- Specified by:
onUnschedulableTaskSetAddedin interfaceSparkListenerInterface- Parameters:
unschedulableTaskSetAdded- (undocumented)
-
onUnschedulableTaskSetRemoved
public void onUnschedulableTaskSetRemoved(SparkListenerUnschedulableTaskSetRemoved unschedulableTaskSetRemoved) Description copied from interface:SparkListenerInterfaceCalled when an unschedulable taskset becomes schedulable and dynamic allocation is enabled.- Specified by:
onUnschedulableTaskSetRemovedin interfaceSparkListenerInterface- Parameters:
unschedulableTaskSetRemoved- (undocumented)
-
onResourceProfileAdded
Description copied from interface:SparkListenerInterfaceCalled when a Resource Profile is added to the manager.- Specified by:
onResourceProfileAddedin interfaceSparkListenerInterface- Parameters:
event- (undocumented)
-
onOtherEvent
Description copied from interface:SparkListenerInterfaceCalled when other events like SQL-specific events are posted.- Specified by:
onOtherEventin interfaceSparkListenerInterface- Parameters:
event- (undocumented)
-