|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
Object org.apache.spark.scheduler.JobLogger
public class JobLogger
:: DeveloperApi :: A logger class to record runtime information for jobs in Spark. This class outputs one log file for each Spark job, containing tasks start/stop and shuffle information. JobLogger is a subclass of SparkListener, use addSparkListener to add JobLogger to a SparkContext after the SparkContext is created. Note that each JobLogger only works for one SparkContext
NOTE: The functionality of this class is heavily stripped down to accommodate for a general refactor of the SparkListener interface. In its place, the EventLoggingListener is introduced to log application information as SparkListenerEvents. To enable this functionality, set spark.eventLog.enabled to true.
Constructor Summary | |
---|---|
JobLogger()
|
|
JobLogger(String user,
String logDirName)
|
Method Summary | |
---|---|
String |
logDirName()
|
void |
onJobEnd(SparkListenerJobEnd jobEnd)
When job ends, recording job completion status and close log file |
void |
onJobStart(SparkListenerJobStart jobStart)
When job starts, record job property and stage graph |
void |
onStageCompleted(SparkListenerStageCompleted stageCompleted)
When stage is completed, record stage completion status |
void |
onStageSubmitted(SparkListenerStageSubmitted stageSubmitted)
When stage is submitted, record stage submit info |
void |
onTaskEnd(SparkListenerTaskEnd taskEnd)
When task ends, record task completion status and metrics |
String |
user()
|
Methods inherited from class Object |
---|
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Methods inherited from interface org.apache.spark.scheduler.SparkListener |
---|
onApplicationEnd, onApplicationStart, onBlockManagerAdded, onBlockManagerRemoved, onEnvironmentUpdate, onExecutorAdded, onExecutorMetricsUpdate, onExecutorRemoved, onTaskGettingResult, onTaskStart, onUnpersistRDD |
Methods inherited from interface org.apache.spark.Logging |
---|
initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning |
Constructor Detail |
---|
public JobLogger(String user, String logDirName)
public JobLogger()
Method Detail |
---|
public String user()
public String logDirName()
public void onStageSubmitted(SparkListenerStageSubmitted stageSubmitted)
onStageSubmitted
in interface SparkListener
stageSubmitted
- Stage submitted eventpublic void onStageCompleted(SparkListenerStageCompleted stageCompleted)
onStageCompleted
in interface SparkListener
stageCompleted
- Stage completed eventpublic void onTaskEnd(SparkListenerTaskEnd taskEnd)
onTaskEnd
in interface SparkListener
taskEnd
- Task end eventpublic void onJobEnd(SparkListenerJobEnd jobEnd)
onJobEnd
in interface SparkListener
jobEnd
- Job end eventpublic void onJobStart(SparkListenerJobStart jobStart)
onJobStart
in interface SparkListener
jobStart
- Job start event
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |