Package org.apache.spark
package org.apache.spark
Core Spark classes in Scala. A few classes here, such as
StorageLevel, are also used in Java, but the
org.apache.spark.api.java package contains the main Java API.-
ClassDescriptionAggregator<K,
V, C> :: DeveloperApi :: A set of functions used to aggregate data.:: Experimental :: ATaskContextwith extra contextual info and tooling for tasks in a barrier stage.:: Experimental :: Carries all task infos of a barrier task.Listener class used when any item has been cleaned by the Cleaner class.Classes that represent cleaning tasks.A WeakReference associated with a CleanupTask.AFutureActionfor actions that could trigger multiple Spark jobs.:: DeveloperApi :: A TaskContext aware iterator.For each barrier stage attempt, only at most one barrier() call can be active at any time, thus we can use (stageId, stageAttemptId) to identify the stage attempt where the barrier() call is from.Dependency<T>:: DeveloperApi :: Base class for dependencies.A reader to load error information from one or more JSON files.Information associated with an error class.Information associated with an error subclass.:: DeveloperApi :: Task failed due to a runtime exception.:: DeveloperApi :: The task failed because the executor that it was running on was lost.:: DeveloperApi :: Task failed to fetch shuffle data from a remote node.FutureAction<T>A future for the result of an action to support cancellation.APartitionerthat implements hash-based partitioning using Java'sObject.hashCode.A collection of fields and methods concerned with internal accumulators that represent task level metrics.:: DeveloperApi :: An iterator that wraps around an existing iterator to provide task killing functionality.Handle via which a "run" function passed to aComplexFutureActioncan submit jobs for execution.:: DeveloperApi :: Base class for dependencies where each partition of the child RDD depends on a small number of partitions of the parent RDD.:: DeveloperApi :: Represents a one-to-one dependency between partitions of the parent and child RDDs.An identifier for a partition in an RDD.An object that defines how the elements in a key-value pair RDD are partitioned by key.PartitionEvaluator<T,U> An evaluator for computing RDD partitions.A factory to createPartitionEvaluator.Query context of aSparkThrowable.:: DeveloperApi :: Represents a one-to-one dependency between ranges of partitions in the parent and child RDDs.RangePartitioner<K,V> APartitionerthat partitions sortable records by range into roughly equal ranges.:: DeveloperApi :: Aorg.apache.spark.scheduler.ShuffleMapTaskthat completed successfully earlier, but we lost the executor before the stage completed.SerializableWritable<T extends org.apache.hadoop.io.Writable>ShuffleDependency<K,V, C> :: DeveloperApi :: Represents a dependency on the output of a shuffle stage.Helper class used by theMapOutputTrackerMasterto perform bookkeeping for a single ShuffleMapStage.AFutureActionholding the result of an action that triggers a single job.Configuration for a Spark application.Main entry point for Spark functionality.:: DeveloperApi :: Holds all the runtime environment objects for a running Spark instance (either master or worker), including the serializer, RpcEnv, block manager, map output tracker, etc.Exposes information about Spark Executors.Resolves paths to files added throughSparkContext.addFile().Class that allows users to receive all SparkListener events.Exposes information about Spark Jobs.A collection of regexes for extracting information from the master string.Exposes information about Spark Stages.Low-level status reporting APIs for monitoring job and stage progress.Interface mixed into Throwables thrown from SparkCompanion object used by instances ofSparkThrowableto access error class information and construct error messages.ASparkListenerthat detects whether spills have occurred in Spark jobs.:: DeveloperApi :: Task succeeded.:: DeveloperApi :: Task requested the driver to commit, but was denied.Contextual information about a task which can be read or mutated during execution.:: DeveloperApi :: Various possible reasons why a task ended.:: DeveloperApi :: Various possible reasons why a task failed.:: DeveloperApi :: Task was killed intentionally and needs to be rescheduled.:: DeveloperApi :: Exception thrown when a task is explicitly killed (i.e., task failure is expected).:: DeveloperApi :: The task finished successfully, but the result was lost from the executor's block manager before it was fetched.An event that SparkContext uses to notify HeartbeatReceiver that SparkContext.taskScheduler is created.Utilities for tests.:: DeveloperApi :: We don't know why the task ended -- for example, because of a ClassNotFound exception when deserializing the task result.