org.apache.spark.resource

Class ResourceProfile

Object

org.apache.spark.resource.ResourceProfile

All Implemented Interfaces:

java.io.Serializable, org.apache.spark.internal.Logging

public class ResourceProfile
extends Object
implements scala.Serializable, org.apache.spark.internal.Logging

Resource profile to associate with an RDD. A ResourceProfile allows the user to specify executor and task requirements for an RDD that will get applied during a stage. This allows the user to change the resource requirements between stages. This is meant to be immutable so user can't change it after building. Users should use ResourceProfileBuilder to build it.

param: executorResources Resource requests for executors. Mapped from the resource name (e.g., cores, memory, CPU) to its specific request. param: taskResources Resource requests for tasks. Mapped from the resource name (e.g., cores, memory, CPU) to its specific request.

See Also:

Serialized Form

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ResourceProfile.DefaultProfileExecutorResources$
static class	ResourceProfile.ExecutorResourcesOrDefaults$

Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

org.apache.spark.internal.Logging.SparkShellLoggingFilter

Constructor Summary

Constructors

Constructor and Description
ResourceProfile(scala.collection.immutable.Map<String,ExecutorResourceRequest> executorResources, scala.collection.immutable.Map<String,TaskResourceRequest> taskResources)

Method Summary

Modifier and Type	Method and Description
static String[]	allSupportedExecutorResources() Return all supported Spark built-in executor resources, custom resources like GPUs/FPGAs are excluded.
static String	CORES() built-in executor resource: cores
static String	CPUS() built-in task resource: cpus
static int	DEFAULT_RESOURCE_PROFILE_ID()
boolean	equals(Object obj)
scala.collection.immutable.Map<String,ExecutorResourceRequest>	executorResources()
java.util.Map<String,ExecutorResourceRequest>	executorResourcesJMap() (Java-specific) gets a Java Map of resources to ExecutorResourceRequest
int	hashCode()
int	id() A unique id of this ResourceProfile
static String	MEMORY() built-in executor resource: cores
static String	OFFHEAP_MEM() built-in executor resource: offHeap
static void	org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)
static org.slf4j.Logger	org$apache$spark$internal$Logging$$log_()
static String	OVERHEAD_MEM() built-in executor resource: memoryOverhead
static String	PYSPARK_MEM() built-in executor resource: pyspark.memory
scala.collection.immutable.Map<String,TaskResourceRequest>	taskResources()
java.util.Map<String,TaskResourceRequest>	taskResourcesJMap() (Java-specific) gets a Java Map of resources to TaskResourceRequest
String	toString()
static int	UNKNOWN_RESOURCE_PROFILE_ID()

Methods inherited from class Object

getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.spark.internal.Logging

$init$, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, initLock, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, uninitialize

Constructor Detail

ResourceProfile

public ResourceProfile(scala.collection.immutable.Map<String,ExecutorResourceRequest> executorResources,
                       scala.collection.immutable.Map<String,TaskResourceRequest> taskResources)

Method Detail

CPUS

public static String CPUS()

built-in task resource: cpus

Returns:

(undocumented)

CORES

public static String CORES()

built-in executor resource: cores

Returns:

(undocumented)

MEMORY

public static String MEMORY()

built-in executor resource: cores

Returns:

(undocumented)

OFFHEAP_MEM

public static String OFFHEAP_MEM()

built-in executor resource: offHeap

Returns:

(undocumented)

OVERHEAD_MEM

public static String OVERHEAD_MEM()

built-in executor resource: memoryOverhead

Returns:

(undocumented)

PYSPARK_MEM

public static String PYSPARK_MEM()

built-in executor resource: pyspark.memory

Returns:

(undocumented)

allSupportedExecutorResources

public static String[] allSupportedExecutorResources()

Return all supported Spark built-in executor resources, custom resources like GPUs/FPGAs are excluded.

Returns:

(undocumented)

UNKNOWN_RESOURCE_PROFILE_ID

public static int UNKNOWN_RESOURCE_PROFILE_ID()

DEFAULT_RESOURCE_PROFILE_ID

public static int DEFAULT_RESOURCE_PROFILE_ID()

org$apache$spark$internal$Logging$$log_

public static org.slf4j.Logger org$apache$spark$internal$Logging$$log_()

org$apache$spark$internal$Logging$$log__$eq

public static void org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)

executorResources

public scala.collection.immutable.Map<String,ExecutorResourceRequest> executorResources()

taskResources

public scala.collection.immutable.Map<String,TaskResourceRequest> taskResources()

id

public int id()

A unique id of this ResourceProfile

Returns:

(undocumented)

taskResourcesJMap

public java.util.Map<String,TaskResourceRequest> taskResourcesJMap()

(Java-specific) gets a Java Map of resources to TaskResourceRequest

Returns:

(undocumented)

executorResourcesJMap

public java.util.Map<String,ExecutorResourceRequest> executorResourcesJMap()

(Java-specific) gets a Java Map of resources to ExecutorResourceRequest

Returns:

(undocumented)

equals

public boolean equals(Object obj)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

toString

public String toString()

Overrides:

toString in class Object