class ExecutorResourceRequests extends Serializable
A set of Executor resource requests. This is used in conjunction with the ResourceProfile to programmatically specify the resources needed for an RDD that will be applied at the stage level.
- Annotations
- @Evolving() @Since( "3.1.0" )
- Source
- ExecutorResourceRequests.scala
- Alphabetic
- By Inheritance
- ExecutorResourceRequests
- Serializable
- Serializable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
- new ExecutorResourceRequests()
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native() @IntrinsicCandidate()
-
def
cores(amount: Int): ExecutorResourceRequests.this.type
Specify number of cores per Executor.
Specify number of cores per Executor. This is a convenient API to add ExecutorResourceRequest for "cores" resource.
- amount
Number of cores to allocate per Executor.
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @IntrinsicCandidate()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native() @IntrinsicCandidate()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
def
memory(amount: String): ExecutorResourceRequests.this.type
Specify heap memory.
Specify heap memory. The value specified will be converted to MiB. This is a convenient API to add ExecutorResourceRequest for "memory" resource.
- amount
Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.
-
def
memoryOverhead(amount: String): ExecutorResourceRequests.this.type
Specify overhead memory.
Specify overhead memory. The value specified will be converted to MiB. This is a convenient API to add ExecutorResourceRequest for "memoryOverhead" resource.
- amount
Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @IntrinsicCandidate()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native() @IntrinsicCandidate()
-
def
offHeapMemory(amount: String): ExecutorResourceRequests.this.type
Specify off heap memory.
Specify off heap memory. The value specified will be converted to MiB. This value only take effect when MEMORY_OFFHEAP_ENABLED is true. This is a convenient API to add ExecutorResourceRequest for "offHeap" resource.
- amount
Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.
-
def
pysparkMemory(amount: String): ExecutorResourceRequests.this.type
Specify pyspark memory.
Specify pyspark memory. The value specified will be converted to MiB. This is a convenient API to add ExecutorResourceRequest for "pyspark.memory" resource.
- amount
Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.
-
def
requests: Map[String, ExecutorResourceRequest]
Returns all the resource requests for the executor.
-
def
requestsJMap: Map[String, ExecutorResourceRequest]
(Java-specific) Returns all the resource requests for the executor.
-
def
resource(resourceName: String, amount: Long, discoveryScript: String = "", vendor: String = ""): ExecutorResourceRequests.this.type
Amount of a particular custom resource(GPU, FPGA, etc) to use.
Amount of a particular custom resource(GPU, FPGA, etc) to use. The resource names supported correspond to the regular Spark configs with the prefix removed. For instance, resources like GPUs are gpu (spark configs spark.executor.resource.gpu.*). If you pass in a resource that the cluster manager doesn't support the result is undefined, it may error or may just be ignored. This is a convenient API to add ExecutorResourceRequest for custom resources.
- resourceName
Name of the resource.
- amount
amount of that resource per executor to use.
- discoveryScript
Optional script used to discover the resources. This is required on some cluster managers that don't tell Spark the addresses of the resources allocated. The script runs on Executors startup to of the resources available.
- vendor
Optional vendor, required for some cluster managers
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- ExecutorResourceRequests → AnyRef → Any
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
Deprecated Value Members
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] ) @Deprecated
- Deprecated