Packages

c

org.apache.spark.resource

ExecutorResourceRequest

class ExecutorResourceRequest extends Serializable

An Executor resource request. This is used in conjunction with the ResourceProfile to programmatically specify the resources needed for an RDD that will be applied at the stage level.

This is used to specify what the resource requirements are for an Executor and how Spark can find out specific details about those resources. Not all the parameters are required for every resource type. Resources like GPUs are supported and have same limitations as using the global spark configs spark.executor.resource.gpu.*. The amount, discoveryScript, and vendor parameters for resources are all the same parameters a user would specify through the configs: spark.executor.resource.{resourceName}.{amount, discoveryScript, vendor}.

For instance, a user wants to allocate an Executor with GPU resources on YARN. The user has to specify the resource name (gpu), the amount or number of GPUs per Executor, the discovery script would be specified so that when the Executor starts up it can discovery what GPU addresses are available for it to use because YARN doesn't tell Spark that, then vendor would not be used because its specific for Kubernetes.

See the configuration and cluster specific docs for more details.

Use ExecutorResourceRequests class as a convenience API.

Annotations
@Evolving() @Since( "3.1.0" )
Source
ExecutorResourceRequest.scala
Linear Supertypes
Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ExecutorResourceRequest
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new ExecutorResourceRequest(resourceName: String, amount: Long, discoveryScript: String = "", vendor: String = "")

    resourceName

    Name of the resource

    amount

    Amount requesting

    discoveryScript

    Optional script used to discover the resources. This is required on some cluster managers that don't tell Spark the addresses of the resources allocated. The script runs on Executors startup to discover the addresses of the resources available.

    vendor

    Optional vendor, required for some cluster managers

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. val amount: Long
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  7. val discoveryScript: String
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(obj: Any): Boolean
    Definition Classes
    ExecutorResourceRequest → AnyRef → Any
  10. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def hashCode(): Int
    Definition Classes
    ExecutorResourceRequest → AnyRef → Any
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. val resourceName: String
  18. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  19. def toString(): String
    Definition Classes
    ExecutorResourceRequest → AnyRef → Any
  20. val vendor: String
  21. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  23. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped