pyspark.RDD.withResources¶

RDD.withResources(profile: pyspark.resource.profile.ResourceProfile) → pyspark.rdd.RDD[T][source]¶

Specify a pyspark.resource.ResourceProfile to use when calculating this RDD. This is only supported on certain cluster managers and currently requires dynamic allocation to be enabled. It will result in new executors with the resources specified being acquired to calculate the RDD.

New in version 3.1.0.

Parameters

profilepyspark.resource.ResourceProfile: a resource profile

Returns

RDD: the same RDD with user specified profile