|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
Object org.apache.spark.launcher.SparkLauncher
public class SparkLauncher
Launcher for Spark applications.
Use this class to start Spark applications programmatically. The class uses a builder pattern to allow clients to configure the Spark application and launch it as a child process.
Field Summary | |
---|---|
static String |
DRIVER_EXTRA_CLASSPATH
Configuration key for the driver class path. |
static String |
DRIVER_EXTRA_JAVA_OPTIONS
Configuration key for the driver VM options. |
static String |
DRIVER_EXTRA_LIBRARY_PATH
Configuration key for the driver native library path. |
static String |
DRIVER_MEMORY
Configuration key for the driver memory. |
static String |
EXECUTOR_CORES
Configuration key for the number of executor CPU cores. |
static String |
EXECUTOR_EXTRA_CLASSPATH
Configuration key for the executor class path. |
static String |
EXECUTOR_EXTRA_JAVA_OPTIONS
Configuration key for the executor VM options. |
static String |
EXECUTOR_EXTRA_LIBRARY_PATH
Configuration key for the executor native library path. |
static String |
EXECUTOR_MEMORY
Configuration key for the executor memory. |
static String |
SPARK_MASTER
The Spark master. |
Constructor Summary | |
---|---|
SparkLauncher()
|
|
SparkLauncher(java.util.Map<String,String> env)
Creates a launcher that will set the given environment variables in the child. |
Method Summary | |
---|---|
SparkLauncher |
addAppArgs(String... args)
Adds command line arguments for the application. |
SparkLauncher |
addFile(String file)
Adds a file to be submitted with the application. |
SparkLauncher |
addJar(String jar)
Adds a jar file to be submitted with the application. |
SparkLauncher |
addPyFile(String file)
Adds a python file / zip / egg to be submitted with the application. |
Process |
launch()
Launches a sub-process that will start the configured Spark application. |
SparkLauncher |
setAppName(String appName)
Set the application name. |
SparkLauncher |
setAppResource(String resource)
Set the main application resource. |
SparkLauncher |
setConf(String key,
String value)
Set a single configuration value for the application. |
SparkLauncher |
setDeployMode(String mode)
Set the deploy mode for the application. |
SparkLauncher |
setJavaHome(String javaHome)
Set a custom JAVA_HOME for launching the Spark application. |
SparkLauncher |
setMainClass(String mainClass)
Sets the application class name for Java/Scala applications. |
SparkLauncher |
setMaster(String master)
Set the Spark master for the application. |
SparkLauncher |
setPropertiesFile(String path)
Set a custom properties file with Spark configuration for the application. |
SparkLauncher |
setSparkHome(String sparkHome)
Set a custom Spark installation location for the application. |
SparkLauncher |
setVerbose(boolean verbose)
Enables verbose reporting for SparkSubmit. |
Methods inherited from class Object |
---|
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
public static final String SPARK_MASTER
public static final String DRIVER_MEMORY
public static final String DRIVER_EXTRA_CLASSPATH
public static final String DRIVER_EXTRA_JAVA_OPTIONS
public static final String DRIVER_EXTRA_LIBRARY_PATH
public static final String EXECUTOR_MEMORY
public static final String EXECUTOR_EXTRA_CLASSPATH
public static final String EXECUTOR_EXTRA_JAVA_OPTIONS
public static final String EXECUTOR_EXTRA_LIBRARY_PATH
public static final String EXECUTOR_CORES
Constructor Detail |
---|
public SparkLauncher()
public SparkLauncher(java.util.Map<String,String> env)
env
- Environment variables to set.Method Detail |
---|
public SparkLauncher setJavaHome(String javaHome)
javaHome
- Path to the JAVA_HOME to use.
public SparkLauncher setSparkHome(String sparkHome)
sparkHome
- Path to the Spark installation to use.
public SparkLauncher setPropertiesFile(String path)
path
- Path to custom properties file to use.
public SparkLauncher setConf(String key, String value)
key
- Configuration key.value
- The value to use.
public SparkLauncher setAppName(String appName)
appName
- Application name.
public SparkLauncher setMaster(String master)
master
- Spark master.
public SparkLauncher setDeployMode(String mode)
mode
- Deploy mode.
public SparkLauncher setAppResource(String resource)
resource
- Path to the main application resource.
public SparkLauncher setMainClass(String mainClass)
mainClass
- Application's main class.
public SparkLauncher addAppArgs(String... args)
args
- Arguments to pass to the application's main class.
public SparkLauncher addJar(String jar)
jar
- Path to the jar file.
public SparkLauncher addFile(String file)
file
- Path to the file.
public SparkLauncher addPyFile(String file)
file
- Path to the file.
public SparkLauncher setVerbose(boolean verbose)
verbose
- Whether to enable verbose output.
public Process launch() throws java.io.IOException
java.io.IOException
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |