org.apache.spark.util

Class DependencyUtils

Object

org.apache.spark.util.DependencyUtils

public class DependencyUtils
extends Object

Constructor Summary

Constructors

Constructor and Description
DependencyUtils()

Method Summary

Modifier and Type	Method and Description
static void	addJarsToClassPath(String jars, MutableURLClassLoader loader)
static void	addJarToClasspath(String localJar, MutableURLClassLoader loader)
static String	downloadFile(String path, java.io.File targetDir, SparkConf sparkConf, org.apache.hadoop.conf.Configuration hadoopConf) Download a file from the remote to a local temporary directory.
static String	downloadFileList(String fileList, java.io.File targetDir, SparkConf sparkConf, org.apache.hadoop.conf.Configuration hadoopConf) Download a list of remote files to temp local files.
static org.apache.spark.util.IvyProperties	getIvyProperties()
static String	mergeFileLists(scala.collection.Seq<String> lists) Merge a sequence of comma-separated file lists, some of which may be null to indicate no files, into a single comma-separated string.
static void	org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)
static org.slf4j.Logger	org$apache$spark$internal$Logging$$log_()
static String	resolveAndDownloadJars(String jars, String userJar, SparkConf sparkConf, org.apache.hadoop.conf.Configuration hadoopConf)
static String	resolveGlobPaths(String paths, org.apache.hadoop.conf.Configuration hadoopConf)
static scala.collection.Seq<String>	resolveMavenDependencies(boolean packagesTransitive, String packagesExclusions, String packages, String repositories, String ivyRepoPath, scala.Option<String> ivySettingsPath)
static scala.collection.Seq<String>	resolveMavenDependencies(java.net.URI uri) Download Ivy URI's dependency jars.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DependencyUtils

public DependencyUtils()

Method Detail

getIvyProperties

public static org.apache.spark.util.IvyProperties getIvyProperties()

resolveMavenDependencies

public static scala.collection.Seq<String> resolveMavenDependencies(java.net.URI uri)

Download Ivy URI's dependency jars.

Parameters:

uri - Ivy URI need to be downloaded. The URI format should be: ivy://group:module:version[?query] Ivy URI query part format should be: parameter=value&parameter=value... Note that currently Ivy URI query part support two parameters: 1. transitive: whether to download dependent jars related to your Ivy URI. transitive=false or transitive=true, if not set, the default value is true. 2. exclude: exclusion list when download Ivy URI jar and dependency jars. The exclude parameter content is a ',' separated group:module pair string : exclude=group:module,group:module...

Returns:

List of jars downloaded.

resolveMavenDependencies

public static scala.collection.Seq<String> resolveMavenDependencies(boolean packagesTransitive,
                                                                    String packagesExclusions,
                                                                    String packages,
                                                                    String repositories,
                                                                    String ivyRepoPath,
                                                                    scala.Option<String> ivySettingsPath)

resolveAndDownloadJars

public static String resolveAndDownloadJars(String jars,
                                            String userJar,
                                            SparkConf sparkConf,
                                            org.apache.hadoop.conf.Configuration hadoopConf)

addJarsToClassPath

public static void addJarsToClassPath(String jars,
                                      MutableURLClassLoader loader)

downloadFileList

public static String downloadFileList(String fileList,
                                      java.io.File targetDir,
                                      SparkConf sparkConf,
                                      org.apache.hadoop.conf.Configuration hadoopConf)

Download a list of remote files to temp local files. If the file is local, the original file will be returned.

Parameters:

fileList - A comma separated file list.

targetDir - A temporary directory for which downloaded files.

sparkConf - Spark configuration.

hadoopConf - Hadoop configuration.

Returns:

A comma separated local files list.

downloadFile

public static String downloadFile(String path,
                                  java.io.File targetDir,
                                  SparkConf sparkConf,
                                  org.apache.hadoop.conf.Configuration hadoopConf)

Download a file from the remote to a local temporary directory. If the input path points to a local path, returns it with no operation.

Parameters:

path - A file path from where the files will be downloaded.

targetDir - A temporary directory for which downloaded files.

sparkConf - Spark configuration.

hadoopConf - Hadoop configuration.

Returns:

Path to the local file.

resolveGlobPaths

public static String resolveGlobPaths(String paths,
                                      org.apache.hadoop.conf.Configuration hadoopConf)

addJarToClasspath

public static void addJarToClasspath(String localJar,
                                     MutableURLClassLoader loader)

mergeFileLists

public static String mergeFileLists(scala.collection.Seq<String> lists)

Merge a sequence of comma-separated file lists, some of which may be null to indicate no files, into a single comma-separated string.

Parameters:

lists - (undocumented)

Returns:

(undocumented)

org$apache$spark$internal$Logging$$log_

public static org.slf4j.Logger org$apache$spark$internal$Logging$$log_()

org$apache$spark$internal$Logging$$log__$eq

public static void org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)