org.apache.spark.ml.image

Class SamplePathFilter

Object

org.apache.hadoop.conf.Configured

org.apache.spark.ml.image.SamplePathFilter

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, org.apache.hadoop.fs.PathFilter

public class SamplePathFilter
extends org.apache.hadoop.conf.Configured
implements org.apache.hadoop.fs.PathFilter

Filter that allows loading a fraction of HDFS files.

Constructor Summary

Constructors

Constructor and Description
SamplePathFilter()

Method Summary

Modifier and Type	Method and Description
boolean	accept(org.apache.hadoop.fs.Path path)
static boolean	isFile(org.apache.hadoop.fs.Path path)
scala.util.Random	random()
static String	ratioParam()
double	sampleRatio()
static String	seedParam()
void	setConf(org.apache.hadoop.conf.Configuration conf)
static <T> T	withPathFilter(double sampleRatio, SparkSession spark, long seed, scala.Function0<T> f) Sets the HDFS PathFilter flag and then restores it.

Methods inherited from class org.apache.hadoop.conf.Configured

getConf

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SamplePathFilter

public SamplePathFilter()

Method Detail

ratioParam

public static String ratioParam()

seedParam

public static String seedParam()

isFile

public static boolean isFile(org.apache.hadoop.fs.Path path)

withPathFilter

public static <T> T withPathFilter(double sampleRatio,
                                   SparkSession spark,
                                   long seed,
                                   scala.Function0<T> f)

Sets the HDFS PathFilter flag and then restores it. Only applies the filter if sampleRatio is less than 1.

Parameters:

sampleRatio - Fraction of the files that the filter picks

spark - Existing Spark session

seed - Random number seed

f - The function to evaluate after setting the flag

Returns:

Returns the evaluation result T of the function

random

public scala.util.Random random()

sampleRatio

public double sampleRatio()

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)

Specified by:

setConf in interface org.apache.hadoop.conf.Configurable

Overrides:

setConf in class org.apache.hadoop.conf.Configured

accept

public boolean accept(org.apache.hadoop.fs.Path path)

Specified by:

accept in interface org.apache.hadoop.fs.PathFilter