public final class ChiSqSelector extends Estimator<T>
numTopFeatures
, percentile
, fpr
,
fdr
, fwe
.
- numTopFeatures
chooses a fixed number of top features according to a chi-squared test.
- percentile
is similar but chooses a fraction of all features instead of a fixed number.
- fpr
chooses all features whose p-value are below a threshold, thus controlling the false
positive rate of selection.
- fdr
uses the [Benjamini-Hochberg procedure]
(https://en.wikipedia.org/wiki/False_discovery_rate#Benjamini.E2.80.93Hochberg_procedure)
to choose all features whose false discovery rate is below a threshold.
- fwe
chooses all features whose p-values are below a threshold. The threshold is scaled by
1/numFeatures, thus controlling the family-wise error rate of selection.
By default, the selection method is numTopFeatures
, with the default number of top features
set to 50.Constructor and Description |
---|
ChiSqSelector()
Deprecated.
|
ChiSqSelector(String uid)
Deprecated.
|
Modifier and Type | Method and Description |
---|---|
ChiSqSelector |
copy(ParamMap extra)
Deprecated.
Creates a copy of this instance with the same UID and some extra params.
|
DoubleParam |
fdr()
The upper bound of the expected false discovery rate.
|
Param<String> |
featuresCol()
Param for features column name.
|
ChiSqSelectorModel |
fit(Dataset<?> dataset)
Deprecated.
Fits a model to the input data.
|
DoubleParam |
fpr()
The highest p-value for features to be kept.
|
DoubleParam |
fwe()
The upper bound of the expected family-wise error rate.
|
Param<String> |
labelCol()
Param for label column name.
|
static ChiSqSelector |
load(String path)
Deprecated.
|
IntParam |
numTopFeatures()
Number of features that selector will select, ordered by ascending p-value.
|
Param<String> |
outputCol()
Param for output column name.
|
DoubleParam |
percentile()
Percentile of features that selector will select, ordered by ascending p-value.
|
static MLReader<T> |
read()
Deprecated.
|
Param<String> |
selectorType()
The selector type.
|
ChiSqSelector |
setFdr(double value)
Deprecated.
|
ChiSqSelector |
setFeaturesCol(String value)
Deprecated.
|
ChiSqSelector |
setFpr(double value)
Deprecated.
|
ChiSqSelector |
setFwe(double value)
Deprecated.
|
ChiSqSelector |
setLabelCol(String value)
Deprecated.
|
ChiSqSelector |
setNumTopFeatures(int value)
Deprecated.
|
ChiSqSelector |
setOutputCol(String value)
Deprecated.
|
ChiSqSelector |
setPercentile(double value)
Deprecated.
|
ChiSqSelector |
setSelectorType(String value)
Deprecated.
|
StructType |
transformSchema(StructType schema)
Deprecated.
Check transform validity and derive the output schema from the input schema.
|
String |
uid()
Deprecated.
An immutable unique ID for the object and its derivatives.
|
params
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
getFdr, getFpr, getFwe, getNumTopFeatures, getPercentile, getSelectorType
getFeaturesCol
getLabelCol
getOutputCol
clear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, onParamChange, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn
toString
write
save
$init$, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, initLock, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, uninitialize
public ChiSqSelector(String uid)
public ChiSqSelector()
public static ChiSqSelector load(String path)
public static MLReader<T> read()
public String uid()
Identifiable
public ChiSqSelector setNumTopFeatures(int value)
public ChiSqSelector setPercentile(double value)
public ChiSqSelector setFpr(double value)
public ChiSqSelector setFdr(double value)
public ChiSqSelector setFwe(double value)
public ChiSqSelector setSelectorType(String value)
public ChiSqSelector setFeaturesCol(String value)
public ChiSqSelector setOutputCol(String value)
public ChiSqSelector setLabelCol(String value)
public ChiSqSelectorModel fit(Dataset<?> dataset)
Estimator
dataset
- (undocumented)public StructType transformSchema(StructType schema)
PipelineStage
We check validity for interactions between parameters during transformSchema
and
raise an exception if any parameter value is invalid. Parameter value checks which
do not depend on other parameters are handled by Param.validate()
.
Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.
schema
- (undocumented)public ChiSqSelector copy(ParamMap extra)
Params
defaultCopy()
.public final DoubleParam fdr()
SelectorParams
fdr
in interface SelectorParams
public final Param<String> featuresCol()
HasFeaturesCol
featuresCol
in interface HasFeaturesCol
public final DoubleParam fpr()
SelectorParams
fpr
in interface SelectorParams
public final DoubleParam fwe()
SelectorParams
fwe
in interface SelectorParams
public final Param<String> labelCol()
HasLabelCol
labelCol
in interface HasLabelCol
public final IntParam numTopFeatures()
SelectorParams
numTopFeatures
in interface SelectorParams
public final Param<String> outputCol()
HasOutputCol
outputCol
in interface HasOutputCol
public final DoubleParam percentile()
SelectorParams
percentile
in interface SelectorParams
public final Param<String> selectorType()
SelectorParams
selectorType
in interface SelectorParams