public class StopWordsRemover extends Transformer implements HasInputCol, HasOutputCol, HasInputCols, HasOutputCols, DefaultParamsWritable
Since 3.0.0, StopWordsRemover
can filter out multiple columns at once by setting the
inputCols
parameter. Note that when both the inputCol
and inputCols
parameters are set,
an Exception will be thrown.
Constructor and Description |
---|
StopWordsRemover() |
StopWordsRemover(String uid) |
Modifier and Type | Method and Description |
---|---|
BooleanParam |
caseSensitive()
Whether to do a case sensitive comparison over the stop words.
|
StopWordsRemover |
copy(ParamMap extra)
Creates a copy of this instance with the same UID and some extra params.
|
boolean |
getCaseSensitive() |
String |
getLocale() |
String[] |
getStopWords() |
Param<String> |
inputCol()
Param for input column name.
|
StringArrayParam |
inputCols()
Param for input column names.
|
static StopWordsRemover |
load(String path) |
static String[] |
loadDefaultStopWords(String language)
Loads the default stop words for the given language.
|
Param<String> |
locale()
Locale of the input for case insensitive matching.
|
Param<String> |
outputCol()
Param for output column name.
|
StringArrayParam |
outputCols()
Param for output column names.
|
static MLReader<T> |
read() |
StopWordsRemover |
setCaseSensitive(boolean value) |
StopWordsRemover |
setInputCol(String value) |
StopWordsRemover |
setInputCols(String[] value) |
StopWordsRemover |
setLocale(String value) |
StopWordsRemover |
setOutputCol(String value) |
StopWordsRemover |
setOutputCols(String[] value) |
StopWordsRemover |
setStopWords(String[] value) |
StringArrayParam |
stopWords()
The words to be filtered out.
|
String |
toString() |
Dataset<Row> |
transform(Dataset<?> dataset)
Transforms the input dataset.
|
StructType |
transformSchema(StructType schema)
Check transform validity and derive the output schema from the input schema.
|
String |
uid()
An immutable unique ID for the object and its derivatives.
|
transform, transform, transform
params
getInputCol
getOutputCol
getInputCols
getOutputCols
clear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, onParamChange, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn
write
save
$init$, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, initLock, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, uninitialize
public StopWordsRemover(String uid)
public StopWordsRemover()
public static StopWordsRemover load(String path)
public static String[] loadDefaultStopWords(String language)
language
- (undocumented)public static MLReader<T> read()
public final StringArrayParam outputCols()
HasOutputCols
outputCols
in interface HasOutputCols
public final StringArrayParam inputCols()
HasInputCols
inputCols
in interface HasInputCols
public final Param<String> outputCol()
HasOutputCol
outputCol
in interface HasOutputCol
public final Param<String> inputCol()
HasInputCol
inputCol
in interface HasInputCol
public String uid()
Identifiable
uid
in interface Identifiable
public StopWordsRemover setInputCol(String value)
public StopWordsRemover setOutputCol(String value)
public StopWordsRemover setInputCols(String[] value)
public StopWordsRemover setOutputCols(String[] value)
public StringArrayParam stopWords()
StopWordsRemover.loadDefaultStopWords()
public StopWordsRemover setStopWords(String[] value)
public String[] getStopWords()
public BooleanParam caseSensitive()
public StopWordsRemover setCaseSensitive(boolean value)
public boolean getCaseSensitive()
public Param<String> locale()
caseSensitive
is true.
Default: the string of default locale (Locale.getDefault
), or Locale.US
if default locale
is not in available locales in JVM.public StopWordsRemover setLocale(String value)
public String getLocale()
public Dataset<Row> transform(Dataset<?> dataset)
Transformer
transform
in class Transformer
dataset
- (undocumented)public StructType transformSchema(StructType schema)
PipelineStage
We check validity for interactions between parameters during transformSchema
and
raise an exception if any parameter value is invalid. Parameter value checks which
do not depend on other parameters are handled by Param.validate()
.
Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.
transformSchema
in class PipelineStage
schema
- (undocumented)public StopWordsRemover copy(ParamMap extra)
Params
defaultCopy()
.copy
in interface Params
copy
in class Transformer
extra
- (undocumented)public String toString()
toString
in interface Identifiable
toString
in class Object