Package org.apache.spark.ml.clustering

Class LDAModel

Object
org.apache.spark.ml.PipelineStage
org.apache.spark.ml.Transformer
org.apache.spark.ml.Model<LDAModel>
org.apache.spark.ml.clustering.LDAModel
All Implemented Interfaces:
Serializable, org.apache.spark.internal.Logging, LDAParams, Params, HasCheckpointInterval, HasFeaturesCol, HasMaxIter, HasSeed, Identifiable, MLWritable, scala.Serializable
Direct Known Subclasses:
DistributedLDAModel, LocalLDAModel
public abstract class LDAModel extends Model<LDAModel> implements LDAParams, org.apache.spark.internal.Logging, MLWritable
Model fitted by LDA.

param: vocabSize Vocabulary size (number of terms or words in the vocabulary) param: sparkSession Used to construct local DataFrames for returning query results

See Also:

  • Method Details

    • checkpointInterval

      public final IntParam checkpointInterval()
      Description copied from interface: HasCheckpointInterval
      Param for set checkpoint interval (&gt;= 1) or disable checkpoint (-1). E.g. 10 means that the cache will get checkpointed every 10 iterations. Note: this setting will be ignored if the checkpoint directory is not set in the SparkContext.
      Specified by:
      checkpointInterval in interface HasCheckpointInterval
      Returns:
      (undocumented)

    • describeTopics

      public Dataset<Row> describeTopics(int maxTermsPerTopic)
      Return the topics described by their top-weighted terms.

      Parameters:
      maxTermsPerTopic - Maximum number of terms to collect for each topic. Default value of 10.
      Returns:
      Local DataFrame with one topic per Row, with columns: - "topic": IntegerType: topic index - "termIndices": ArrayType(IntegerType): term indices, sorted in order of decreasing term importance - "termWeights": ArrayType(DoubleType): corresponding sorted term weights

    • describeTopics

      public Dataset<Row> describeTopics()

    • docConcentration

      public final DoubleArrayParam docConcentration()
      Description copied from interface: LDAParams
      Concentration parameter (commonly named "alpha") for the prior placed on documents' distributions over topics ("theta").

      This is the parameter to a Dirichlet distribution, where larger values mean more smoothing (more regularization).

      If not set by the user, then docConcentration is set automatically. If set to singleton vector [alpha], then alpha is replicated to a vector of length k in fitting. Otherwise, the LDAParams.docConcentration() vector must be length k. (default = automatic)

      Optimizer-specific parameter settings: - EM - Currently only supports symmetric distributions, so all values in the vector should be the same. - Values should be greater than 1.0 - default = uniformly (50 / k) + 1, where 50/k is common in LDA libraries and +1 follows from Asuncion et al. (2009), who recommend a +1 adjustment for EM. - Online - Values should be greater than or equal to 0 - default = uniformly (1.0 / k), following the implementation from here.

      Specified by:
      docConcentration in interface LDAParams
      Returns:
      (undocumented)

    • estimatedDocConcentration

      public Vector estimatedDocConcentration()
      Value for docConcentration() estimated from data. If Online LDA was used and optimizeDocConcentration() was set to false, then this returns the fixed (given) value for the docConcentration() parameter.
      Returns:
      (undocumented)

    • featuresCol

      public final Param<String> featuresCol()
      Description copied from interface: HasFeaturesCol
      Param for features column name.
      Specified by:
      featuresCol in interface HasFeaturesCol
      Returns:
      (undocumented)

    • isDistributed

      public abstract boolean isDistributed()
      Indicates whether this instance is of type DistributedLDAModel

    • k

      public final IntParam k()
      Description copied from interface: LDAParams
      Param for the number of topics (clusters) to infer. Must be &gt; 1. Default: 10.

      Specified by:
      k in interface LDAParams
      Returns:
      (undocumented)

    • keepLastCheckpoint

      public final BooleanParam keepLastCheckpoint()
      Description copied from interface: LDAParams
      For EM optimizer only: LDAParams.optimizer() = "em".

      If using checkpointing, this indicates whether to keep the last checkpoint. If false, then the checkpoint will be deleted. Deleting the checkpoint can cause failures if a data partition is lost, so set this bit with care. Note that checkpoints will be cleaned up via reference counting, regardless.

      See DistributedLDAModel.getCheckpointFiles for getting remaining checkpoints and DistributedLDAModel.deleteCheckpointFiles for removing remaining checkpoints.

      Default: true

      Specified by:
      keepLastCheckpoint in interface LDAParams
      Returns:
      (undocumented)

    • learningDecay

      public final DoubleParam learningDecay()
      Description copied from interface: LDAParams
      For Online optimizer only: LDAParams.optimizer() = "online".

      Learning rate, set as an exponential decay rate. This should be between (0.5, 1.0] to guarantee asymptotic convergence. This is called "kappa" in the Online LDA paper (Hoffman et al., 2010). Default: 0.51, based on Hoffman et al.

      Specified by:
      learningDecay in interface LDAParams
      Returns:
      (undocumented)

    • learningOffset

      public final DoubleParam learningOffset()
      Description copied from interface: LDAParams
      For Online optimizer only: LDAParams.optimizer() = "online".

      A (positive) learning parameter that downweights early iterations. Larger values make early iterations count less. This is called "tau0" in the Online LDA paper (Hoffman et al., 2010) Default: 1024, following Hoffman et al.

      Specified by:
      learningOffset in interface LDAParams
      Returns:
      (undocumented)

    • logLikelihood

      public double logLikelihood(Dataset<?> dataset)
      Calculates a lower bound on the log likelihood of the entire corpus.

      See Equation (16) in the Online LDA paper (Hoffman et al., 2010).

      WARNING: If this model is an instance of DistributedLDAModel (produced when optimizer() is set to "em"), this involves collecting a large topicsMatrix() to the driver. This implementation may be changed in the future.

      Parameters:
      dataset - test corpus to use for calculating log likelihood
      Returns:
      variational lower bound on the log likelihood of the entire corpus

    • logPerplexity

      public double logPerplexity(Dataset<?> dataset)
      Calculate an upper bound on perplexity. (Lower is better.) See Equation (16) in the Online LDA paper (Hoffman et al., 2010).

      WARNING: If this model is an instance of DistributedLDAModel (produced when optimizer() is set to "em"), this involves collecting a large topicsMatrix() to the driver. This implementation may be changed in the future.

      Parameters:
      dataset - test corpus to use for calculating perplexity
      Returns:
      Variational upper bound on log perplexity per token.

    • maxIter

      public final IntParam maxIter()
      Description copied from interface: HasMaxIter
      Param for maximum number of iterations (&gt;= 0).
      Specified by:
      maxIter in interface HasMaxIter
      Returns:
      (undocumented)

    • optimizeDocConcentration

      public final BooleanParam optimizeDocConcentration()
      Description copied from interface: LDAParams
      For Online optimizer only (currently): LDAParams.optimizer() = "online".

      Indicates whether the docConcentration (Dirichlet parameter for document-topic distribution) will be optimized during training. Setting this to true will make the model more expressive and fit the training data better. Default: false

      Specified by:
      optimizeDocConcentration in interface LDAParams
      Returns:
      (undocumented)

    • optimizer

      public final Param<String> optimizer()
      Description copied from interface: LDAParams
      Optimizer or inference algorithm used to estimate the LDA model. Currently supported (case-insensitive): - "online": Online Variational Bayes (default) - "em": Expectation-Maximization

      For details, see the following papers: - Online LDA: Hoffman, Blei and Bach. "Online Learning for Latent Dirichlet Allocation." Neural Information Processing Systems, 2010. See here - EM: Asuncion et al. "On Smoothing and Inference for Topic Models." Uncertainty in Artificial Intelligence, 2009. See here

      Specified by:
      optimizer in interface LDAParams
      Returns:
      (undocumented)

    • seed

      public final LongParam seed()
      Description copied from interface: HasSeed
      Param for random seed.
      Specified by:
      seed in interface HasSeed
      Returns:
      (undocumented)

    • setFeaturesCol

      public LDAModel setFeaturesCol(String value)
      The features for LDA should be a Vector representing the word counts in a document. The vector should be of length vocabSize, with counts for each term (word).

      Parameters:
      value - (undocumented)
      Returns:
      (undocumented)

    • setSeed

      public LDAModel setSeed(long value)

    • setTopicDistributionCol

      public LDAModel setTopicDistributionCol(String value)

    • subsamplingRate

      public final DoubleParam subsamplingRate()
      Description copied from interface: LDAParams
      For Online optimizer only: LDAParams.optimizer() = "online".

      Fraction of the corpus to be sampled and used in each iteration of mini-batch gradient descent, in range (0, 1].

      Note that this should be adjusted in synch with LDA.maxIter so the entire corpus is used. Specifically, set both so that maxIterations * miniBatchFraction greater than or equal to 1.

      Note: This is the same as the miniBatchFraction parameter in OnlineLDAOptimizer.

      Default: 0.05, i.e., 5% of total documents.

      Specified by:
      subsamplingRate in interface LDAParams
      Returns:
      (undocumented)

    • supportedOptimizers

      public final String[] supportedOptimizers()
      Description copied from interface: LDAParams
      Supported values for Param LDAParams.optimizer().
      Specified by:
      supportedOptimizers in interface LDAParams

    • topicConcentration

      public final DoubleParam topicConcentration()
      Description copied from interface: LDAParams
      Concentration parameter (commonly named "beta" or "eta") for the prior placed on topics' distributions over terms.

      This is the parameter to a symmetric Dirichlet distribution.

      Note: The topics' distributions over terms are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many later papers such as Asuncion et al., 2009.

      If not set by the user, then topicConcentration is set automatically. (default = automatic)

      Optimizer-specific parameter settings: - EM - Value should be greater than 1.0 - default = 0.1 + 1, where 0.1 gives a small amount of smoothing and +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM. - Online - Value should be greater than or equal to 0 - default = (1.0 / k), following the implementation from here.

      Specified by:
      topicConcentration in interface LDAParams
      Returns:
      (undocumented)

    • topicDistributionCol

      public final Param<String> topicDistributionCol()
      Description copied from interface: LDAParams
      Output column with estimates of the topic mixture distribution for each document (often called "theta" in the literature). Returns a vector of zeros for an empty document.

      This uses a variational approximation following Hoffman et al. (2010), where the approximate distribution is called "gamma." Technically, this method returns this approximation "gamma" for each document.

      Specified by:
      topicDistributionCol in interface LDAParams
      Returns:
      (undocumented)

    • topicsMatrix

      public Matrix topicsMatrix()
      Inferred topics, where each topic is represented by a distribution over terms. This is a matrix of size vocabSize x k, where each column is a topic. No guarantees are given about the ordering of the topics.

      WARNING: If this model is actually a DistributedLDAModel instance produced by the Expectation-Maximization ("em") optimizer(), then this method could involve collecting a large amount of data to the driver (on the order of vocabSize x k).

      Returns:
      (undocumented)

    • transform

      public Dataset<Row> transform(Dataset<?> dataset)
      Transforms the input dataset.

      WARNING: If this model is an instance of DistributedLDAModel (produced when optimizer() is set to "em"), this involves collecting a large topicsMatrix() to the driver. This implementation may be changed in the future.

      Specified by:
      transform in class Transformer
      Parameters:
      dataset - (undocumented)
      Returns:
      (undocumented)

    • transformSchema

      public StructType transformSchema(StructType schema)
      Description copied from class: PipelineStage
      Check transform validity and derive the output schema from the input schema.

      We check validity for interactions between parameters during transformSchema and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled by Param.validate().

      Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.

      Specified by:
      transformSchema in class PipelineStage
      Parameters:
      schema - (undocumented)
      Returns:
      (undocumented)

    • uid

      public String uid()
      Description copied from interface: Identifiable
      An immutable unique ID for the object and its derivatives.
      Specified by:
      uid in interface Identifiable
      Returns:
      (undocumented)

    • vocabSize

      public int vocabSize()