public class StreamingKMeansModel extends KMeansModel implements org.apache.spark.internal.Logging
The update algorithm uses the "mini-batch" KMeans rule, generalized to incorporate forgetfulness (i.e. decay). The update rule (for each cluster) is:
$$ \begin{align} c_{t+1} &= [(c_t * n_t * a) + (x_t * m_t)] / [n_t + m_t] \\ n_{t+1} &= n_t * a + m_t \end{align} $$
Where c_t is the previously estimated centroid for that cluster, n_t is the number of points assigned to it thus far, x_t is the centroid estimated on the current batch, and m_t is the number of points assigned to that centroid in the current batch.
The decay factor 'a' scales the contribution of the clusters as estimated thus far, by applying a as a discount weighting on the current point when evaluating new incoming data. If a=1, all batches are weighted equally. If a=0, new centroids are determined entirely by recent data. Lower values correspond to more forgetting.
Decay can optionally be specified by a half life and associated time unit. The time unit can either be a batch of data or a single data point. Considering data arrived at time t, the half life h is defined such that at time t + h the discount applied to the data from t is 0.5. The definition remains the same whether the time unit is given as batches or points.
KMeansModel.Cluster$, KMeansModel.SaveLoadV1_0$, KMeansModel.SaveLoadV2_0$| Constructor and Description |
|---|
StreamingKMeansModel(Vector[] clusterCenters,
double[] clusterWeights) |
| Modifier and Type | Method and Description |
|---|---|
Vector[] |
clusterCenters() |
double[] |
clusterWeights() |
StreamingKMeansModel |
update(RDD<Vector> data,
double decayFactor,
String timeUnit)
Perform a k-means update on a batch of data.
|
computeCost, distanceMeasure, k, load, predict, predict, predict, save, trainingCostequals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait$init$, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, initLock, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, uninitializepublic StreamingKMeansModel(Vector[] clusterCenters, double[] clusterWeights)
public Vector[] clusterCenters()
clusterCenters in class KMeansModelpublic double[] clusterWeights()
public StreamingKMeansModel update(RDD<Vector> data, double decayFactor, String timeUnit)
data - (undocumented)decayFactor - (undocumented)timeUnit - (undocumented)