org.apache.spark.mllib.stat

Class Statistics

Object

org.apache.spark.mllib.stat.Statistics

public class Statistics
extends Object

API for statistical functions in MLlib.

Constructor Summary

Constructors

Constructor and Description
Statistics()

Method Summary

Modifier and Type	Method and Description
static ChiSqTestResult[]	chiSqTest(JavaRDD<LabeledPoint> data) Java-friendly version of chiSqTest()
static ChiSqTestResult	chiSqTest(Matrix observed) Conduct Pearson's independence test on the input contingency matrix, which cannot contain negative entries or columns or rows that sum up to 0.
static ChiSqTestResult[]	chiSqTest(RDD<LabeledPoint> data) Conduct Pearson's independence test for every feature against the label across the input RDD.
static ChiSqTestResult	chiSqTest(Vector observed) Conduct Pearson's chi-squared goodness of fit test of the observed data against the uniform distribution, with each category having an expected frequency of 1 / observed.size.
static ChiSqTestResult	chiSqTest(Vector observed, Vector expected) Conduct Pearson's chi-squared goodness of fit test of the observed data against the expected distribution.
static MultivariateStatisticalSummary	colStats(RDD<Vector> X) Computes column-wise summary statistics for the input RDD[Vector].
static double	corr(JavaRDD<Double> x, JavaRDD<Double> y) Java-friendly version of corr()
static double	corr(JavaRDD<Double> x, JavaRDD<Double> y, String method) Java-friendly version of corr()
static double	corr(RDD<Object> x, RDD<Object> y) Compute the Pearson correlation for the input RDDs.
static double	corr(RDD<Object> x, RDD<Object> y, String method) Compute the correlation for the input RDDs using the specified method.
static Matrix	corr(RDD<Vector> X) Compute the Pearson correlation matrix for the input RDD of Vectors.
static Matrix	corr(RDD<Vector> X, String method) Compute the correlation matrix for the input RDD of Vectors using the specified method.
static KolmogorovSmirnovTestResult	kolmogorovSmirnovTest(JavaDoubleRDD data, String distName, double... params) Java-friendly version of kolmogorovSmirnovTest()
static KolmogorovSmirnovTestResult	kolmogorovSmirnovTest(JavaDoubleRDD data, String distName, scala.collection.Seq<Object> params) Java-friendly version of kolmogorovSmirnovTest()
static KolmogorovSmirnovTestResult	kolmogorovSmirnovTest(RDD<Object> data, scala.Function1<Object,Object> cdf) Conduct the two-sided Kolmogorov-Smirnov (KS) test for data sampled from a continuous distribution.
static KolmogorovSmirnovTestResult	kolmogorovSmirnovTest(RDD<Object> data, String distName, double... params) Convenience function to conduct a one-sample, two-sided Kolmogorov-Smirnov test for probability distribution equality.
static KolmogorovSmirnovTestResult	kolmogorovSmirnovTest(RDD<Object> data, String distName, scala.collection.Seq<Object> params) Convenience function to conduct a one-sample, two-sided Kolmogorov-Smirnov test for probability distribution equality.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Statistics

public Statistics()

Method Detail

kolmogorovSmirnovTest

public static KolmogorovSmirnovTestResult kolmogorovSmirnovTest(RDD<Object> data,
                                                                String distName,
                                                                double... params)

Convenience function to conduct a one-sample, two-sided Kolmogorov-Smirnov test for probability distribution equality. Currently supports the normal distribution, taking as parameters the mean and standard deviation. (distName = "norm")

Parameters:

data - an RDD[Double] containing the sample of data to test

distName - a String name for a theoretical distribution

params - Double* specifying the parameters to be used for the theoretical distribution

Returns:

KolmogorovSmirnovTestResult object containing test statistic, p-value, and null hypothesis.

kolmogorovSmirnovTest

public static KolmogorovSmirnovTestResult kolmogorovSmirnovTest(JavaDoubleRDD data,
                                                                String distName,
                                                                double... params)

Java-friendly version of kolmogorovSmirnovTest()

Parameters:

data - (undocumented)

distName - (undocumented)

params - (undocumented)

Returns:

(undocumented)

colStats

public static MultivariateStatisticalSummary colStats(RDD<Vector> X)

Computes column-wise summary statistics for the input RDD[Vector].

Parameters:

X - an RDD[Vector] for which column-wise summary statistics are to be computed.

Returns:

MultivariateStatisticalSummary object containing column-wise summary statistics.

corr

public static Matrix corr(RDD<Vector> X)

Compute the Pearson correlation matrix for the input RDD of Vectors. Columns with 0 covariance produce NaN entries in the correlation matrix.

Parameters:

X - an RDD[Vector] for which the correlation matrix is to be computed.

Returns:

Pearson correlation matrix comparing columns in X.

corr

public static Matrix corr(RDD<Vector> X,
                          String method)

Compute the correlation matrix for the input RDD of Vectors using the specified method. Methods currently supported: pearson (default), spearman.

Parameters:

X - an RDD[Vector] for which the correlation matrix is to be computed.

method - String specifying the method to use for computing correlation. Supported: pearson (default), spearman

Returns:

Correlation matrix comparing columns in X.

Note:

For Spearman, a rank correlation, we need to create an RDD[Double] for each column and sort it in order to retrieve the ranks and then join the columns back into an RDD[Vector], which is fairly costly. Cache the input RDD before calling corr with method = "spearman" to avoid recomputing the common lineage.

corr

public static double corr(RDD<Object> x,
                          RDD<Object> y)

Compute the Pearson correlation for the input RDDs. Returns NaN if either vector has 0 variance.

Parameters:

x - RDD[Double] of the same cardinality as y.

y - RDD[Double] of the same cardinality as x.

Returns:

A Double containing the Pearson correlation between the two input RDD[Double]s

Note:

The two input RDDs need to have the same number of partitions and the same number of elements in each partition.

corr

public static double corr(JavaRDD<Double> x,
                          JavaRDD<Double> y)

Java-friendly version of corr()

Parameters:

x - (undocumented)

y - (undocumented)

Returns:

(undocumented)

corr

public static double corr(RDD<Object> x,
                          RDD<Object> y,
                          String method)

Compute the correlation for the input RDDs using the specified method. Methods currently supported: pearson (default), spearman.

Parameters:

x - RDD[Double] of the same cardinality as y.

y - RDD[Double] of the same cardinality as x.

method - String specifying the method to use for computing correlation. Supported: pearson (default), spearman

Returns:

A Double containing the correlation between the two input RDD[Double]s using the specified method.

Note:

The two input RDDs need to have the same number of partitions and the same number of elements in each partition.

corr

public static double corr(JavaRDD<Double> x,
                          JavaRDD<Double> y,
                          String method)

Java-friendly version of corr()

Parameters:

x - (undocumented)

y - (undocumented)

method - (undocumented)

Returns:

(undocumented)

chiSqTest

public static ChiSqTestResult chiSqTest(Vector observed,
                                        Vector expected)

Conduct Pearson's chi-squared goodness of fit test of the observed data against the expected distribution.

Parameters:

observed - Vector containing the observed categorical counts/relative frequencies.

expected - Vector containing the expected categorical counts/relative frequencies. expected is rescaled if the expected sum differs from the observed sum.

Returns:

ChiSquaredTest object containing the test statistic, degrees of freedom, p-value, the method used, and the null hypothesis.

Note:

The two input Vectors need to have the same size. observed cannot contain negative values. expected cannot contain nonpositive values.

chiSqTest

public static ChiSqTestResult chiSqTest(Vector observed)

Conduct Pearson's chi-squared goodness of fit test of the observed data against the uniform distribution, with each category having an expected frequency of 1 / observed.size.

Parameters:

observed - Vector containing the observed categorical counts/relative frequencies.

Returns:

ChiSquaredTest object containing the test statistic, degrees of freedom, p-value, the method used, and the null hypothesis.

Note:

observed cannot contain negative values.

chiSqTest

public static ChiSqTestResult chiSqTest(Matrix observed)

Conduct Pearson's independence test on the input contingency matrix, which cannot contain negative entries or columns or rows that sum up to 0.

Parameters:

observed - The contingency matrix (containing either counts or relative frequencies).

Returns:

ChiSquaredTest object containing the test statistic, degrees of freedom, p-value, the method used, and the null hypothesis.

chiSqTest

public static ChiSqTestResult[] chiSqTest(RDD<LabeledPoint> data)

Conduct Pearson's independence test for every feature against the label across the input RDD. For each feature, the (feature, label) pairs are converted into a contingency matrix for which the chi-squared statistic is computed. All label and feature values must be categorical.

Parameters:

data - an RDD[LabeledPoint] containing the labeled dataset with categorical features. Real-valued features will be treated as categorical for each distinct value.

Returns:

an array containing the ChiSquaredTestResult for every feature against the label. The order of the elements in the returned array reflects the order of input features.

chiSqTest

public static ChiSqTestResult[] chiSqTest(JavaRDD<LabeledPoint> data)

Java-friendly version of chiSqTest()

Parameters:

data - (undocumented)

Returns:

(undocumented)

kolmogorovSmirnovTest

public static KolmogorovSmirnovTestResult kolmogorovSmirnovTest(RDD<Object> data,
                                                                scala.Function1<Object,Object> cdf)

Conduct the two-sided Kolmogorov-Smirnov (KS) test for data sampled from a continuous distribution. By comparing the largest difference between the empirical cumulative distribution of the sample data and the theoretical distribution we can provide a test for the the null hypothesis that the sample data comes from that theoretical distribution. For more information on KS Test:

Parameters:

data - an RDD[Double] containing the sample of data to test

cdf - a Double => Double function to calculate the theoretical CDF at a given value

Returns:

KolmogorovSmirnovTestResult object containing test statistic, p-value, and null hypothesis.

See Also:

Kolmogorov-Smirnov test (Wikipedia)

kolmogorovSmirnovTest

public static KolmogorovSmirnovTestResult kolmogorovSmirnovTest(RDD<Object> data,
                                                                String distName,
                                                                scala.collection.Seq<Object> params)

Convenience function to conduct a one-sample, two-sided Kolmogorov-Smirnov test for probability distribution equality. Currently supports the normal distribution, taking as parameters the mean and standard deviation. (distName = "norm")

Parameters:

data - an RDD[Double] containing the sample of data to test

distName - a String name for a theoretical distribution

params - Double* specifying the parameters to be used for the theoretical distribution

Returns:

KolmogorovSmirnovTestResult object containing test statistic, p-value, and null hypothesis.

kolmogorovSmirnovTest

public static KolmogorovSmirnovTestResult kolmogorovSmirnovTest(JavaDoubleRDD data,
                                                                String distName,
                                                                scala.collection.Seq<Object> params)

Java-friendly version of kolmogorovSmirnovTest()

Parameters:

data - (undocumented)

distName - (undocumented)

params - (undocumented)

Returns:

(undocumented)