Package org.apache.spark.util

Class AccumulatorV2<IN,OUT>

Object
org.apache.spark.util.AccumulatorV2<IN,OUT>
All Implemented Interfaces:
Serializable
Direct Known Subclasses:
CollectionAccumulator, DoubleAccumulator, LongAccumulator, MapperRowCounter
public abstract class AccumulatorV2<IN,OUT> extends Object implements Serializable
The base class for accumulators, that can accumulate inputs of type IN, and produce output of type OUT.

OUT should be a type that can be read atomically (e.g., Int, Long), or thread-safely (e.g., synchronized collections) because it will be read from other threads.

See Also:

  • Constructor Summary

    Constructors
    Constructor
    Description
    AccumulatorV2()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    abstract void
    add(IN v)
    Takes the inputs and accumulates.
    abstract AccumulatorV2<IN,OUT>
    copy()
    Creates a new copy of this accumulator.
    AccumulatorV2<IN,OUT>
    copyAndReset()
    Creates a new copy of this accumulator, which is zero value.
    boolean
    excludeFromHeartbeat()
     
    final long
    id()
    Returns the id of this accumulator, can only be called after registration.
    final boolean
    isRegistered()
    Returns true if this accumulator has been registered.
    abstract boolean
    isZero()
    Returns if this accumulator is zero value or not.
    abstract void
    merge(AccumulatorV2<IN,OUT> other)
    Merges another same-type accumulator into this one and update its state, i.e.
    final scala.Option<String>
    name()
    Returns the name of this accumulator, can only be called after registration.
    abstract void
    reset()
    Resets this accumulator, which is zero value.
    String
    toString()
     
    abstract OUT
    value()
    Defines the current value of this accumulator

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

  • Constructor Details

    • AccumulatorV2

      public AccumulatorV2()

  • Method Details

    • add

      public abstract void add(IN v)
      Takes the inputs and accumulates.
      Parameters:
      v - (undocumented)

    • copy

      public abstract AccumulatorV2<IN,OUT> copy()
      Creates a new copy of this accumulator.
      Returns:
      (undocumented)

    • copyAndReset

      public AccumulatorV2<IN,OUT> copyAndReset()
      Creates a new copy of this accumulator, which is zero value. i.e. call isZero on the copy must return true.
      Returns:
      (undocumented)

    • excludeFromHeartbeat

      public boolean excludeFromHeartbeat()

    • id

      public final long id()
      Returns the id of this accumulator, can only be called after registration.
      Returns:
      (undocumented)

    • isRegistered

      public final boolean isRegistered()
      Returns true if this accumulator has been registered.

      Returns:
      (undocumented)
      Note:
      All accumulators must be registered before use, or it will throw exception.

    • isZero

      public abstract boolean isZero()
      Returns if this accumulator is zero value or not. e.g. for a counter accumulator, 0 is zero value; for a list accumulator, Nil is zero value.
      Returns:
      (undocumented)

    • merge

      public abstract void merge(AccumulatorV2<IN,OUT> other)
      Merges another same-type accumulator into this one and update its state, i.e. this should be merge-in-place.
      Parameters:
      other - (undocumented)

    • name

      public final scala.Option<String> name()
      Returns the name of this accumulator, can only be called after registration.
      Returns:
      (undocumented)

    • reset

      public abstract void reset()
      Resets this accumulator, which is zero value. i.e. call isZero must return true.

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • value

      public abstract OUT value()
      Defines the current value of this accumulator
      Returns:
      (undocumented)