|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
Object org.apache.spark.broadcast.Broadcast<T>
public abstract class Broadcast<T>
A broadcast variable. Broadcast variables allow the programmer to keep a read-only variable cached on each machine rather than shipping a copy of it with tasks. They can be used, for example, to give every node a copy of a large input dataset in an efficient manner. Spark also attempts to distribute broadcast variables using efficient broadcast algorithms to reduce communication cost.
Broadcast variables are created from a variable v
by calling
SparkContext.broadcast(T, scala.reflect.ClassTag
.
The broadcast variable is a wrapper around v
, and its value can be accessed by calling the
value
method. The interpreter session below shows this:
scala> val broadcastVar = sc.broadcast(Array(1, 2, 3))
broadcastVar: org.apache.spark.broadcast.Broadcast[Array[Int} = Broadcast(0)
scala> broadcastVar.value
res0: Array[Int] = Array(1, 2, 3)
After the broadcast variable is created, it should be used instead of the value v
in any
functions run on the cluster so that v
is not shipped to the nodes more than once.
In addition, the object v
should not be modified after it is broadcast in order to ensure
that all nodes get the same value of the broadcast variable (e.g. if the variable is shipped
to a new node later).
param: id A unique identifier for the broadcast variable.
Constructor Summary | |
---|---|
Broadcast(long id,
scala.reflect.ClassTag<T> evidence$1)
|
Method Summary | |
---|---|
void |
destroy()
Destroy all data and metadata related to this broadcast variable. |
long |
id()
|
String |
toString()
|
void |
unpersist()
Asynchronously delete cached copies of this broadcast on the executors. |
void |
unpersist(boolean blocking)
Delete cached copies of this broadcast on the executors. |
T |
value()
Get the broadcasted value. |
Methods inherited from class Object |
---|
equals, getClass, hashCode, notify, notifyAll, wait, wait, wait |
Methods inherited from interface org.apache.spark.Logging |
---|
initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning |
Constructor Detail |
---|
public Broadcast(long id, scala.reflect.ClassTag<T> evidence$1)
Method Detail |
---|
public long id()
public T value()
public void unpersist()
public void unpersist(boolean blocking)
blocking
- Whether to block until unpersisting has completedpublic void destroy()
public String toString()
toString
in class Object
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |