Packages

trait DeltaBatchWrite extends BatchWrite

An interface that defines how to write a delta of rows during batch processing.

Annotations
@Experimental()
Source
DeltaBatchWrite.java
Since

3.4.0

Linear Supertypes
BatchWrite, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DeltaBatchWrite
  2. BatchWrite
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Abstract Value Members

  1. abstract def abort(messages: Array[WriterCommitMessage]): Unit

    Aborts this writing job because some data writers are failed and keep failing when retry, or the Spark job fails with some unknown reasons, or #onDataWriterCommit(WriterCommitMessage) fails, or #commit(WriterCommitMessage[]) fails.

    Aborts this writing job because some data writers are failed and keep failing when retry, or the Spark job fails with some unknown reasons, or #onDataWriterCommit(WriterCommitMessage) fails, or #commit(WriterCommitMessage[]) fails.

    If this method fails (by throwing an exception), the underlying data source may require manual cleanup.

    Unless the abort is triggered by the failure of commit, the given messages should have some null slots as there maybe only a few data writers that are committed before the abort happens, or some data writers were committed but their commit messages haven't reached the driver when the abort is triggered. So this is just a "best effort" for data sources to clean up the data left by data writers.

    Definition Classes
    BatchWrite
  2. abstract def commit(messages: Array[WriterCommitMessage]): Unit

    Commits this writing job with a list of commit messages.

    Commits this writing job with a list of commit messages. The commit messages are collected from successful data writers and are produced by DataWriter#commit().

    If this method fails (by throwing an exception), this writing job is considered to to have been failed, and #abort(WriterCommitMessage[]) would be called. The state of the destination is undefined and @#abort(WriterCommitMessage[]) may not be able to deal with it.

    Note that speculative execution may cause multiple tasks to run for a partition. By default, Spark uses the commit coordinator to allow at most one task to commit. Implementations can disable this behavior by overriding #useCommitCoordinator(). If disabled, multiple tasks may have committed successfully and one successful commit message per task will be passed to this commit method. The remaining commit messages are ignored by Spark.

    Definition Classes
    BatchWrite
  3. abstract def createBatchWriterFactory(info: PhysicalWriteInfo): DeltaWriterFactory

    Creates a writer factory which will be serialized and sent to executors.

    Creates a writer factory which will be serialized and sent to executors.

    If this method fails (by throwing an exception), the action will fail and no Spark job will be submitted.

    info

    Physical information about the input data that will be written to this table.

    Definition Classes
    DeltaBatchWriteBatchWrite
    Annotations
    @Override()

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  6. def commit(messages: Array[WriterCommitMessage], metrics: Map[String, Long]): Unit

    Commits this writing job with a list of commit messages and operation metrics.

    Commits this writing job with a list of commit messages and operation metrics.

    If this method fails (by throwing an exception), this writing job is considered to to have been failed, and #abort(WriterCommitMessage[]) would be called. The state of the destination is undefined and @#abort(WriterCommitMessage[]) may not be able to deal with it.

    Note that speculative execution may cause multiple tasks to run for a partition. By default, Spark uses the commit coordinator to allow at most one task to commit. Implementations can disable this behavior by overriding #useCommitCoordinator(). If disabled, multiple tasks may have committed successfully and one successful commit message per task will be passed to this commit method. The remaining commit messages are ignored by Spark.

    messages

    a list of commit messages from successful data writers, produced by DataWriter#commit().

    metrics

    a map of operation metrics collected from the query producing write. The keys will be prefixed by operation type, eg merge. Currently supported metrics are:

    • Operation Type = merge
    • numTargetRowsCopied: number of target rows copied unmodified because they did not match any action
    • numTargetRowsDeleted: number of target rows deleted
    • numTargetRowsUpdated: number of target rows updated
    • numTargetRowsInserted: number of target rows inserted
    • numTargetRowsMatchedUpdated: number of target rows updated by a matched clause
    • numTargetRowsMatchedDeleted: number of target rows deleted by a matched clause
    • numTargetRowsNotMatchedBySourceUpdated: number of target rows updated by a not matched by source clause
    • numTargetRowsNotMatchedBySourceDeleted: number of target rows deleted by a not matched by source clause
    Definition Classes
    BatchWrite
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  9. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  14. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  15. def onDataWriterCommit(message: WriterCommitMessage): Unit

    Handles a commit message on receiving from a successful data writer.

    Handles a commit message on receiving from a successful data writer.

    If this method fails (by throwing an exception), this writing job is considered to to have been failed, and #abort(WriterCommitMessage[]) would be called.

    Definition Classes
    BatchWrite
  16. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  17. def toString(): String
    Definition Classes
    AnyRef → Any
  18. def useCommitCoordinator(): Boolean

    Returns whether Spark should use the commit coordinator to ensure that at most one task for each partition commits.

    Returns whether Spark should use the commit coordinator to ensure that at most one task for each partition commits.

    returns

    true if commit coordinator should be used, false otherwise.

    Definition Classes
    BatchWrite
  19. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  20. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  21. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from BatchWrite

Inherited from AnyRef

Inherited from Any

Ungrouped