Package org.apache.spark.sql.pipelines.graph

Interface Flow

All Superinterfaces:
GraphElement, org.apache.spark.internal.Logging
All Known Subinterfaces:
ResolutionCompletedFlow, ResolvedFlow
All Known Implementing Classes:
AppendOnceFlow, CompleteFlow, ResolutionFailedFlow, StreamingFlow, UnresolvedFlow
public interface Flow extends GraphElement, org.apache.spark.internal.Logging
A Flow is a node of data transformation in a dataflow graph. It describes the movement of data into a particular dataset.

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

    org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter

  • Method Summary

    Modifier and Type
    Method
    Description
    scala.Option<String>
    comment()
    The comment associated with this flow
    org.apache.spark.sql.catalyst.TableIdentifier
    destinationIdentifier()
    The dataset that this Flow represents a write to.
    FlowFunction
    func()
    The FlowFunction containing the user's query.
    org.apache.spark.sql.catalyst.TableIdentifier
    identifier()
    Returns the unique identifier for this GraphElement.
    boolean
    once()
    Whether this is a ONCE flow.
    QueryContext
    queryContext()
    The current query context (catalog and database) when the query is defined.
    scala.collection.immutable.Map<String,String>
    sqlConf()
     

    Methods inherited from interface org.apache.spark.sql.pipelines.graph.GraphElement

    displayName, origin, spark

    Methods inherited from interface org.apache.spark.internal.Logging

    initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logBasedOnLevel, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext

  • Method Details

    • comment

      scala.Option<String> comment()
      The comment associated with this flow

    • destinationIdentifier

      org.apache.spark.sql.catalyst.TableIdentifier destinationIdentifier()
      The dataset that this Flow represents a write to.
      Returns:
      (undocumented)

    • func

      FlowFunction func()
      The FlowFunction containing the user's query.

    • identifier

      org.apache.spark.sql.catalyst.TableIdentifier identifier()
      Description copied from interface: GraphElement
      Returns the unique identifier for this GraphElement.
      Specified by:
      identifier in interface GraphElement

    • once

      boolean once()
      Whether this is a ONCE flow. ONCE flows should run only once per full refresh.
      Returns:
      (undocumented)

    • queryContext

      QueryContext queryContext()
      The current query context (catalog and database) when the query is defined.

    • sqlConf

      scala.collection.immutable.Map<String,String> sqlConf()