Package org.apache.spark.sql.pipelines.graph

Interface FlowFunction

All Superinterfaces:
org.apache.spark.internal.Logging
public interface FlowFunction extends org.apache.spark.internal.Logging
A wrapper for the lambda function that defines a Flow.

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

    org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter

  • Method Summary

    Modifier and Type
    Method
    Description
    FlowFunctionResult
    call(scala.collection.immutable.Set<org.apache.spark.sql.catalyst.TableIdentifier> allInputs, scala.collection.immutable.Seq<Input> availableInputs, scala.collection.immutable.Map<String,String> configuration, QueryContext queryContext)
    This function defines the transformations performed by a flow, expressed as a DataFrame.

    Methods inherited from interface org.apache.spark.internal.Logging

    initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logBasedOnLevel, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext

  • Method Details

    • call

      FlowFunctionResult call(scala.collection.immutable.Set<org.apache.spark.sql.catalyst.TableIdentifier> allInputs, scala.collection.immutable.Seq<Input> availableInputs, scala.collection.immutable.Map<String,String> configuration, QueryContext queryContext)
      This function defines the transformations performed by a flow, expressed as a DataFrame.

      Parameters:
      allInputs - the set of identifiers for all the Inputs defined in the DataflowGraph.
      availableInputs - the list of all Inputs available to this flow
      configuration - the spark configurations that apply to this flow.
      queryContext - The context of the query being evaluated.
      Returns:
      the inputs actually used, and the DataFrame expression for the flow