Package org.apache.spark.sql.pipelines.graph

Class VirtualTableInput

Object
org.apache.spark.sql.pipelines.graph.VirtualTableInput
All Implemented Interfaces:
Serializable, org.apache.spark.internal.Logging, GraphElement, Input, TableInput, scala.Equals, scala.Product
public class VirtualTableInput extends Object implements TableInput, org.apache.spark.internal.Logging, scala.Product, Serializable
A type of TableInput that returns data from a specified schema or from the inferred Flows that write to the table.
See Also:

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

    org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter

  • Constructor Summary

    Constructors
    Constructor
    Description
    VirtualTableInput(org.apache.spark.sql.catalyst.TableIdentifier identifier, scala.Option<StructType> specifiedSchema, scala.collection.immutable.Set<org.apache.spark.sql.catalyst.TableIdentifier> incomingFlowIdentifiers, scala.collection.immutable.Seq<ResolvedFlow> availableFlows)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    abstract static R
    apply(T1 v1, T2 v2, T3 v3, T4 v4)
     
    scala.collection.immutable.Seq<ResolvedFlow>
    availableFlows()
     
    org.apache.spark.sql.catalyst.TableIdentifier
    identifier()
    Returns the unique identifier for this GraphElement.
    scala.collection.immutable.Set<org.apache.spark.sql.catalyst.TableIdentifier>
    incomingFlowIdentifiers()
     
    org.apache.spark.sql.classic.Dataset<Row>
    load(InputReadOptions readOptions)
    Returns a DataFrame that is a result of loading data from this Input.
    QueryOrigin
    origin()
    Contains provenance to tie back this GraphElement to the user code that defined it.
    scala.Option<StructType>
    specifiedSchema()
    The user-specified schema for this table.
    static String
    toString()
     

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface scala.Equals

    canEqual, equals

    Methods inherited from interface org.apache.spark.sql.pipelines.graph.GraphElement

    displayName, spark

    Methods inherited from interface org.apache.spark.internal.Logging

    initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logBasedOnLevel, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext

    Methods inherited from interface scala.Product

    productArity, productElement, productElementName, productElementNames, productIterator, productPrefix

  • Constructor Details

    • VirtualTableInput

      public VirtualTableInput(org.apache.spark.sql.catalyst.TableIdentifier identifier, scala.Option<StructType> specifiedSchema, scala.collection.immutable.Set<org.apache.spark.sql.catalyst.TableIdentifier> incomingFlowIdentifiers, scala.collection.immutable.Seq<ResolvedFlow> availableFlows)

  • Method Details

    • apply

      public abstract static R apply(T1 v1, T2 v2, T3 v3, T4 v4)

    • toString

      public static String toString()

    • identifier

      public org.apache.spark.sql.catalyst.TableIdentifier identifier()
      Description copied from interface: GraphElement
      Returns the unique identifier for this GraphElement.
      Specified by:
      identifier in interface GraphElement

    • specifiedSchema

      public scala.Option<StructType> specifiedSchema()
      Description copied from interface: TableInput
      The user-specified schema for this table.
      Specified by:
      specifiedSchema in interface TableInput

    • incomingFlowIdentifiers

      public scala.collection.immutable.Set<org.apache.spark.sql.catalyst.TableIdentifier> incomingFlowIdentifiers()

    • availableFlows

      public scala.collection.immutable.Seq<ResolvedFlow> availableFlows()

    • origin

      public QueryOrigin origin()
      Description copied from interface: GraphElement
      Contains provenance to tie back this GraphElement to the user code that defined it.

      This must be set when a GraphElement is directly created by some user code. Subsequently, this initial origin must be propagated as is without modification. If this GraphElement is copied or converted to a different type, then this origin must be copied as is.

      Specified by:
      origin in interface GraphElement
      Returns:
      (undocumented)

    • load

      public org.apache.spark.sql.classic.Dataset<Row> load(InputReadOptions readOptions)
      Description copied from interface: Input
      Returns a DataFrame that is a result of loading data from this Input.
      Specified by:
      load in interface Input
      Parameters:
      readOptions - Type of input. Used to determine streaming/batch
      Returns:
      Streaming or batch DataFrame of this Input's data.