Package org.apache.spark.sql.pipelines.graph

Class Table

Object
org.apache.spark.sql.pipelines.graph.Table
All Implemented Interfaces:
Serializable, GraphElement, Input, Output, TableInput, scala.Equals, scala.Product
public class Table extends Object implements TableInput, Output, scala.Product, Serializable
A table representing a materialized dataset in a DataflowGraph.

param: identifier The identifier of this table within the graph. param: specifiedSchema The user-specified schema for this table. param: partitionCols What columns the table should be partitioned by when materialized. param: normalizedPath Normalized storage location for the table based on the user-specified table path (if not defined, we will normalize a managed storage path for it). param: properties Table Properties to set in table metadata. param: comment User-specified comment that can be placed on the table. param: isStreamingTable if the table is a streaming table, as defined by the source code.

See Also:

  • Constructor Details

    • Table

      public Table(org.apache.spark.sql.catalyst.TableIdentifier identifier, scala.Option<StructType> specifiedSchema, scala.Option<scala.collection.immutable.Seq<String>> partitionCols, scala.Option<String> normalizedPath, scala.collection.immutable.Map<String,String> properties, scala.Option<String> comment, QueryOrigin baseOrigin, boolean isStreamingTable, scala.Option<String> format)

  • Method Details

    • apply

      public abstract static R apply(T1 v1, T2 v2, T3 v3, T4 v4, T5 v5, T6 v6, T7 v7, T8 v8, T9 v9)

    • toString

      public static String toString()

    • identifier

      public org.apache.spark.sql.catalyst.TableIdentifier identifier()
      Description copied from interface: GraphElement
      Returns the unique identifier for this GraphElement.
      Specified by:
      identifier in interface GraphElement

    • specifiedSchema

      public scala.Option<StructType> specifiedSchema()
      Description copied from interface: TableInput
      The user-specified schema for this table.
      Specified by:
      specifiedSchema in interface TableInput

    • partitionCols

      public scala.Option<scala.collection.immutable.Seq<String>> partitionCols()

    • normalizedPath

      public scala.Option<String> normalizedPath()
      Description copied from interface: Output
      Normalized storage location used for storing materializations for this Output. If None, it means this Output has not been normalized yet.
      Specified by:
      normalizedPath in interface Output
      Returns:
      (undocumented)

    • properties

      public scala.collection.immutable.Map<String,String> properties()

    • comment

      public scala.Option<String> comment()

    • baseOrigin

      public QueryOrigin baseOrigin()

    • isStreamingTable

      public boolean isStreamingTable()

    • format

      public scala.Option<String> format()

    • origin

      public QueryOrigin origin()
      Description copied from interface: GraphElement
      Contains provenance to tie back this GraphElement to the user code that defined it.

      This must be set when a GraphElement is directly created by some user code. Subsequently, this initial origin must be propagated as is without modification. If this GraphElement is copied or converted to a different type, then this origin must be copied as is.

      Specified by:
      origin in interface GraphElement
      Returns:
      (undocumented)

    • load

      public org.apache.spark.sql.classic.Dataset<Row> load(InputReadOptions readOptions)
      Description copied from interface: Input
      Returns a DataFrame that is a result of loading data from this Input.
      Specified by:
      load in interface Input
      Parameters:
      readOptions - Type of input. Used to determine streaming/batch
      Returns:
      Streaming or batch DataFrame of this Input's data.

    • path

      public String path()
      Returns the normalized storage location to this Table.
      Specified by:
      path in interface Output
      Returns:
      (undocumented)

    • datasetType

      public DatasetType datasetType()
      Get the DatasetType of the table
      Returns:
      (undocumented)