Package org.apache.spark.sql.connector.read

Interface PartitionReader<T>

All Superinterfaces:
AutoCloseable, Closeable
All Known Subinterfaces:
ContinuousPartitionReader<T>
@Evolving public interface PartitionReader<T> extends Closeable
A partition reader returned by PartitionReaderFactory.createReader(InputPartition) or PartitionReaderFactory.createColumnarReader(InputPartition). It's responsible for outputting data for a RDD partition.

Note that, Currently the type `T` can only be InternalRow for normal data sources, or ColumnarBatch for columnar data sources(whose PartitionReaderFactory.supportColumnarReads(InputPartition) returns true).

Since:
3.0.0

  • Method Summary

    Modifier and Type
    Method
    Description
    default CustomTaskMetric[]
    currentMetricsValues()
    Returns an array of custom task metrics.
    T
    get()
    Return the current record.
    boolean
    next()
    Proceed to next record, returns false if there is no more records.

    Methods inherited from interface java.io.Closeable

    close

  • Method Details

    • next

      boolean next() throws IOException
      Proceed to next record, returns false if there is no more records.
      Throws:
      IOException - if failure happens during disk/network IO like reading files.

    • get

      T get()
      Return the current record. This method should return same value until `next` is called.

    • currentMetricsValues

      default CustomTaskMetric[] currentMetricsValues()
      Returns an array of custom task metrics. By default it returns empty array. Note that it is not recommended to put heavy logic in this method as it may affect reading performance.