org.apache.spark.sql.connector.read.streaming

Interface ContinuousPartitionReaderFactory

All Superinterfaces:

PartitionReaderFactory, java.io.Serializable

@Evolving
public interface ContinuousPartitionReaderFactory
extends PartitionReaderFactory

A variation on PartitionReaderFactory that returns ContinuousPartitionReader instead of PartitionReader. It's used for continuous streaming processing.

Since:

3.0.0

Method Summary

Modifier and Type	Method and Description
default ContinuousPartitionReader<ColumnarBatch>	createColumnarReader(InputPartition partition) Returns a columnar partition reader to read data from the given InputPartition.
ContinuousPartitionReader<org.apache.spark.sql.catalyst.InternalRow>	createReader(InputPartition partition) Returns a row-based partition reader to read data from the given InputPartition.

Methods inherited from interface org.apache.spark.sql.connector.read.PartitionReaderFactory

supportColumnarReads

Method Detail

createReader

ContinuousPartitionReader<org.apache.spark.sql.catalyst.InternalRow> createReader(InputPartition partition)

Description copied from interface: PartitionReaderFactory

Returns a row-based partition reader to read data from the given InputPartition.

Implementations probably need to cast the input partition to the concrete InputPartition class defined for the data source.

Specified by:

createReader in interface PartitionReaderFactory

createColumnarReader

default ContinuousPartitionReader<ColumnarBatch> createColumnarReader(InputPartition partition)

Description copied from interface: PartitionReaderFactory

Returns a columnar partition reader to read data from the given InputPartition.

Implementations probably need to cast the input partition to the concrete InputPartition class defined for the data source.

Specified by:

createColumnarReader in interface PartitionReaderFactory