org.apache.spark.sql.connector.read.streaming

Interface SupportsAdmissionControl

All Superinterfaces:

SparkDataStream

All Known Subinterfaces:

SupportsTriggerAvailableNow

@Evolving
public interface SupportsAdmissionControl
extends SparkDataStream

A mix-in interface for SparkDataStream streaming sources to signal that they can control the rate of data ingested into the system. These rate limits can come implicitly from the contract of triggers, e.g. Trigger.Once() requires that a micro-batch process all data available to the system at the start of the micro-batch. Alternatively, sources can decide to limit ingest through data source options.

Through this interface, a MicroBatchStream should be able to return the next offset that it will process until given a ReadLimit.

Since:

3.0.0

Method Summary

Modifier and Type	Method and Description
default ReadLimit	getDefaultReadLimit() Returns the read limits potentially passed to the data source through options when creating the data source.
Offset	latestOffset(Offset startOffset, ReadLimit limit) Returns the most recent offset available given a read limit.
default Offset	reportLatestOffset() Returns the most recent offset available.

Methods inherited from interface org.apache.spark.sql.connector.read.streaming.SparkDataStream

commit, deserializeOffset, initialOffset, stop

Method Detail

getDefaultReadLimit

default ReadLimit getDefaultReadLimit()

Returns the read limits potentially passed to the data source through options when creating the data source.

latestOffset

Offset latestOffset(Offset startOffset,
                    ReadLimit limit)

Returns the most recent offset available given a read limit. The start offset can be used to figure out how much new data should be read given the limit. Users should implement this method instead of latestOffset for a MicroBatchStream or getOffset for Source.

When this method is called on a `Source`, the source can return `null` if there is no data to process. In addition, for the very first micro-batch, the `startOffset` will be null as well.

When this method is called on a MicroBatchStream, the `startOffset` will be `initialOffset` for the very first micro-batch. The source can return `null` if there is no data to process.

reportLatestOffset

default Offset reportLatestOffset()

Returns the most recent offset available.

The source can return `null`, if there is no data to process or the source does not support to this method.