Package org.apache.spark.sql.connector.read.streaming

Interface SupportsAdmissionControl

All Superinterfaces:
SparkDataStream
All Known Subinterfaces:
SupportsTriggerAvailableNow
@Evolving public interface SupportsAdmissionControl extends SparkDataStream
A mix-in interface for SparkDataStream streaming sources to signal that they can control the rate of data ingested into the system. These rate limits can come implicitly from the contract of triggers, e.g. Trigger.Once() requires that a micro-batch process all data available to the system at the start of the micro-batch. Alternatively, sources can decide to limit ingest through data source options.

Through this interface, a MicroBatchStream should be able to return the next offset that it will process until given a ReadLimit.

Since:
3.0.0

  • Method Details

    • getDefaultReadLimit

      default ReadLimit getDefaultReadLimit()
      Returns the read limits potentially passed to the data source through options when creating the data source.

    • latestOffset

      Offset latestOffset(Offset startOffset, ReadLimit limit)
      Returns the most recent offset available given a read limit. The start offset can be used to figure out how much new data should be read given the limit. Users should implement this method instead of latestOffset for a MicroBatchStream or getOffset for Source.

      When this method is called on a `Source`, the source can return `null` if there is no data to process. In addition, for the very first micro-batch, the `startOffset` will be null as well.

      When this method is called on a MicroBatchStream, the `startOffset` will be `initialOffset` for the very first micro-batch. The source can return `null` if there is no data to process.

    • reportLatestOffset

      default Offset reportLatestOffset()
      Returns the most recent offset available.

      The source can return `null`, if there is no data to process or the source does not support to this method.