Interface SupportsRuntimeFiltering
- All Superinterfaces:
Scan
,SupportsRuntimeV2Filtering
Scan
. Data sources can implement this interface if they can
filter initially planned InputPartition
s using predicates Spark infers at runtime.
Note that Spark will push runtime filters only if they are beneficial.
- Since:
- 3.2.0
-
Nested Class Summary
Nested classes/interfaces inherited from interface org.apache.spark.sql.connector.read.Scan
Scan.ColumnarSupportMode
-
Method Summary
Modifier and TypeMethodDescriptiondefault void
Filters this scan using runtime predicates.void
Filters this scan using runtime filters.Returns attributes this scan can be filtered by at runtime.Methods inherited from interface org.apache.spark.sql.connector.read.Scan
columnarSupportMode, description, readSchema, reportDriverMetrics, supportedCustomMetrics, toBatch, toContinuousStream, toMicroBatchStream
-
Method Details
-
filterAttributes
NamedReference[] filterAttributes()Returns attributes this scan can be filtered by at runtime.Spark will call
filter(Filter[])
if it can derive a runtime predicate for any of the filter attributes.- Specified by:
filterAttributes
in interfaceSupportsRuntimeV2Filtering
-
filter
Filters this scan using runtime filters.The provided expressions must be interpreted as a set of filters that are ANDed together. Implementations may use the filters to prune initially planned
InputPartition
s.If the scan also implements
SupportsReportPartitioning
, it must preserve the originally reported partitioning during runtime filtering. While applying runtime filters, the scan may detect that someInputPartition
s have no matching data. It can omit such partitions entirely only if it does not report a specific partitioning. Otherwise, the scan can replace the initially plannedInputPartition
s that have no matching data with emptyInputPartition
s but must preserve the overall number of partitions.Note that Spark will call
Scan.toBatch()
again after filtering the scan at runtime.- Parameters:
filters
- data source filters used to filter the scan at runtime
-
filter
Description copied from interface:SupportsRuntimeV2Filtering
Filters this scan using runtime predicates.The provided expressions must be interpreted as a set of predicates that are ANDed together. Implementations may use the predicates to prune initially planned
InputPartition
s.If the scan also implements
SupportsReportPartitioning
, it must preserve the originally reported partitioning during runtime filtering. While applying runtime predicates, the scan may detect that someInputPartition
s have no matching data. It can omit such partitions entirely only if it does not report a specific partitioning. Otherwise, the scan can replace the initially plannedInputPartition
s that have no matching data with emptyInputPartition
s but must preserve the overall number of partitions.Note that Spark will call
Scan.toBatch()
again after filtering the scan at runtime.- Specified by:
filter
in interfaceSupportsRuntimeV2Filtering
- Parameters:
predicates
- data source V2 predicates used to filter the scan at runtime
-