trait SupportsPushDownV2Filters extends ScanBuilder
A mix-in interface for ScanBuilder. Data sources can implement this interface to
push down V2 Predicate to the data source and reduce the size of the data to be read.
Please Note that this interface is preferred over SupportsPushDownFilters, which uses
V1 org.apache.spark.sql.sources.Filter and is less efficient due to the
internal -> external data conversion.
- Annotations
- @Evolving()
- Source
- SupportsPushDownV2Filters.java
- Since
3.3.0
- Alphabetic
- By Inheritance
- SupportsPushDownV2Filters
- ScanBuilder
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def build(): Scan
- Definition Classes
- ScanBuilder
- abstract def pushPredicates(predicates: Array[Predicate]): Array[Predicate]
Pushes down predicates, and returns predicates that need to be evaluated after scanning.
Pushes down predicates, and returns predicates that need to be evaluated after scanning.
Rows should be returned from the data source if and only if all of the predicates match. That is, predicates must be interpreted as ANDed together.
- abstract def pushedPredicates(): Array[Predicate]
Returns the predicates that are pushed to the data source via
#pushPredicates(Predicate[]).Returns the predicates that are pushed to the data source via
#pushPredicates(Predicate[]).There are 3 kinds of predicates:
- pushable predicates which don't need to be evaluated again after scanning.
- pushable predicates which still need to be evaluated after scanning, e.g. parquet row group predicate.
- non-pushable predicates.
Both case 1 and 2 should be considered as pushed predicates and should be returned by this method.
It's possible that there is no predicates in the query and
#pushPredicates(Predicate[])is never called, empty array should be returned for this case.
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)