Class

org.apache.spark.sql

SparkSessionExtensions

Related Doc: package sql

Permalink

class SparkSessionExtensions extends AnyRef

:: Experimental :: Holder for injection points to the SparkSession. We make NO guarantee about the stability regarding binary compatibility and source compatibility of methods here.

This current provides the following extension points: - Analyzer Rules. - Check Analysis Rules - Optimizer Rules. - Planning Strategies. - Customized Parser. - (External) Catalog listeners.

The extensions can be used by calling withExtension on the SparkSession.Builder, for example:

SparkSession.builder()
  .master("...")
  .conf("...", true)
  .withExtensions { extensions =>
    extensions.injectResolutionRule { session =>
      ...
    }
    extensions.injectParser { (session, parser) =>
      ...
    }
  }
  .getOrCreate()

Note that none of the injected builders should assume that the SparkSession is fully initialized and should not touch the session's internals (e.g. the SessionState).

Annotations
@DeveloperApi() @Experimental() @Unstable()
Source
SparkSessionExtensions.scala
Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SparkSessionExtensions
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SparkSessionExtensions()

    Permalink

Type Members

  1. type CheckRuleBuilder = (SparkSession) ⇒ (LogicalPlan) ⇒ Unit

    Permalink
  2. type ParserBuilder = (SparkSession, ParserInterface) ⇒ ParserInterface

    Permalink
  3. type RuleBuilder = (SparkSession) ⇒ Rule[LogicalPlan]

    Permalink
  4. type StrategyBuilder = (SparkSession) ⇒ Strategy

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  6. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  7. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  9. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  10. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  11. def injectCheckRule(builder: CheckRuleBuilder): Unit

    Permalink

    Inject an check analysis Rule builder into the SparkSession.

    Inject an check analysis Rule builder into the SparkSession. The injected rules will be executed after the analysis phase. A check analysis rule is used to detect problems with a LogicalPlan and should throw an exception when a problem is found.

  12. def injectOptimizerRule(builder: RuleBuilder): Unit

    Permalink

    Inject an optimizer Rule builder into the SparkSession.

    Inject an optimizer Rule builder into the SparkSession. The injected rules will be executed during the operator optimization batch. An optimizer rule is used to improve the quality of an analyzed logical plan; these rules should never modify the result of the LogicalPlan.

  13. def injectParser(builder: ParserBuilder): Unit

    Permalink

    Inject a custom parser into the SparkSession.

    Inject a custom parser into the SparkSession. Note that the builder is passed a session and an initial parser. The latter allows for a user to create a partial parser and to delegate to the underlying parser for completeness. If a user injects more parsers, then the parsers are stacked on top of each other.

  14. def injectPlannerStrategy(builder: StrategyBuilder): Unit

    Permalink

    Inject a planner Strategy builder into the SparkSession.

    Inject a planner Strategy builder into the SparkSession. The injected strategy will be used to convert a LogicalPlan into a executable org.apache.spark.sql.execution.SparkPlan.

  15. def injectPostHocResolutionRule(builder: RuleBuilder): Unit

    Permalink

    Inject an analyzer Rule builder into the SparkSession.

    Inject an analyzer Rule builder into the SparkSession. These analyzer rules will be executed after resolution.

  16. def injectResolutionRule(builder: RuleBuilder): Unit

    Permalink

    Inject an analyzer resolution Rule builder into the SparkSession.

    Inject an analyzer resolution Rule builder into the SparkSession. These analyzer rules will be executed as part of the resolution phase of analysis.

  17. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  18. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  19. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  20. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  21. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  22. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  23. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  24. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  25. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped