trait CatalogExtension extends TableCatalog with FunctionCatalog with SupportsNamespaces
An API to extend the Spark built-in session catalog. Implementation can get the built-in session
catalog from #setDelegateCatalog(CatalogPlugin), implement catalog functions with
some custom logic and call the built-in session catalog at the end. For example, they can
implement createTable, do something else before calling createTable of the
built-in session catalog.
- Annotations
- @Evolving()
- Source
- CatalogExtension.java
- Since
3.0.0
- Alphabetic
- By Inheritance
- CatalogExtension
- SupportsNamespaces
- FunctionCatalog
- TableCatalog
- CatalogPlugin
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def alterNamespace(namespace: Array[String], changes: <repeated...>[NamespaceChange]): Unit
Apply a set of metadata changes to a namespace in the catalog.
Apply a set of metadata changes to a namespace in the catalog.
- namespace
a multi-part namespace
- changes
a collection of changes to apply to the namespace
- Definition Classes
- SupportsNamespaces
- Exceptions thrown
NoSuchNamespaceExceptionIf the namespace does not exist (optional)UnsupportedOperationExceptionIf namespace properties are not supported
- abstract def alterTable(ident: Identifier, changes: <repeated...>[TableChange]): Table
Apply a set of
changesto a table in the catalog.Apply a set of
changesto a table in the catalog.Implementations may reject the requested changes. If any change is rejected, none of the changes should be applied to the table.
The requested changes must be applied in the order given.
If the catalog supports views and contains a view for the identifier and not a table, this must throw
NoSuchTableException.- ident
a table identifier
- changes
changes to apply to the table
- returns
updated metadata for the table. This can be null if getting the metadata for the updated table is expensive. Spark always discard the returned table here.
- Definition Classes
- TableCatalog
- Exceptions thrown
IllegalArgumentExceptionIf any change is rejected by the implementation.NoSuchTableExceptionIf the table doesn't exist or is a view
- abstract def createNamespace(namespace: Array[String], metadata: Map[String, String]): Unit
Create a namespace in the catalog.
Create a namespace in the catalog.
- namespace
a multi-part namespace
- metadata
a string map of properties for the given namespace
- Definition Classes
- SupportsNamespaces
- Exceptions thrown
NamespaceAlreadyExistsExceptionIf the namespace already existsUnsupportedOperationExceptionIf create is not a supported operation
- abstract def dropNamespace(namespace: Array[String], cascade: Boolean): Boolean
Drop a namespace from the catalog with cascade mode, recursively dropping all objects within the namespace if cascade is true.
Drop a namespace from the catalog with cascade mode, recursively dropping all objects within the namespace if cascade is true.
If the catalog implementation does not support this operation, it may throw
UnsupportedOperationException.- namespace
a multi-part namespace
- cascade
When true, deletes all objects under the namespace
- returns
true if the namespace was dropped
- Definition Classes
- SupportsNamespaces
- Exceptions thrown
NoSuchNamespaceExceptionIf the namespace does not exist (optional)NonEmptyNamespaceExceptionIf the namespace is non-empty and cascade is falseUnsupportedOperationExceptionIf drop is not a supported operation
- abstract def dropTable(ident: Identifier): Boolean
Drop a table in the catalog.
Drop a table in the catalog.
If the catalog supports views and contains a view for the identifier and not a table, this must not drop the view and must return false.
- ident
a table identifier
- returns
true if a table was deleted, false if no table exists for the identifier
- Definition Classes
- TableCatalog
- abstract def initialize(name: String, options: CaseInsensitiveStringMap): Unit
Called to initialize configuration.
Called to initialize configuration.
This method is called once, just after the provider is instantiated.
- name
the name used to identify and load this catalog
- options
a case-insensitive string map of configuration
- Definition Classes
- CatalogPlugin
- abstract def listFunctions(namespace: Array[String]): Array[Identifier]
List the functions in a namespace from the catalog.
List the functions in a namespace from the catalog.
If there are no functions in the namespace, implementations should return an empty array.
- namespace
a multi-part namespace
- returns
an array of Identifiers for functions
- Definition Classes
- FunctionCatalog
- Exceptions thrown
NoSuchNamespaceExceptionIf the namespace does not exist (optional).
- abstract def listNamespaces(namespace: Array[String]): Array[Array[String]]
List namespaces in a namespace.
List namespaces in a namespace.
If an object such as a table, view, or function exists, its parent namespaces must also exist and must be returned by this discovery method. For example, if table a.b.t exists, this method invoked as listNamespaces(["a"]) must return ["a", "b"] in the result array.
- namespace
a multi-part namespace
- returns
an array of multi-part namespace names
- Definition Classes
- SupportsNamespaces
- Exceptions thrown
NoSuchNamespaceExceptionIf the namespace does not exist (optional)
- abstract def listNamespaces(): Array[Array[String]]
List top-level namespaces from the catalog.
List top-level namespaces from the catalog.
If an object such as a table, view, or function exists, its parent namespaces must also exist and must be returned by this discovery method. For example, if table a.b.t exists, this method must return ["a"] in the result array.
- returns
an array of multi-part namespace names
- Definition Classes
- SupportsNamespaces
- abstract def listTables(namespace: Array[String]): Array[Identifier]
List the tables in a namespace from the catalog.
List the tables in a namespace from the catalog.
If the catalog supports views, this must return identifiers for only tables and not views.
- namespace
a multi-part namespace
- returns
an array of Identifiers for tables
- Definition Classes
- TableCatalog
- Exceptions thrown
NoSuchNamespaceExceptionIf the namespace does not exist (optional).
- abstract def loadFunction(ident: Identifier): UnboundFunction
Load a function by
identifierfrom the catalog.Load a function by
identifierfrom the catalog.- ident
a function identifier
- returns
an unbound function instance
- Definition Classes
- FunctionCatalog
- Exceptions thrown
NoSuchFunctionExceptionIf the function doesn't exist
- abstract def loadNamespaceMetadata(namespace: Array[String]): Map[String, String]
Load metadata properties for a namespace.
Load metadata properties for a namespace.
- namespace
a multi-part namespace
- returns
a string map of properties for the given namespace
- Definition Classes
- SupportsNamespaces
- Exceptions thrown
NoSuchNamespaceExceptionIf the namespace does not exist (optional)UnsupportedOperationExceptionIf namespace properties are not supported
- abstract def loadTable(ident: Identifier): Table
Load table metadata by
identifierfrom the catalog.Load table metadata by
identifierfrom the catalog.If the catalog supports views and contains a view for the identifier and not a table, this must throw
NoSuchTableException.- ident
a table identifier
- returns
the table's metadata
- Definition Classes
- TableCatalog
- Exceptions thrown
NoSuchTableExceptionIf the table doesn't exist or is a view
- abstract def name(): String
Called to get this catalog's name.
Called to get this catalog's name.
This method is only called after
CaseInsensitiveStringMap)is called to pass the catalog's name.- Definition Classes
- CatalogPlugin
- abstract def renameTable(oldIdent: Identifier, newIdent: Identifier): Unit
Renames a table in the catalog.
Renames a table in the catalog.
If the catalog supports views and contains a view for the old identifier and not a table, this throws
NoSuchTableException. Additionally, if the new identifier is a table or a view, this throwsTableAlreadyExistsException.If the catalog does not support table renames between namespaces, it throws
UnsupportedOperationException.- oldIdent
the table identifier of the existing table to rename
- newIdent
the new table identifier of the table
- Definition Classes
- TableCatalog
- Exceptions thrown
NoSuchTableExceptionIf the table to rename doesn't exist or is a viewTableAlreadyExistsExceptionIf the new table name already exists or is a viewUnsupportedOperationExceptionIf the namespaces of old and new identifiers do not match (optional)
- abstract def setDelegateCatalog(delegate: CatalogPlugin): Unit
This will be called only once by Spark to pass in the Spark built-in session catalog, after
CaseInsensitiveStringMap)is called.
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def capabilities(): Set[TableCatalogCapability]
- returns
the set of capabilities for this TableCatalog
- Definition Classes
- TableCatalog
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def createTable(ident: Identifier, columns: Array[Column], partitions: Array[Transform], properties: Map[String, String]): Table
Create a table in the catalog.
Create a table in the catalog.
- ident
a table identifier
- columns
the columns of the new table.
- partitions
transforms to use for partitioning data in the table
- properties
a string map of table properties
- returns
metadata for the new table. This can be null if getting the metadata for the new table is expensive. Spark will call
#loadTable(Identifier)if needed (e.g. CTAS).
- Definition Classes
- TableCatalog
- Exceptions thrown
NoSuchNamespaceExceptionIf the identifier namespace does not exist (optional)TableAlreadyExistsExceptionIf a table or view already exists for the identifierUnsupportedOperationExceptionIf a requested partition transform is not supported
- def defaultNamespace(): Array[String]
Return a default namespace for the catalog.
Return a default namespace for the catalog.
When this catalog is set as the current catalog, the namespace returned by this method will be set as the current namespace.
The namespace returned by this method is not required to exist.
- returns
a multi-part namespace
- Definition Classes
- CatalogPlugin
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def functionExists(ident: Identifier): Boolean
Returns true if the function exists, false otherwise.
Returns true if the function exists, false otherwise.
- Definition Classes
- FunctionCatalog
- Since
3.3.0
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def invalidateTable(ident: Identifier): Unit
Invalidate cached table metadata for an
identifier.Invalidate cached table metadata for an
identifier.If the table is already loaded or cached, drop cached data. If the table does not exist or is not cached, do nothing. Calling this method should not query remote services.
- ident
a table identifier
- Definition Classes
- TableCatalog
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def loadTable(ident: Identifier, timestamp: Long): Table
Load table metadata at a specific time by
identifierfrom the catalog.Load table metadata at a specific time by
identifierfrom the catalog.If the catalog supports views and contains a view for the identifier and not a table, this must throw
NoSuchTableException.- ident
a table identifier
- timestamp
timestamp of the table, which is microseconds since 1970-01-01 00:00:00 UTC
- returns
the table's metadata
- Definition Classes
- TableCatalog
- Exceptions thrown
NoSuchTableExceptionIf the table doesn't exist or is a view
- def loadTable(ident: Identifier, version: String): Table
Load table metadata of a specific version by
identifierfrom the catalog.Load table metadata of a specific version by
identifierfrom the catalog.If the catalog supports views and contains a view for the identifier and not a table, this must throw
NoSuchTableException.- ident
a table identifier
- version
version of the table
- returns
the table's metadata
- Definition Classes
- TableCatalog
- Exceptions thrown
NoSuchTableExceptionIf the table doesn't exist or is a view
- def loadTable(ident: Identifier, writePrivileges: Set[TableWritePrivilege]): Table
Load table metadata by
identifierfrom the catalog.Load table metadata by
identifierfrom the catalog. Spark will write data into this table later.If the catalog supports views and contains a view for the identifier and not a table, this must throw
NoSuchTableException.- ident
a table identifier
- returns
the table's metadata
- Definition Classes
- TableCatalog
- Since
3.5.3
- Exceptions thrown
NoSuchTableExceptionIf the table doesn't exist or is a view
- def namespaceExists(namespace: Array[String]): Boolean
Test whether a namespace exists.
Test whether a namespace exists.
If an object such as a table, view, or function exists, its parent namespaces must also exist. For example, if table a.b.t exists, this method invoked as namespaceExists(["a"]) or namespaceExists(["a", "b"]) must return true.
- namespace
a multi-part namespace
- returns
true if the namespace exists, false otherwise
- Definition Classes
- SupportsNamespaces
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def purgeTable(ident: Identifier): Boolean
Drop a table in the catalog and completely remove its data by skipping a trash even if it is supported.
Drop a table in the catalog and completely remove its data by skipping a trash even if it is supported.
If the catalog supports views and contains a view for the identifier and not a table, this must not drop the view and must return false.
If the catalog supports to purge a table, this method should be overridden. The default implementation throws
UnsupportedOperationException.- ident
a table identifier
- returns
true if a table was deleted, false if no table exists for the identifier
- Definition Classes
- TableCatalog
- Since
3.1.0
- Exceptions thrown
UnsupportedOperationExceptionIf table purging is not supported
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def tableExists(ident: Identifier): Boolean
Test whether a table exists using an
identifierfrom the catalog.Test whether a table exists using an
identifierfrom the catalog.If the catalog supports views and contains a view for the identifier and not a table, this must return false.
- ident
a table identifier
- returns
true if the table exists, false otherwise
- Definition Classes
- TableCatalog
- def toString(): String
- Definition Classes
- AnyRef → Any
- def useNullableQuerySchema(): Boolean
If true, mark all the fields of the query schema as nullable when executing CREATE/REPLACE TABLE ...
If true, mark all the fields of the query schema as nullable when executing CREATE/REPLACE TABLE ... AS SELECT ... and creating the table.
- Definition Classes
- TableCatalog
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def createTable(ident: Identifier, schema: StructType, partitions: Array[Transform], properties: Map[String, String]): Table
Create a table in the catalog.
Create a table in the catalog.
- Definition Classes
- TableCatalog
- Annotations
- @Deprecated
- Deprecated
(Since version 3.4.0)
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)