Package org.apache.spark.sql.jdbc

Class DatabricksDialect

Object
org.apache.spark.sql.jdbc.JdbcDialect
org.apache.spark.sql.jdbc.DatabricksDialect
All Implemented Interfaces:
Serializable, org.apache.spark.internal.Logging, NoLegacyJDBCError, scala.Equals, scala.Product
public class DatabricksDialect extends JdbcDialect implements NoLegacyJDBCError, scala.Product, Serializable
See Also:

  • Constructor Details

    • DatabricksDialect

      public DatabricksDialect()

  • Method Details

    • apply

      public abstract static R apply()

    • toString

      public static String toString()

    • canHandle

      public boolean canHandle(String url)
      Description copied from class: JdbcDialect
      Check if this dialect instance can handle a certain jdbc url.
      Specified by:
      canHandle in class JdbcDialect
      Parameters:
      url - the jdbc url.
      Returns:
      True if the dialect can be applied on the given jdbc url.

    • isObjectNotFoundException

      public boolean isObjectNotFoundException(SQLException e)
      Overrides:
      isObjectNotFoundException in class JdbcDialect

    • getCatalystType

      public scala.Option<DataType> getCatalystType(int sqlType, String typeName, int size, MetadataBuilder md)
      Description copied from class: JdbcDialect
      Get the custom datatype mapping for the given jdbc meta information.

      Guidelines for mapping database defined timestamps to Spark SQL timestamps:

      • TIMESTAMP WITHOUT TIME ZONE if preferTimestampNTZ -> TimestampNTZType
      • TIMESTAMP WITHOUT TIME ZONE if !preferTimestampNTZ -> TimestampType(LTZ)
      • TIMESTAMP WITH TIME ZONE -> TimestampType(LTZ)
      • TIMESTAMP WITH LOCAL TIME ZONE -> TimestampType(LTZ)
      • If the TIMESTAMP cannot be distinguished by sqlType and typeName, preferTimestampNTZ is respected for now, but we may need to add another option in the future if necessary.

      Overrides:
      getCatalystType in class JdbcDialect
      Parameters:
      sqlType - Refers to Types constants, or other constants defined by the target database, e.g. -101 is Oracle's TIMESTAMP WITH TIME ZONE type. This value is returned by ResultSetMetaData.getColumnType(int).
      typeName - The column type name used by the database (e.g. "BIGINT UNSIGNED"). This is sometimes used to determine the target data type when sqlType is not sufficient if multiple database types are conflated into a single id. This value is returned by ResultSetMetaData.getColumnTypeName(int).
      size - The size of the type, e.g. the maximum precision for numeric types, length for character string, etc. This value is returned by ResultSetMetaData.getPrecision(int).
      md - Result metadata associated with this type. This contains additional information from ResultSetMetaData or user specified options.
      Returns:
      An option the actual DataType (subclasses of DataType) or None if the default type mapping should be used.

    • getJDBCType

      public scala.Option<JdbcType> getJDBCType(DataType dt)
      Description copied from class: JdbcDialect
      Retrieve the jdbc / sql type for a given datatype.
      Overrides:
      getJDBCType in class JdbcDialect
      Parameters:
      dt - The datatype (e.g. StringType)
      Returns:
      The new JdbcType if there is an override for this DataType

    • isSyntaxErrorBestEffort

      public boolean isSyntaxErrorBestEffort(SQLException exception)
      Description copied from class: JdbcDialect
      Attempts to determine if the given SQLException is a SQL syntax error.

      This check is best-effort: it may not detect all syntax errors across all JDBC dialects. However, if this method returns true, the exception is guaranteed to be a syntax error.

      This is used to decide whether to wrap the exception in a more appropriate Spark exception.

      Overrides:
      isSyntaxErrorBestEffort in class JdbcDialect
      Parameters:
      exception - (undocumented)
      Returns:
      true if the exception is confidently identified as a syntax error; false otherwise.

    • quoteIdentifier

      public String quoteIdentifier(String colName)
      Description copied from class: JdbcDialect
      Quotes the identifier. This is used to put quotes around the identifier in case the column name is a reserved keyword, or in case it contains characters that require quotes (e.g. space).
      Overrides:
      quoteIdentifier in class JdbcDialect
      Parameters:
      colName - (undocumented)
      Returns:
      (undocumented)

    • supportsLimit

      public boolean supportsLimit()
      Description copied from class: JdbcDialect
      Returns ture if dialect supports LIMIT clause.

      Note: Some build-in dialect supports LIMIT clause with some trick, please see: OracleDialect.OracleSQLQueryBuilder and MsSqlServerDialect.MsSqlServerSQLQueryBuilder.

      Overrides:
      supportsLimit in class JdbcDialect
      Returns:
      (undocumented)

    • supportsOffset

      public boolean supportsOffset()
      Description copied from class: JdbcDialect
      Returns ture if dialect supports OFFSET clause.

      Note: Some build-in dialect supports OFFSET clause with some trick, please see: OracleDialect.OracleSQLQueryBuilder and MySQLDialect.MySQLSQLQueryBuilder.

      Overrides:
      supportsOffset in class JdbcDialect
      Returns:
      (undocumented)

    • supportsTableSample

      public boolean supportsTableSample()
      Overrides:
      supportsTableSample in class JdbcDialect

    • getTableSample

      public String getTableSample(org.apache.spark.sql.execution.datasources.v2.TableSampleInfo sample)
      Overrides:
      getTableSample in class JdbcDialect

    • supportsHint

      public boolean supportsHint()
      Overrides:
      supportsHint in class JdbcDialect

    • schemasExists

      public boolean schemasExists(Connection conn, org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions options, String schema)
      Description copied from class: JdbcDialect
      Check schema exists or not.
      Overrides:
      schemasExists in class JdbcDialect
      Parameters:
      conn - (undocumented)
      options - (undocumented)
      schema - (undocumented)
      Returns:
      (undocumented)

    • listSchemas

      public String[][] listSchemas(Connection conn, org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions options)
      Description copied from class: JdbcDialect
      Lists all the schemas in this table.
      Overrides:
      listSchemas in class JdbcDialect
      Parameters:
      conn - (undocumented)
      options - (undocumented)
      Returns:
      (undocumented)