Package org.apache.spark.sql.connector.read

Interface VariantExtraction

All Superinterfaces:
Serializable
@Experimental public interface VariantExtraction extends Serializable
Variant extraction information that describes a single field extraction from a variant column.

This interface captures the information needed by data sources to optimize reading variant columns. Each instance represents one field extraction operation (e.g., from variant_get or try_variant_get).

For example, if a query contains `variant_get(v, '$.a', 'int')`, this would be represented as a VariantExtraction with columnName=["v"], path="$.a", and expectedDataType=IntegerType.

Since:
4.1.0

  • Method Summary

    Modifier and Type
    Method
    Description
    String[]
    columnName()
    Returns the path to the variant column.
    DataType
    expectedDataType()
    Returns the expected data type for the extracted value.
    Metadata
    metadata()
    Returns the metadata associated with this variant extraction.

  • Method Details

    • columnName

      String[] columnName()
      Returns the path to the variant column. For top-level variant columns, this is a single element array containing the column name. For nested variant columns within structs, this is an array representing the path (e.g., ["structCol", "innerStruct", "variantCol"]).

    • expectedDataType

      DataType expectedDataType()
      Returns the expected data type for the extracted value. This is the target type specified in variant_get (e.g., IntegerType, StringType).

    • metadata

      Metadata metadata()
      Returns the metadata associated with this variant extraction. This may include additional information needed by the data source: - "path": the extraction path from variant_get or try_variant_get. This follows JSON path syntax (e.g., "$.a", "$.b.c", "$[0]"). - "failOnError": whether the extraction to expected data type should throw an exception or return null if the cast fails. - "timeZoneId": a string identifier of a time zone. It is required by timestamp-related casts.