org.apache.spark.sql.types

Class DataType

Object

org.apache.spark.sql.types.DataType

Direct Known Subclasses:

ArrayType, BinaryType, BooleanType, CalendarIntervalType, CharType, DateType, DayTimeIntervalType, MapType, NullType, NumericType, ObjectType, StringType, StructType, TimestampNTZType, TimestampType, UserDefinedType, VarcharType, YearMonthIntervalType

public abstract class DataType
extends Object

The base type of all Spark SQL data types.

Since:

1.3.0

Constructor Summary

Constructors

Constructor and Description
DataType()

Method Summary

Modifier and Type	Method and Description
static boolean	canWrite(DataType write, DataType read, boolean byName, scala.Function2<String,String,Object> resolver, String context, scala.Enumeration.Value storeAssignmentPolicy, scala.Function1<String,scala.runtime.BoxedUnit> addError) Returns true if the write data type can be read using the read data type.
String	catalogString() String representation for the type saved in external catalogs.
abstract int	defaultSize() The default size of a value of this data type, used internally for size estimation.
static boolean	equalsStructurally(DataType from, DataType to, boolean ignoreNullability) Returns true if the two data types share the same "shape", i.e.
static boolean	equalsStructurallyByName(DataType from, DataType to, scala.Function2<String,String,Object> resolver) Returns true if the two data types have the same field names in order recursively.
static DataType	fromDDL(String ddl)
static DataType	fromJson(String json)
String	json() The compact JSON representation of this data type.
static DataType	parseTypeWithFallback(String schema, scala.Function1<String,DataType> parser, scala.Function1<String,DataType> fallbackParser) Parses data type from a string with schema.
String	prettyJson() The pretty (i.e.
String	simpleString() Readable string representation for the type.
String	sql()
String	typeName() Name of the type used in JSON serialization.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataType

public DataType()

Method Detail

fromDDL

public static DataType fromDDL(String ddl)

parseTypeWithFallback

public static DataType parseTypeWithFallback(String schema,
                                             scala.Function1<String,DataType> parser,
                                             scala.Function1<String,DataType> fallbackParser)

Parses data type from a string with schema. It calls parser for schema. If it fails, calls fallbackParser. If the fallback function fails too, combines error message from parser and fallbackParser.

Parameters:

schema - The schema string to parse by parser or fallbackParser.

parser - The function that should be invoke firstly.

fallbackParser - The function that is called when parser fails.

Returns:

The data type parsed from the schema schema.

fromJson

public static DataType fromJson(String json)

equalsStructurally

public static boolean equalsStructurally(DataType from,
                                         DataType to,
                                         boolean ignoreNullability)

Returns true if the two data types share the same "shape", i.e. the types are the same, but the field names don't need to be the same.

Parameters:

ignoreNullability - whether to ignore nullability when comparing the types

from - (undocumented)

to - (undocumented)

Returns:

(undocumented)

equalsStructurallyByName

public static boolean equalsStructurallyByName(DataType from,
                                               DataType to,
                                               scala.Function2<String,String,Object> resolver)

Returns true if the two data types have the same field names in order recursively.

Parameters:

from - (undocumented)

to - (undocumented)

resolver - (undocumented)

Returns:

(undocumented)

canWrite

public static boolean canWrite(DataType write,
                               DataType read,
                               boolean byName,
                               scala.Function2<String,String,Object> resolver,
                               String context,
                               scala.Enumeration.Value storeAssignmentPolicy,
                               scala.Function1<String,scala.runtime.BoxedUnit> addError)

Returns true if the write data type can be read using the read data type.

The write type is compatible with the read type if: - Both types are arrays, the array element types are compatible, and element nullability is compatible (read allows nulls or write does not contain nulls). - Both types are maps and the map key and value types are compatible, and value nullability is compatible (read allows nulls or write does not contain nulls). - Both types are structs and have the same number of fields. The type and nullability of each field from read/write is compatible. If byName is true, the name of each field from read/write needs to be the same. - Both types are atomic and the write type can be safely cast to the read type.

Extra fields in write-side structs are not allowed to avoid accidentally writing data that the read schema will not read, and to ensure map key equality is not changed when data is read.

Parameters:

write - a write-side data type to validate against the read type

read - a read-side data type

byName - (undocumented)

resolver - (undocumented)

context - (undocumented)

storeAssignmentPolicy - (undocumented)

addError - (undocumented)

Returns:

true if data written with the write type can be read using the read type

defaultSize

public abstract int defaultSize()

The default size of a value of this data type, used internally for size estimation.

Returns:

(undocumented)

typeName

public String typeName()

Name of the type used in JSON serialization.

json

public String json()

The compact JSON representation of this data type.

prettyJson

public String prettyJson()

The pretty (i.e. indented) JSON representation of this data type.

simpleString

public String simpleString()

Readable string representation for the type.

catalogString

public String catalogString()

String representation for the type saved in external catalogs.

sql

public String sql()