Package org.apache.spark.sql.util

Class ArrowUtils

Object

org.apache.spark.sql.util.ArrowUtils

public class ArrowUtils
extends Object

Constructor Summary

Constructors

Constructor	Description
ArrowUtils()

Method Summary

Modifier and Type	Method	Description
static DataType	fromArrowField(org.apache.arrow.vector.types.pojo.Field field)
static StructType	fromArrowSchema(org.apache.arrow.vector.types.pojo.Schema schema)
static DataType	fromArrowType(org.apache.arrow.vector.types.pojo.ArrowType dt)
static boolean	isVariantField(org.apache.arrow.vector.types.pojo.Field field)
static org.apache.arrow.memory.RootAllocator	rootAllocator()
static org.apache.arrow.vector.types.pojo.Field	toArrowField(String name, DataType dt, boolean nullable, String timeZoneId, boolean largeVarTypes)	Maps field from Spark to Arrow.
static org.apache.arrow.vector.types.pojo.Schema	toArrowSchema(StructType schema, String timeZoneId, boolean errorOnDuplicatedFieldNames, boolean largeVarTypes)	Maps schema from Spark to Arrow.
static org.apache.arrow.vector.types.pojo.ArrowType	toArrowType(DataType dt, String timeZoneId, boolean largeVarTypes)	Maps data type from Spark to Arrow.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

ArrowUtils

public ArrowUtils()

Method Details

rootAllocator

public static org.apache.arrow.memory.RootAllocator rootAllocator()

toArrowType

public static org.apache.arrow.vector.types.pojo.ArrowType toArrowType(DataType dt,
 String timeZoneId,
 boolean largeVarTypes)

Maps data type from Spark to Arrow. NOTE: timeZoneId required for TimestampTypes

fromArrowType

public static DataType fromArrowType(org.apache.arrow.vector.types.pojo.ArrowType dt)

toArrowField

public static org.apache.arrow.vector.types.pojo.Field toArrowField(String name,
 DataType dt,
 boolean nullable,
 String timeZoneId,
 boolean largeVarTypes)

Maps field from Spark to Arrow. NOTE: timeZoneId required for TimestampType

isVariantField

public static boolean isVariantField(org.apache.arrow.vector.types.pojo.Field field)

fromArrowField

public static DataType fromArrowField(org.apache.arrow.vector.types.pojo.Field field)

toArrowSchema

public static org.apache.arrow.vector.types.pojo.Schema toArrowSchema(StructType schema,
 String timeZoneId,
 boolean errorOnDuplicatedFieldNames,
 boolean largeVarTypes)

Maps schema from Spark to Arrow. NOTE: timeZoneId required for TimestampType in StructType

Parameters:

schema - (undocumented)

timeZoneId - (undocumented)

errorOnDuplicatedFieldNames - (undocumented)

largeVarTypes - (undocumented)

Returns:

(undocumented)

fromArrowSchema

public static StructType fromArrowSchema(org.apache.arrow.vector.types.pojo.Schema schema)