Package org.apache.spark.sql.protobuf

Class functions

Object

org.apache.spark.sql.protobuf.functions

public class functions
extends Object

Constructor Summary

Constructors

Constructor	Description
functions()

Method Summary

Modifier and Type	Method	Description
static Column	from_protobuf(Column data, String messageClassName)	Converts a binary column of Protobuf format into its corresponding catalyst value.
static Column	from_protobuf(Column data, String messageName, byte[] binaryFileDescriptorSet)	Converts a binary column of Protobuf format into its corresponding catalyst value.The Protobuf definition is provided through Protobuf FileDescriptorSet.
static Column	from_protobuf(Column data, String messageName, byte[] binaryFileDescriptorSet, Map<String,String> options)	Converts a binary column of Protobuf format into its corresponding catalyst value.The Protobuf definition is provided through Protobuf FileDescriptorSet.
static Column	from_protobuf(Column data, String messageName, String descFilePath)	Converts a binary column of Protobuf format into its corresponding catalyst value.
static Column	from_protobuf(Column data, String messageName, String descFilePath, Map<String,String> options)	Converts a binary column of Protobuf format into its corresponding catalyst value.
static Column	from_protobuf(Column data, String messageClassName, Map<String,String> options)	Converts a binary column of Protobuf format into its corresponding catalyst value.
static Column	to_protobuf(Column data, String messageClassName)	Converts a column into binary of protobuf format.
static Column	to_protobuf(Column data, String messageName, byte[] binaryFileDescriptorSet)	Converts a column into binary of protobuf format.The Protobuf definition is provided through Protobuf FileDescriptorSet.
static Column	to_protobuf(Column data, String messageName, byte[] binaryFileDescriptorSet, Map<String,String> options)	Converts a column into binary of protobuf format.The Protobuf definition is provided through Protobuf FileDescriptorSet.
static Column	to_protobuf(Column data, String messageName, String descFilePath)	Converts a column into binary of protobuf format.
static Column	to_protobuf(Column data, String messageName, String descFilePath, Map<String,String> options)	Converts a column into binary of protobuf format.
static Column	to_protobuf(Column data, String messageClassName, Map<String,String> options)	Converts a column into binary of protobuf format.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

functions

public functions()

Method Details

from_protobuf

public static Column from_protobuf(Column data,
 String messageName,
 String descFilePath,
 Map<String,String> options)

Converts a binary column of Protobuf format into its corresponding catalyst value. The Protobuf definition is provided through Protobuf descriptor file.

Parameters:

data - the binary column.

messageName - the protobuf message name to look for in descriptor file.

descFilePath - The Protobuf descriptor file. This file is usually created using protoc with --descriptor_set_out and --include_imports options.

options -

Returns:

(undocumented)

Since:

3.4.0

from_protobuf

public static Column from_protobuf(Column data,
 String messageName,
 byte[] binaryFileDescriptorSet,
 Map<String,String> options)

Converts a binary column of Protobuf format into its corresponding catalyst value.The Protobuf definition is provided through Protobuf FileDescriptorSet.

Parameters:

data - the binary column.

messageName - the protobuf MessageName to look for in the descriptor set.

binaryFileDescriptorSet - Serialized Protobuf descriptor (FileDescriptorSet). Typically contents of file created using protoc with --descriptor_set_out and --include_imports options.

options -

Returns:

(undocumented)

Since:

3.5.0

from_protobuf

public static Column from_protobuf(Column data,
 String messageName,
 String descFilePath)

Converts a binary column of Protobuf format into its corresponding catalyst value. The Protobuf definition is provided through Protobuf descriptor file.

Parameters:

data - the binary column.

messageName - the protobuf MessageName to look for in descriptor file.

descFilePath - The Protobuf descriptor file. This file is usually created using protoc with --descriptor_set_out and --include_imports options.

Returns:

(undocumented)

Since:

3.4.0

from_protobuf

public static Column from_protobuf(Column data,
 String messageName,
 byte[] binaryFileDescriptorSet)

Converts a binary column of Protobuf format into its corresponding catalyst value.The Protobuf definition is provided through Protobuf FileDescriptorSet.

Parameters:

data - the binary column.

messageName - the protobuf MessageName to look for in the descriptor set.

binaryFileDescriptorSet - Serialized Protobuf descriptor (FileDescriptorSet). Typically contents of file created using protoc with --descriptor_set_out and --include_imports options.

Returns:

(undocumented)

Since:

3.5.0

from_protobuf

public static Column from_protobuf(Column data,
 String messageClassName)

Converts a binary column of Protobuf format into its corresponding catalyst value. messageClassName points to Protobuf Java class. The jar containing Java class should be shaded. Specifically, com.google.protobuf.* should be shaded to org.sparkproject.spark_protobuf.protobuf.*. https://github.com/rangadi/shaded-protobuf-classes is useful to create shaded jar from Protobuf files.

Parameters:

data - the binary column.

messageClassName - The full name for Protobuf Java class. E.g. com.example.protos.ExampleEvent. The jar with these classes needs to be shaded as described above.

Returns:

(undocumented)

Since:

3.4.0

from_protobuf

public static Column from_protobuf(Column data,
 String messageClassName,
 Map<String,String> options)

Converts a binary column of Protobuf format into its corresponding catalyst value. messageClassName points to Protobuf Java class. The jar containing Java class should be shaded. Specifically, com.google.protobuf.* should be shaded to org.sparkproject.spark_protobuf.protobuf.*. https://github.com/rangadi/shaded-protobuf-classes is useful to create shaded jar from Protobuf files.

Parameters:

data - the binary column.

messageClassName - The full name for Protobuf Java class. E.g. com.example.protos.ExampleEvent. The jar with these classes needs to be shaded as described above.

options -

Returns:

(undocumented)

Since:

3.4.0

to_protobuf

public static Column to_protobuf(Column data,
 String messageName,
 String descFilePath)

Converts a column into binary of protobuf format. The Protobuf definition is provided through Protobuf descriptor file.

Parameters:

data - the data column.

messageName - the protobuf MessageName to look for in descriptor file.

descFilePath - The Protobuf descriptor file. This file is usually created using protoc with --descriptor_set_out and --include_imports options.

Returns:

(undocumented)

Since:

3.4.0

to_protobuf

public static Column to_protobuf(Column data,
 String messageName,
 byte[] binaryFileDescriptorSet)

Converts a column into binary of protobuf format.The Protobuf definition is provided through Protobuf FileDescriptorSet.

Parameters:

data - the binary column.

messageName - the protobuf MessageName to look for in the descriptor set.

binaryFileDescriptorSet - Serialized Protobuf descriptor (FileDescriptorSet). Typically contents of file created using protoc with --descriptor_set_out and --include_imports options.

Returns:

(undocumented)

Since:

3.5.0

to_protobuf

public static Column to_protobuf(Column data,
 String messageName,
 String descFilePath,
 Map<String,String> options)

Converts a column into binary of protobuf format. The Protobuf definition is provided through Protobuf descriptor file.

Parameters:

data - the data column.

messageName - the protobuf MessageName to look for in descriptor file.

descFilePath - the protobuf descriptor file.

options -

Returns:

(undocumented)

Since:

3.4.0

to_protobuf

public static Column to_protobuf(Column data,
 String messageName,
 byte[] binaryFileDescriptorSet,
 Map<String,String> options)

Converts a column into binary of protobuf format.The Protobuf definition is provided through Protobuf FileDescriptorSet.

Parameters:

data - the binary column.

messageName - the protobuf MessageName to look for in the descriptor set.

binaryFileDescriptorSet - Serialized Protobuf descriptor (FileDescriptorSet). Typically contents of file created using protoc with --descriptor_set_out and --include_imports options.

options -

Returns:

(undocumented)

Since:

3.5.0

to_protobuf

public static Column to_protobuf(Column data,
 String messageClassName)

Converts a column into binary of protobuf format. messageClassName points to Protobuf Java class. The jar containing Java class should be shaded. Specifically, com.google.protobuf.* should be shaded to org.sparkproject.spark_protobuf.protobuf.*. https://github.com/rangadi/shaded-protobuf-classes is useful to create shaded jar from Protobuf files.

Parameters:

data - the data column.

messageClassName - The full name for Protobuf Java class. E.g. com.example.protos.ExampleEvent. The jar with these classes needs to be shaded as described above.

Returns:

(undocumented)

Since:

3.4.0

to_protobuf

public static Column to_protobuf(Column data,
 String messageClassName,
 Map<String,String> options)

Converts a column into binary of protobuf format. messageClassName points to Protobuf Java class. The jar containing Java class should be shaded. Specifically, com.google.protobuf.* should be shaded to org.sparkproject.spark_protobuf.protobuf.*. https://github.com/rangadi/shaded-protobuf-classes is useful to create shaded jar from Protobuf files.

Parameters:

data - the data column.

messageClassName - The full name for Protobuf Java class. E.g. com.example.protos.ExampleEvent. The jar with these classes needs to be shaded as described above.

options -

Returns:

(undocumented)

Since:

3.4.0