Spark 3.4.0 ScalaDoc - org.apache.spark.sql.SparkSession

final def !=(arg0: Any): Boolean

Definition Classes: AnyRef → Any

final def ##(): Int

Definition Classes: AnyRef → Any

final def ==(arg0: Any): Boolean

Definition Classes: AnyRef → Any

final def asInstanceOf[T0]: T0

Definition Classes: Any

def baseRelationToDataFrame(baseRelation: BaseRelation): DataFrame

Convert a BaseRelation created for external data sources into a DataFrame.

Since: 2.0.0

lazy val catalog: Catalog

Interface through which the user may create, drop, alter or query underlying databases, tables, functions etc.

Annotations: @transient()
Since: 2.0.0

def clone(): AnyRef

Attributes: protected[lang]
Definition Classes: AnyRef
Annotations: @throws( ... ) @native()

def close(): Unit

Synonym for stop().

Definition Classes: SparkSession → Closeable → AutoCloseable
Since: 2.1.0

lazy val conf: RuntimeConfig

Runtime configuration interface for Spark.

This is the interface through which the user can get and set all Spark and Hadoop configurations that are relevant to Spark SQL. When getting the value of a config, this defaults to the value set in the underlying SparkContext, if any.

Annotations: @transient()
Since: 2.0.0

def createDataFrame(data: List[_], beanClass: Class[_]): DataFrame

Applies a schema to a List of Java Beans.

WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

Since: 1.6.0

def createDataFrame(rdd: JavaRDD[_], beanClass: Class[_]): DataFrame

Applies a schema to an RDD of Java Beans.

WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

Since: 2.0.0

def createDataFrame(rdd: RDD[_], beanClass: Class[_]): DataFrame

Applies a schema to an RDD of Java Beans.

WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

Since: 2.0.0

def createDataFrame(rows: List[Row], schema: StructType): DataFrame

:: DeveloperApi :: Creates a DataFrame from a java.util.List containing Rows using the given schema.

:: DeveloperApi :: Creates a DataFrame from a java.util.List containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided List matches the provided schema. Otherwise, there will be runtime exception.

Annotations: @DeveloperApi()
Since: 2.0.0

def createDataFrame(rowRDD: JavaRDD[Row], schema: StructType): DataFrame

:: DeveloperApi :: Creates a DataFrame from a JavaRDD containing Rows using the given schema.

:: DeveloperApi :: Creates a DataFrame from a JavaRDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception.

Annotations: @DeveloperApi()
Since: 2.0.0

def createDataFrame(rowRDD: RDD[Row], schema: StructType): DataFrame

:: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema.

:: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception. Example:

import org.apache.spark.sql._
import org.apache.spark.sql.types._
val sparkSession = new org.apache.spark.sql.SparkSession(sc)

val schema =
  StructType(
    StructField("name", StringType, false) ::
    StructField("age", IntegerType, true) :: Nil)

val people =
  sc.textFile("examples/src/main/resources/people.txt").map(
    _.split(",")).map(p => Row(p(0), p(1).trim.toInt))
val dataFrame = sparkSession.createDataFrame(people, schema)
dataFrame.printSchema
// root
// |-- name: string (nullable = false)
// |-- age: integer (nullable = true)

dataFrame.createOrReplaceTempView("people")
sparkSession.sql("select name from people").collect.foreach(println)

Annotations: @DeveloperApi()
Since: 2.0.0

def createDataFrame[A <: Product](data: Seq[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame

Creates a DataFrame from a local Seq of Product.

Since: 2.0.0

def createDataFrame[A <: Product](rdd: RDD[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame

Creates a DataFrame from an RDD of Product (e.g.

Creates a DataFrame from an RDD of Product (e.g. case classes, tuples).

Since: 2.0.0

def createDataset[T](data: List[T])(implicit arg0: Encoder[T]): Dataset[T]

Creates a Dataset from a java.util.List of a given type.

Creates a Dataset from a java.util.List of a given type. This method requires an encoder (to convert a JVM object of type T to and from the internal Spark SQL representation) that is generally created automatically through implicits from a SparkSession, or can be created explicitly by calling static methods on Encoders.

Java Example

List<String> data = Arrays.asList("hello", "world");
Dataset<String> ds = spark.createDataset(data, Encoders.STRING());

Since: 2.0.0

def createDataset[T](data: RDD[T])(implicit arg0: Encoder[T]): Dataset[T]

Creates a Dataset from an RDD of a given type.

Creates a Dataset from an RDD of a given type. This method requires an encoder (to convert a JVM object of type T to and from the internal Spark SQL representation) that is generally created automatically through implicits from a SparkSession, or can be created explicitly by calling static methods on Encoders.

Since: 2.0.0

def createDataset[T](data: Seq[T])(implicit arg0: Encoder[T]): Dataset[T]

Creates a Dataset from a local Seq of data of a given type.

Creates a Dataset from a local Seq of data of a given type. This method requires an encoder (to convert a JVM object of type T to and from the internal Spark SQL representation) that is generally created automatically through implicits from a SparkSession, or can be created explicitly by calling static methods on Encoders.

Example

import spark.implicits._
case class Person(name: String, age: Long)
val data = Seq(Person("Michael", 29), Person("Andy", 30), Person("Justin", 19))
val ds = spark.createDataset(data)

ds.show()
// +-------+---+
// |   name|age|
// +-------+---+
// |Michael| 29|
// |   Andy| 30|
// | Justin| 19|
// +-------+---+

Since: 2.0.0

lazy val emptyDataFrame: DataFrame

Returns a DataFrame with no rows or columns.

Annotations: @transient()
Since: 2.0.0

def emptyDataset[T](implicit arg0: Encoder[T]): Dataset[T]

Creates a new Dataset of type T containing zero elements.

Since: 2.0.0

final def eq(arg0: AnyRef): Boolean

Definition Classes: AnyRef

def equals(arg0: Any): Boolean

Definition Classes: AnyRef → Any

def executeCommand(runner: String, command: String, options: Map[String, String]): DataFrame

Execute an arbitrary string command inside an external execution engine rather than Spark.

Execute an arbitrary string command inside an external execution engine rather than Spark. This could be useful when user wants to execute some commands out of Spark. For example, executing custom DDL/DML command for JDBC, creating index for ElasticSearch, creating cores for Solr and so on.

The command will be eagerly executed after this method is called and the returned DataFrame will contain the output of the command(if any).

runner: The class name of the runner that implements ExternalCommandRunner.
command: The target command to be executed
options: The options for the runner.

Annotations: @Unstable()
Since: 3.0.0

def experimental: ExperimentalMethods

:: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.

Annotations: @Experimental() @Unstable()
Since: 2.0.0

def finalize(): Unit

Attributes: protected[lang]
Definition Classes: AnyRef
Annotations: @throws( classOf[java.lang.Throwable] )

final def getClass(): Class[_]

Definition Classes: AnyRef → Any
Annotations: @native()

def hashCode(): Int

Definition Classes: AnyRef → Any
Annotations: @native()

def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

Attributes: protected
Definition Classes: Logging

def initializeLogIfNecessary(isInterpreter: Boolean): Unit

Attributes: protected
Definition Classes: Logging

final def isInstanceOf[T0]: Boolean

Definition Classes: Any

def isTraceEnabled(): Boolean

Attributes: protected
Definition Classes: Logging

def listenerManager: ExecutionListenerManager

An interface to register custom org.apache.spark.sql.util.QueryExecutionListeners that listen for execution metrics.

Since: 2.0.0

def log: Logger

Attributes: protected
Definition Classes: Logging

def logDebug(msg: ⇒ String, throwable: Throwable): Unit

Attributes: protected
Definition Classes: Logging

def logDebug(msg: ⇒ String): Unit

Attributes: protected
Definition Classes: Logging

def logError(msg: ⇒ String, throwable: Throwable): Unit

Attributes: protected
Definition Classes: Logging

def logError(msg: ⇒ String): Unit

Attributes: protected
Definition Classes: Logging

def logInfo(msg: ⇒ String, throwable: Throwable): Unit

Attributes: protected
Definition Classes: Logging

def logInfo(msg: ⇒ String): Unit

Attributes: protected
Definition Classes: Logging

def logName: String

Attributes: protected
Definition Classes: Logging

def logTrace(msg: ⇒ String, throwable: Throwable): Unit

Attributes: protected
Definition Classes: Logging

def logTrace(msg: ⇒ String): Unit

Attributes: protected
Definition Classes: Logging

def logWarning(msg: ⇒ String, throwable: Throwable): Unit

Attributes: protected
Definition Classes: Logging

def logWarning(msg: ⇒ String): Unit

Attributes: protected
Definition Classes: Logging

final def ne(arg0: AnyRef): Boolean

Definition Classes: AnyRef

def newSession(): SparkSession

Start a new session with isolated SQL configurations, temporary tables, registered functions are isolated, but sharing the underlying SparkContext and cached data.

Since: 2.0.0
Note: Other than the SparkContext, all shared state is initialized lazily. This method will force the initialization of the shared state to ensure that parent and child sessions are set up with the same shared state. If the underlying catalog implementation is Hive, this will initialize the metastore, which may take some time.

final def notify(): Unit

Definition Classes: AnyRef
Annotations: @native()

final def notifyAll(): Unit

Definition Classes: AnyRef
Annotations: @native()

def parseDataType(dataTypeString: String): DataType

Parses the data type in our internal string representation.

Parses the data type in our internal string representation. The data type string should have the same format as the one generated by toString in scala. It is only used by PySpark.

Attributes: protected[sql]

def range(start: Long, end: Long, step: Long, numPartitions: Int): Dataset[Long]

Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value, with partition number specified.

Since: 2.0.0

def range(start: Long, end: Long, step: Long): Dataset[Long]

Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value.

Since: 2.0.0

def range(start: Long, end: Long): Dataset[Long]

Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with step value 1.

Since: 2.0.0

def range(end: Long): Dataset[Long]

Creates a Dataset with a single LongType column named id, containing elements in a range from 0 to end (exclusive) with step value 1.

Since: 2.0.0

def read: DataFrameReader

Returns a DataFrameReader that can be used to read non-streaming data in as a DataFrame.

sparkSession.read.parquet("/path/to/file.parquet")
sparkSession.read.schema(schema).json("/path/to/file.json")

Since: 2.0.0

def readStream: DataStreamReader

Returns a DataStreamReader that can be used to read streaming data in as a DataFrame.

sparkSession.readStream.parquet("/path/to/directory/of/parquet/files")
sparkSession.readStream.schema(schema).json("/path/to/directory/of/json/files")

Since: 2.0.0

lazy val sessionState: SessionState

State isolated across sessions, including SQL configurations, temporary tables, registered functions, and everything else that accepts a org.apache.spark.sql.internal.SQLConf.

State isolated across sessions, including SQL configurations, temporary tables, registered functions, and everything else that accepts a org.apache.spark.sql.internal.SQLConf. If parentSessionState is not null, the SessionState will be a copy of the parent.

This is internal to Spark and there is no guarantee on interface stability.

Annotations: @Unstable() @transient()
Since: 2.2.0

lazy val sharedState: SharedState

State shared across sessions, including the SparkContext, cached data, listener, and a catalog that interacts with external systems.

This is internal to Spark and there is no guarantee on interface stability.

Annotations: @Unstable() @transient()
Since: 2.2.0

val sparkContext: SparkContext

def sql(sqlText: String): DataFrame

Executes a SQL query using Spark, returning the result as a DataFrame.

Executes a SQL query using Spark, returning the result as a DataFrame. This API eagerly runs DDL/DML commands, but not for SELECT queries.

Since: 2.0.0

def sql(sqlText: String, args: Map[String, Any]): DataFrame

Executes a SQL query substituting named parameters by the given arguments, returning the result as a DataFrame.

Executes a SQL query substituting named parameters by the given arguments, returning the result as a DataFrame. This API eagerly runs DDL/DML commands, but not for SELECT queries.

sqlText: A SQL statement with named parameters to execute.
args: A map of parameter names to Java/Scala objects that can be converted to SQL literal expressions. See Supported Data Types for supported value types in Scala/Java. For example, map keys: "rank", "name", "birthdate"; map values: 1, "Steven", LocalDate.of(2023, 4, 2). Map value can be also a Column of literal expression, in that case it is taken as is.

Annotations: @Experimental()
Since: 3.4.0

def sql(sqlText: String, args: Map[String, Any]): DataFrame

Executes a SQL query substituting named parameters by the given arguments, returning the result as a DataFrame.

Executes a SQL query substituting named parameters by the given arguments, returning the result as a DataFrame. This API eagerly runs DDL/DML commands, but not for SELECT queries.

sqlText: A SQL statement with named parameters to execute.
args: A map of parameter names to Java/Scala objects that can be converted to SQL literal expressions. See Supported Data Types for supported value types in Scala/Java. For example, map keys: "rank", "name", "birthdate"; map values: 1, "Steven", LocalDate.of(2023, 4, 2). Map value can be also a Column of literal expression, in that case it is taken as is.

Annotations: @Experimental()
Since: 3.4.0

val sqlContext: SQLContext

A wrapped version of this session in the form of a SQLContext, for backward compatibility.

Since: 2.0.0

def stop(): Unit

Stop the underlying SparkContext.

Since: 2.0.0

def streams: StreamingQueryManager

Returns a StreamingQueryManager that allows managing all the StreamingQuerys active on this.

Annotations: @Unstable()
Since: 2.0.0

final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes: AnyRef

def table(tableName: String): DataFrame

Returns the specified table/view as a DataFrame.

Returns the specified table/view as a DataFrame. If it's a table, it must support batch reading and the returned DataFrame is the batch scan query plan of this table. If it's a view, the returned DataFrame is simply the query plan of the view, which can either be a batch or streaming query plan.

tableName: is either a qualified or unqualified name that designates a table or view. If a database is specified, it identifies the table/view from the database. Otherwise, it first attempts to find a temporary view with the given name and then match the table/view from the current database. Note that, the global temporary view database is also valid here.

Since: 2.0.0

def time[T](f: ⇒ T): T

Executes some code block and prints to stdout the time taken to execute the block.

Executes some code block and prints to stdout the time taken to execute the block. This is available in Scala only and is used primarily for interactive testing and debugging.

Since: 2.1.0

def toString(): String

Definition Classes: AnyRef → Any

def udf: UDFRegistration

A collection of methods for registering user-defined functions (UDF).

The following example registers a Scala closure as UDF:

sparkSession.udf.register("myUDF", (arg1: Int, arg2: String) => arg2 + arg1)

The following example registers a UDF in Java:

sparkSession.udf().register("myUDF",
    (Integer arg1, String arg2) -> arg2 + arg1,
    DataTypes.StringType);

Since: 2.0.0
Note: The user-defined functions must be deterministic. Due to optimization, duplicate invocations may be eliminated or the function may even be invoked more times than it is present in the query.

def version: String

The version of Spark on which this application is running.

Since: 2.0.0

final def wait(): Unit

Definition Classes: AnyRef
Annotations: @throws( ... )

final def wait(arg0: Long, arg1: Int): Unit

Definition Classes: AnyRef
Annotations: @throws( ... )

final def wait(arg0: Long): Unit

Definition Classes: AnyRef
Annotations: @throws( ... ) @native()

object implicits extends SQLImplicits with Serializable

(Scala-specific) Implicit methods available in Scala for converting common Scala objects into DataFrames.

val sparkSession = SparkSession.builder.getOrCreate()
import sparkSession.implicits._

Since: 2.0.0

Packages

SparkSession

Companion object SparkSession

class SparkSession extends Serializable with Closeable with Logging

Value Members

Java Example

Example

Inherited from Logging

Inherited from Closeable

Inherited from AutoCloseable

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped

Packages

SparkSession 

Companion object SparkSession

class SparkSession extends Serializable with Closeable with Logging

Value Members

Java Example

Example

Inherited from Logging

Inherited from Closeable

Inherited from AutoCloseable

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped

SparkSession