ORC Files

Apache ORC is a columnar format which has more advanced features like native zstd compression, bloom filter and columnar encryption.

ORC Implementation

Spark supports two ORC implementations (native and hive) which is controlled by spark.sql.orc.impl. Two implementations share most functionalities with different design goals.

For example, historically, native implementation handles CHAR/VARCHAR with Spark’s native String while hive implementation handles it via Hive CHAR/VARCHAR. The query results are different. Since Spark 3.1.0, SPARK-33480 removes this difference by supporting CHAR/VARCHAR from Spark-side.

Vectorized Reader

native implementation supports a vectorized ORC reader and has been the default ORC implementation since Spark 2.3. The vectorized reader is used for the native ORC tables (e.g., the ones created using the clause USING ORC) when spark.sql.orc.impl is set to native and spark.sql.orc.enableVectorizedReader is set to true.

For the Hive ORC serde tables (e.g., the ones created using the clause USING HIVE OPTIONS (fileFormat 'ORC')), the vectorized reader is used when spark.sql.hive.convertMetastoreOrc is also set to true, and is turned on by default.

Schema Merging

Like Protocol Buffer, Avro, and Thrift, ORC also supports schema evolution. Users can start with a simple schema, and gradually add more columns to the schema as needed. In this way, users may end up with multiple ORC files with different but mutually compatible schemas. The ORC data source is now able to automatically detect this case and merge schemas of all these files.

Since schema merging is a relatively expensive operation, and is not a necessity in most cases, we turned it off by default . You may enable it by

  1. setting data source option mergeSchema to true when reading ORC files, or
  2. setting the global SQL option spark.sql.orc.mergeSchema to true.


Since Spark 3.2, you can take advantage of Zstandard compression in ORC files. Please see Zstandard for the benefits.

CREATE TABLE compressed (
  key STRING,
  value STRING
  compression 'zstd'

Bloom Filters

You can control bloom filters and dictionary encodings for ORC data sources. The following ORC example will create bloom filter and use dictionary encoding only for favorite_color. To find more detailed information about the extra ORC options, visit the official Apache ORC websites.

CREATE TABLE users_with_options (
  name STRING,
  favorite_color STRING,
  favorite_numbers array<integer>
  orc.bloom.filter.columns 'favorite_color',
  orc.dictionary.key.threshold '1.0',
  orc.column.encoding.direct 'name'

Columnar Encryption

Since Spark 3.2, columnar encryption is supported for ORC tables with Apache ORC 1.6. The following example is using Hadoop KMS as a key provider with the given location. Please visit Apache Hadoop KMS for the detail.

CREATE TABLE encrypted (
  ssn STRING,
  email STRING,
  name STRING
  hadoop.security.key.provider.path "kms://http@localhost:9600/kms",
  orc.key.provider "hadoop",
  orc.encrypt "pii:ssn,email",
  orc.mask "nullify:ssn;sha256:email"

Hive metastore ORC table conversion

When reading from Hive metastore ORC tables and inserting to Hive metastore ORC tables, Spark SQL will try to use its own ORC support instead of Hive SerDe for better performance. For CTAS statement, only non-partitioned Hive metastore ORC tables are converted. This behavior is controlled by the spark.sql.hive.convertMetastoreOrc configuration, and is turned on by default.


Property NameDefaultMeaningSince Version
spark.sql.orc.impl native The name of ORC implementation. It can be one of native and hive. native means the native ORC support. hive means the ORC library in Hive. 2.3.0
spark.sql.orc.enableVectorizedReader true Enables vectorized orc decoding in native implementation. If false, a new non-vectorized ORC reader is used in native implementation. For hive implementation, this is ignored. 2.3.0
spark.sql.orc.columnarReaderBatchSize 4096 The number of rows to include in an orc vectorized reader batch. The number should be carefully chosen to minimize overhead and avoid OOMs in reading data. 2.4.0
spark.sql.orc.columnarWriterBatchSize 1024 The number of rows to include in an orc vectorized writer batch. The number should be carefully chosen to minimize overhead and avoid OOMs in writing data. 3.4.0
spark.sql.orc.enableNestedColumnVectorizedReader true Enables vectorized orc decoding in native implementation for nested data types (array, map and struct). If spark.sql.orc.enableVectorizedReader is set to false, this is ignored. 3.2.0
spark.sql.orc.filterPushdown true When true, enable filter pushdown for ORC files. 1.4.0
spark.sql.orc.aggregatePushdown false If true, aggregates will be pushed down to ORC for optimization. Support MIN, MAX and COUNT as aggregate expression. For MIN/MAX, support boolean, integer, float and date type. For COUNT, support all data types. If statistics is missing from any ORC file footer, exception would be thrown. 3.3.0
spark.sql.orc.mergeSchema false

When true, the ORC data source merges schemas collected from all data files, otherwise the schema is picked from a random data file.

spark.sql.hive.convertMetastoreOrc true When set to false, Spark SQL will use the Hive SerDe for ORC tables instead of the built in support. 2.0.0

Data Source Option

Data source options of ORC can be set via:

Property NameDefaultMeaningScope
mergeSchema false sets whether we should merge schemas collected from all ORC part-files. This will override spark.sql.orc.mergeSchema. The default value is specified in spark.sql.orc.mergeSchema. read
compression snappy compression codec to use when saving to file. This can be one of the known case-insensitive shorten names (none, snappy, zlib, lzo, zstd and lz4). This will override orc.compress and spark.sql.orc.compression.codec. write

Other generic options can be found in Generic File Source Options.