Advantages
of using appropriate file formats:
Faster read
Faster write
Splitable files support
Schema evolution can be supported
Advanced compression can be achieved
Some things to consider when choosing the
format are:
The structure of your data: Some formats accept nested data such
as JSON, Avro or Parquet and others do not. Even, the ones that do, may not be
highly optimized for it. Avro is the most efficient format for nested data, I
recommend not to use Parquet nested types because they are very inefficient. Process
nested JSON is also very CPU intensive. In general, it is recommended to flat
the data when ingesting it.
Performance: Some formats such as Avro and Parquet perform
better than other such JSON. Even between Avro and Parquet for different use
cases one will be better than others. For example, since Parquet is a column-based
format, it is great to query your data lake using SQL whereas Avro is better
for ETL row level transformation.
Easy to read: Consider if you need people to read the data or
not. JSON or CSV are text formats and are human readable whereas more
performant formats such parquet or Avro are binary.
Compression: Some formats offer higher compression rates than
others.
Schema evolution: Adding or removing fields is far more
complicated in a data lake than in a database. Some formats like Avro or
Parquet provide some degree of schema evolution which allows you to change the
data schema and still query the data. Tools such Delta Lake format
provide even better tools to deal with changes in Schemas.
Compatibility: JSON or CSV are widely adopted and compatible
with almost any tool while more performant options have less integration points
Big Data file formats
CSV
CSV files
(comma-separated values) are usually used to exchange tabular data between
systems using plain text. CSV is a row-based file format, which means that each
row of the file is a row in the table. Essentially, CSV contains a header row
that contains column names for the data, otherwise, files are considered
partially structured. CSV files may not initially contain hierarchical or
relational data. Data connections are usually established using multiple CSV
files. Foreign keys are stored in columns of one or more files, but connections
between these files are not expressed by the format itself. In addition, the
CSV format is not fully standardized, and files may use separators other than
commas, such as tabs or spaces.
One of the other
properties of CSV files is that they are only splitable when it is a raw,
uncompressed file or when splitable compression format is used such as bzip2 or lzo (note: lzo needs
to be indexed to be splitable).
CSV is good option for compatibility, spreadsheet processing and
human readable data. The data must be flat. It is not efficient and cannot
handle nested data. There may be issues with the separator which can lead to
data quality issues. Use this format for exploratory analysis, POCs or small
data sets
Advantages
CSV is human-readable and
easy to edit manually;
CSV provides a simple scheme;
CSV can be processed by almost all existing
applications;
CSV is easy to implement and parse;
CSV is compact. For XML, you start a tag and
end a tag for each column in each row. In CSV, the column headers are written
only once.
Disadvantages
CSV allows you to work
with flat data. Complex data structures have to be processed separately from
the format;
No support for column
types. No difference between text and numeric columns;
There is no standard way
to present binary data;
Problems with CSV import
(for example, no difference between NULL and quotes);
Poor support for special
characters;
Lack of a universal
standard.
JSON
JSON (JavaScript object notation) data are
presented as key-value pairs in a partially structured format. JSON is often
compared to XML because it can store data in a hierarchical format. Both
formats are user-readable, but JSON documents are typically much smaller than
XML. They are therefore more commonly used in network communication, especially
with the rise of REST-based web services.
Since much data is
already transmitted in JSON format, most web languages initially support JSON.
With this huge support, JSON is used to represent data structures, exchange
formats for hot data, and cold data warehouses.
Many streaming
packages and modules support JSON serialization and deserialization. While the
data contained in JSON documents can ultimately be stored in more
performance-optimized formats such as Parquet or Avro, they serve as raw data,
which is very important for data processing (if necessary).
JSON (JavaScript object notation) data are
presented as key-value pairs in a partially structured format. JSON is often
compared to XML because it can store data in a hierarchical format. Both
formats are user-readable, but JSON documents are typically much smaller than
XML. They are therefore more commonly used in network communication, especially
with the rise of REST-based web services.
Since much data is
already transmitted in JSON format, most web languages initially support JSON.
With this huge support, JSON is used to represent data structures, exchange
formats for hot data, and cold data warehouses.
Many streaming
packages and modules support JSON serialization and deserialization. While the
data contained in JSON documents can ultimately be stored in more
performance-optimized formats such as Parquet or Avro, they serve as raw data,
which is very important for data processing (if necessary)
Advantages
JSON supports hierarchical structures,
simplifying the storage of related data in a single document and presenting
complex relationships;
Most languages provide simplified JSON
serialization libraries or built-in support for JSON
serialization/deserialization;
JSON supports lists of objects, helping to
avoid chaotic list conversion to a relational data model;
JSON is a widely used file format for NoSQL
databases such as MongoDB, Couchbase and Azure Cosmos DB
Built-in support in most modern tools;
Disadvantages
JSON consumes more memory due to repeatable
column names;
Poor support for special characters;
JSON is not very splitable;
JSON lacks indexing;
It is less compact as compared to over binary
formats.
Parquet
Parquet: Columnar storage. It has schema support. It works very well
with Hive and Spark as a way to store columnar data in deep storage that is
queried using SQL. Because it stores data in columns, query engines will only
read files that have the selected columns and not the entire data set as
opposed to Avro. Use it as a reporting layer
Unlike CSV and JSON, parquet files are binary
files that contain metadata about their contents. Therefore, without
reading/parsing the contents of the file(s), Spark can simply rely on metadata
to determine column names, compression/encoding, data types, and even some
basic statistical characteristics. Column metadata for a Parquet file is stored
at the end of the file, which allows for fast, single-pass writing.
Parquet is optimized for the paradigm Write
Once Read Many (WORM). It writes slowly but reads incredibly quickly,
especially when you only access a subset of columns. Parquet is good choice for
heavy workloads when reading portions of data. For cases where you need to work
with whole rows of data, you should use a format like CSV or AVRO.
Advantages
Parquet is a columnar format. Only the
required columns will be retrieved/read, this reduces disk I/O. The concept is
called projection pushdown.
The scheme travels with the data, so the data
is self-describing;
Although it is designed for HDFS, data can be
stored on other file systems such as GlusterFs or NFS;
Parquet just files, which means it's easy to
work, move, backup and replicate them;
Built-in support in Spark makes it easy to
simply take and save a file in storage;
Parquet provides very good compression up to
75% when using even compression formats like snappy;
As practice shows, this format is the fastest
for read-heavy processes compared to other file formats;
Parquet is well suited for data storage
solutions where aggregation on a particular column over a huge set of data is
required;
Parquet can be read and written using the Avro
API and Avro Schema (which gives the idea of storing all raw data in the Avro
format, but all processed data in parquet);
Disadvantages
It also provides predicate pushdown, thus
reducing the further cost of transferring data from storage to the processing
engine for filtering;
The column-based design makes you think about
the schema and data types;
Parquet
does not always have built-in support in tools other than Spark;
It does
not support data modification (Parquet files are immutable) and scheme
evolution. Of course, Spark knows how to combine the schema if you change it
over time (you must specify a special option while reading), but you can only
change something in an existing file by overwriting it
Avro
Avro: Great for
storing row data, very efficient. It has a schema and supports evolution. Great
integration with Kafka. Supports file splitting. Use it for row level
operations or in Kafka. Great to write data, slower to read.
Apache Avro was
released by the Hadoop working group in 2009. It is a row-based format that has
a high degree of splitting. It is also described as a data serialization system
similar to Java Serialization. The schema is stored in JSON format, while the
data is stored in binary format, which minimizes file size and maximizes
efficiency. Avro has reliable support for schema evolution by managing added,
missing, and changed fields. This allows old software to read new data, and new
software to read old data — it is a critical feature if your data can change.
Avro's ability to
manage scheme evolution allows components to be updated independently, at
different times, with a low risk of incompatibility. This eliminates the need
for applications to write if-else statements to handle different versions of
schema and eliminates the need for the developer to look at old code to
understand the old schema. Since all versions of the schema are stored in a
human-readable JSON header, it is easy to understand all the fields available
to you.
Since the schema is
stored in JSON and the data is stored in binary form, Avro is a relatively
compact option for both permanent storage and wire transfer. Since Avro is a
row-based format, it is the preferred format for handling large amounts of
records as it is easy to add new rows.
Advantages
Avro is a linguistic-neutral serialization of
data.
Avro stores the schema
in a file header, so the data is self-describing;
Easy and fast data serialization and
deserialization, which can provide
very good ingestion
performance.
As with the Sequence files, the Avro files
also contain synchronization markers to separate blocks. This makes it highly splitable.
Files formatted in Avro are splitable
and compressible and are therefore a good candidate for data storage
in the Hadoop ecosystem.
The schema used to read Avro files does not
necessarily have to be the same as the one used to write the files. This allows
new fields to be added independently of each other.
Disadvantages
Makes you think about the schema and data
types;
Its data is not human-readable;
Not integrated into every programming
language.
ORC
ORC, short for
Optimized Row Columnar, is a free and open-source columnar storage format
designed for Hadoop workloads. As the name suggests, ORC is a self-describing,
optimized file format that stores data in columns which enables users to read
and decompress just the pieces they need. It is a successor to the traditional
Record Columnar File (RCFile) format designed to overcome limitations of other
Hive file formats. It takes significantly less time to access data and also
reduces the size of the data up to 75 percent. ORC provides a more efficient
and better way to store data to be accessed through SQL-on-Hadoop solutions
such as Hive using Tez. ORC provides many advantages over other Hive file
formats such as high data compression, faster performance, predictive push down
feature, and moreover, the stored data is organized into stripes, which enable
large, efficient reads from HDFS
