Module 11: Advanced Big Data Architecture

staging area

operational data store (ODS)

data mart

analytical database

Although a single EDW can house multiple ODSs, because their primary role is to facilitate near-realtime reporting, their use is optional.

It may not be possible to extract data from all systems at the same time because of various technical or business-related issues. Due to this, a storage buffer where data extracted from different systems at varying times with differing frequencies can be stored is required

It is generally an insert/read-only database utilizing either shared-nothing MPP architecture or shared-everything architecture. Data is fed from the data warehouse into the analytical database on regular intervals

It usually includes an ETL process that ferries data from source systems into a temporary storage area. This process also contains data cleansing, validation and model transformation operations

generally contains recent data. However, the degree of “data freshness” depends upon the reporting requirements. As a result, the range of data stored may span from hours to months

a relational database that acts as the single version of truth for the enterprise by storing standardized data from across the enterprise in a denormalized form that is fit for reporting and data analysis

stores data related to various business entities, such as products or customers. Unlike an OLTP system, data is either inserted or retrieved but not updated in a data warehouse

the queries are generally more complex, involving multiple tables spanning a longer range of data.

It is generally an insert/read-only database utilizing either shared-nothing MPP architecture or shared-everything architecture

Data is highly standardized because it has gone through data cleansing, validation, quality and de-duplication processes, further suggesting that the data is of high value

Some basic level of data model transformation and denormalization may also be performed in support of efficient reporting

These are generally expensive and may come bundled with the required hardware and software in the form of an appliance

Contain high value data that has gone through rigorous validation and quality control checks

On the other hand, Big Data datasets must be stored in their raw unstructured forms, and their values are unknown

Big Data requires a repository that acts as a sink for a variety of data sources where data is stored as is

Stores data related to various business entities, such as products or customers

Big Data requires a distributed and highly scalable storage and processing architecture with scale-out support

Most implementations of the Big Data appliance enable realtime and near-realtime analytics without the need for integrating multiple disparate technologies

A batch processing engine, such as MapReduce, can be used to convert semi- and unstructured data into meaningful structured data

The next-generation data warehouse consists of heterogeneous technologies providing support for structured as well as semi- and unstructured data storage and analysis

The introduction of the Big Data platform in this configuration is comparatively less disruptive because the Big Data platform is essentially an add-on module for processing semi- and unstructured data

Provides a highly scalable data storage and processing environment

BI tools and other analytical applications are unable to make use of the Big Data platform directly

The implementation and maintenance of the interconnect can become complex if it incorporates complicated data processing, such as translation between different data types

relational and non-relational storage

configuration, management and application development environments

an interconnect (between data storage and processing resources)

is analogous to the parallel approach and is also known as the logical data warehouse

It requires complex initial configuration, which usually results in consultation costs

It is generally implemented as Data-as-a-Service (DaaS) by applying service-orientation principles.

This approach makes non-relational data (Big Data datasets) more accessible through the use of standardized interfaces

Is generally implemented through complex software that can be expensive to acquire

Can be utilized as a technology-enabler for Big Data under such circumstances

Ingested data is stored in a distributed file

A single dataset may be of interest to multiple clients developed using different technologies that require data to be available in a specific format

Specialized form of distributed computing that introduces utilization models for remotely provisioning scalable and measured IT resources

Processing and storage technologies that use cluster-based processing and storage resources

The on-demand and elastic nature provides the ability for a much quicker setup of a Big Data platform

Has the potential to provide the basic components for a Big Data solution environment, including data, storage and processing resources

Whether processing data in batch or realtime mode, the pay-per-use model can be fully utilized to build a cluster whose size can be regulated based on the volume and velocity characteristics of Big Data

Infrastructure-as-a-Service (IaaS)

Platform-as-a-Service (PaaS)

Software-as-a-Service (SaaS)

Component-as-a-Service (CaaS)

Heterogeneous Cloud

Private Cloud

Managed Cloud

Hybrid Cloud

Is ideal for enterprises that initially built up Big Data analytics in-house but now want to scale out.

Can be used when input datasets are already stored in the cloud

Is generally less secure but more scalable due to larger pooling of storage and processing resources

It is also ideal when datasets reside within an enterprise’s firewall.

It is also ideal when workloads vary

Is generally less secure but more scalable due to larger pooling of storage and processing resources

It is also ideal when datasets reside within an enterprise’s firewall

Can help develop low latency data analysis capabilities

It is also ideal when workloads vary

Can be used when input datasets are already stored in the cloud

Is a suitable choice when starting a Big Data project

is a suitable choice when using a combination of sensitive data and public datasets

data privacy

regulatory compliance

network connectivity

data virtualization

Cloud-based Big Data Analysis

Cloud-based Big Data Visualization

Cloud-based Big Data Storage

Cloud-based Big Data Processing

This pattern can also be employed when the data sources, such as the CRM system, reside in the same cloud (faster data transfer) or a proof-of-concept is being developed

This ability to store raw data spanning over longer periods of time increases the overall potential of finding valuable insights

Represents a solution environment comprised of inexpensive NoSQL storage

Is associated with the storage device (distributed file system/NoSQL) and data transfer engine mechanisms

This generally involves the use of NoSQL databases such that the downstream applications can communicate directly with these databases using RESTful APIs

The underlying idea is to be able to ingest large amounts of raw data and pre-process it in order to make it suitable for traditional enterprise systems

Keeping multiple copies of the same dataset in different formats is not only inefficient but also adds operational and storage overheads

The involved operations can include data cleansing, validation, model transformation and format transformation, as well as the joining of disparate datasets

Poly Source

Large-Scale Batch Processing

High Volume Tabular Storage

Large-Scale Graph Processing

Ingesting large amounts of data in order to calculate certain statistics or execute a machine learning and then to feed results to enterprise systems

This generally involves the use of NoSQL databases such that the downstream applications can communicate directly with these databases using RESTful APIs

The underlying idea is to be able to ingest large amounts of raw data and pre-process it in order to make it suitable for traditional enterprise systems

A dedicated storage layer helps store, pre-process and further integrate data with structured data without impacting the current storage infrastructure

High Volume Tabular Storage

Large-Scale Graph Processing

Canonical Data Format

Data Size Reduction

The results are fed directly to various downstream applications, such as an e-commerce application

Is generally employed by enterprises that have just embarked on a Big Data journey

Represents a standalone solution environment

Offloads existing databases from having to perform complex and long-running data transformation jobs on large datasets

Poly Storage

Poly Source

Poly Sink

Confidential Data Storage

The analytical operations performed in support of BI, data mining and creating statistical and machine learning models do not affect the performance

This configuration is generally opted for by enterprises that want to move towards predictive and prescriptive analytics by creating richer statistical and machine learning models

Capable of ingesting and storing large amounts of semi-structured and unstructured data to develop highfidelity statistical and machine learning models for performing predictive and prescriptive analytics

Although analogous to the use of a cloud, this approach provides a better alternative in terms of uploading data to the cloud as well as data security and privacy issues

Random Access Storage

Automated Dataset Execution

File-based Sink

Big Data Processing Environment.

The ingested data is stored to the distributed file system, where it is enriched via batch processing and then stored on a NoSQL database for performing analytical queries

Their current storage infrastructure does not allow them to store semi-structured and unstructured data

A solution environment where the sole purpose of using the Big Data platform is to offload processing of large amounts of structured data

This approach provides a better alternative in terms of uploading data to the cloud as well as data security and privacy issues

Canonical Data Format

Relational Sink

Automatic Data Replication and Reconstruction

Automatic Data Sharding

Storing and analyzing very large amounts of structured, unstructured and semi-structured Big Data datasets

A solution environment comprised of inexpensive storage used to store large amounts of data from both internal and external data sources in an online fashion ready for consumption by any enterprise system

Large data volumes are available and the data itself has not lost its value because it is kept unprocessed in its raw form

The sole purpose of using the Big Data platform is to offload processing of large amounts of structured data

Automated Dataset Execution

Streaming Access Storage

Random Access Storage

Canonical Data Format

This configuration is generally opted for by enterprises that want to move towards predictive and prescriptive analytics by creating richer statistical and machine learning models

Large data volumes are available and the data itself has not lost its value because it is kept unprocessed in its raw form

Storing and analyzing very large amounts of structured, unstructured and semi-structured Big Data datasets

Data from structured sources and from unstructured sources can first be stored on a distributed file system

Automatic Data Sharding

Canonical Data Format

Random Access Storage

Confidential Data Storage

a solution environment comprised of inexpensive NoSQL storage that is utilized as ___________ where large amounts of transactional data from operational systems across the enterprise are collected for operational BI and reporting

Data from structured sources and from unstructured sources can first be stored on a distributed file system

Large data volumes are available and the data itself has not lost its value because it is kept unprocessed in its raw form

Larger amounts of data that spreads over longer time periods can be stored, thereby providing the opportunity to enrich operational BI

High Volume Tabular Storage

Relational Sink

Indirect Data Access

Automated Dataset Execution

The data can be imported into fit-forpurpose NoSQL databases, where it can be easily accessed in support of BI, reporting and other analytical use cases

A solution environment capable of processing streams of data in realtime or near-realtime, such as performing analytics on machine-generated or social media data

The streaming data can be stored in disk-based storage, such as the distributed file system, for further analysis

Enable the processing of such datasets, which requires the use of a batch processing engine

Large-Scale Batch Processing

Streaming Source

Automatic Data Replication and Reconstruction

Data Size Reduction

Centralized Dataset Governance

Fan-in Ingress

Centralized Dataset Management

Streaming Egress Pattern

The sole purpose of using this kind of platform is to offload processing of large amounts of structured data

Type of Big Data solution architecture that is comprised of multiple layers and forms the basis for developing highly scalable, available, eventually consistent, fault tolerant and low latency realtime Big Data solutions

Uses a combination of both realtime and batch components that operate in parallel to process data without any delay

Additional processing is generally required to put the data in the correct structure

View

Model

Indexed View

Indexing

This not only helps process voluminous data faster but also helps cater to infrequent or ad-hoc data processing requests that require above-average storage and processing resources

Data architectures are becoming difficult to design and maintain due to the ever-increasing volume, velocity and variety of data.

Efficient data storage and efficient querying have incompatible requirements that require following different strategies

Data is either stored in a disk-based NoSQL or a memory-based storage device, which can be a NoSQL or some other cluster-based storage technology, that enables low latency data access to perform realtime or near-realtime analytics

Processes raw data by employing both realtime and batch data processing techniques in parallel

Maintain data lineage and details about operations performed on the data across multiple processing stages

The results generated by realtime processing are based on incremental algorithms that may not be consistent/accurate

Batch data processing eliminates the complexity of maintaining data consistency across nodes by storing only immutable data

Batch

Serving

Speed

Query

Processing of raw data

Storage of raw data

Ad hoc reporting

Calculation of views

Uses incremental algorithms and processes comparatively smaller amounts of data to provide low latency results

Consists of a storage device (distributed file system), batch processing engine and a workflow engine

Uses a recomputation algorithm to provide consistent accurate views and further provides fault-tolerance when compared with an incremental algorithm

Comprises an enhanced version of the query engine with logic that can automatically and intelligently combine serving and speed views based on the query criteria

Although raw data is stored, for achieving consistency, some structure needs to be applied to the data before storage

The storage device used in this layer only needs to support batch write (no random write) with random read capabilities

As the layer follows the mutable storage model and the processing results are generated more frequently, the storage device that stores the views needs to support random writes with random reads

For keeping the complexity to a minimum and providing faster reads, normally a simple key-value NoSQL database is used

The use of an append-only and streaming data storage device keeps complexity to a minimum

The views created by the batch layer are not amenable to random querying, as these are generally stored in the distributed file system

A memory-based storage device for the storage of raw data and a memory or disk-based NoSQL storage device for the storage of views is generally used

Event data is captured using the event data transfer engine and is processed in memory via the realtime processing engine to create indexed views that are generally stored inside a NoSQL database

For easier integration, the speed and serving views should be constructed in a modular manner

Merging the results from views residing in the speed and serving layers for successfully executing a query

Once the latest batch view is available via the serving layer, the corresponding results in the realtime views can be ignored or flushed

Is a high latency layer such that there is a time lag before the latest version of the views, based on fresher data, is available

Raw data is fed simultaneously to the batch and speed layers, generally using the same event data transfer engine

The batch layer can be further used for deep analytics, as it contains complete datasets

The limitations of the SCV principle are also relaxed

Although the speed layer is responsible for processing the entire set of fresh data while the corresponding batch view is not ready, it does not process the entire set as a single job because doing so adds to the latency and results in excessive resource usage

Algorithms for the speed layer can be complex or might need some time to understand, as they use incremental or approximation (probability)-based techniques that the batch equivalent may not be using

The complexity of the architecture is restricted to the speed layer, as that is where the incremental algorithms and read/write database are used

The immutable nature of the batch layer helps re-process data as a result of a data processing logic change that may occur due to new business requirements or a bug fix

Realtime data processing capability is required with consistent results

Realtime data processing capability is required with consistent results

Fault-tolerance and accuracy need to be added to the existing realtime system

Loss of data is not acceptable

Polyglot persistence by employing fit-for-purpose storage devices at each layer

Configuring the batch layer to process data in small batches reduces load on the speed layer

Raw data is fed simultaneously to the batch and speed layers, generally using the same event data transfer engine, and each layer can be implemented via a different set of technologies

Complexity is greatly increased, as two separate layers need building and maintaining while ensuring that each provides the same functionality

Requires schema adherence in the batch layer, which adds complexity, adds another step before data can actually be persisted and requires prior knowledge about the structure of the incoming data

Employing the same processing engine for both the speed and batch layer, such as Spark, helps keep system complexity to a minimum

The key-value storage model employed in the serving layer may not be sufficient for all types of query requirements

The immutable nature of the batch layer helps re-process data as a result of a data processing logic change that may occur due to new business requirements or a bug fix

A balance is required based on the processing requirements, as the throughput obtained from employing small batches may be less than from larger batches and will further require frequent updates to the serving layer

batch layer

serving layer

speed layer

query layer

A dedicated storage layer helps store, pre-process and further integrate data with structured data without impacting the current storage infrastructure

The underlying idea is to be able to ingest large amounts of raw data and pre-process it in order to make it suitable for traditional enterprise systems

Is ideal for enriching the EDW with unstructured data

This generally involves the use of NoSQL databases such that the downstream applications can communicate directly with these databases using RESTful APIs

A dedicated storage layer helps store, pre-process and further integrate data with structured data without impacting the current storage infrastructure

Certain statistics are calculated by processing large amounts of data, or a statistical/machine learning model is run

Solution environment capable of storing high-volume and high-variety data in order to perform various analytics in isolation from other enterprise systems

Examples of functionality enhancement include personalized recommendations and discounts as well as targeted advertisements

Although analogous to the use of a cloud, this approach provides a better alternative in terms of uploading data to the cloud as well as data security and privacy issues

The underlying idea is to be able to ingest large amounts of raw data and pre-process it in order to make it suitable for traditional enterprise systems

Is not integrated with the EDW and is instead used directly to explore data and perform analytics

Keep the Big Data initiative separate from existing IT operations and systems

This configuration is generally opted for by enterprises that want to move towards predictive and prescriptive analytics

Although analogous to the use of a cloud, this approach provides a better alternative in terms of uploading data to the cloud as well as data security and privacy issues

A dedicated storage layer helps store, pre-process and further integrate data with structured data without impacting the current storage infrastructure

Generally, the ingested data is stored to the distributed file system, where it is enriched via batch processing and then stored on a NoSQL database for performing analytical queries

Such a solution is generally employed by enterprises that have just embarked on a Big Data journey

Once processed, the streaming data can be stored in disk-based storage, such as the distributed file system, for further analysis

This not only helps process voluminous data faster but also helps cater to infrequent or ad-hoc data processing requests

Although analogous to the use of a cloud, this approach provides a better alternative in terms of uploading data to the cloud as well as data security and privacy issues

The data can be imported into fit-forpurpose NoSQL databases, where it can be easily accessed in support of BI

Large data volumes are available and the data itself has not lost its value because it is kept unprocessed in its raw form

Based on the data storage requirements, a distributed file system or a NoSQL database can be used for data storage

Large amounts of transactional data from operational systems across the enterprise are collected

This not only helps process voluminous data faster but also helps cater to infrequent or ad-hoc data processing requests that require above-average storage and processing resources

Setting up a cluster in-house may result in under-utilization of processing resources, as it would not be utilized at all times

Is associated with the processing engine, storage device, resource manager and coordination engine mechanisms

Enable the processing of such datasets, which requires the use of a batch processing engine