Big Data Analysis
differs from traditional data analysis primarily because of the volume, velocity and variety characteristics of the data it processes
When two variables are considered to be _____________ they are considered to be aligned based on a linear relationship
This means that when one variable changes, the other variable also changes proportionally and constantly
this helps maintain data provenance throughout the big data analysis lifecycle, which helps establish and preserve data accuracy and quality
step-by-step process
is needed to organize the task involved with retrieving, processing, producing and repurposing data
therefore, it is advisable to store a verbatim copy of the original dataset before proceeding with the filtering. To save on required storage space, the verbatim copy is compressed before storage
Although the data format may be the same, the data model may be different
depending on the business scope of the analysis project and nature of the business problems being addressed, the requiered datasets and their sources can be internal and/or external to the enerprise
Big Data Analysis Lifecycle
Business Case Evaluation
Data Identification
A/B Testing
An ________ is employed when the comparatively simple data manipulation functions of a query engine are insufficient
Data Adquisition & Filtering
Data Extraction
suggestions commonly pertain to recommending items, such as movies, books, web pages, people, etc.
the processing engine mechanism will often use the ___________ to coordinate data processing across a large number of servers. This way, the processing engine does not require its own coordination logic
Data Validation & Cleansing
Data Aggregation & Representation
The ______ itself is a visual, color-coded representation of data values
Data Analysis
Data Visualization
A _______ is generally expressed using a line chart, with time plotted on the x-axis and recorded data value plotted on the y-axis
Utilization of Analysis Results
requires that a business case be created, assessed and approved prior to proceeding with the actual hands-on analysis task
helps decision-makers understand the business resources that will need to be utilized and which business challenges the analysis will tackle
Unstructured text is generally much more difficult to analyze and search, compared to structured text
is an example of the application of the law of large numbers
the further identification of KPI during this stage helps determine how closely the data analysis outcome needs to meet the identified goals and objectives
based on the business requirements documented, it can be determined whether the business problems being addressed are really Big Data problems
Applications for ___________ include fraud detection, medical diagnosis, network data analysis and sensor data analysis
can be applied to the categorization of unknown documents, and to personalized marketing campaings by grouping together customers with similar behavior
in order to qualify as a Big Data Problem, a business problem needs to be directly related to one or more of the Big Data characteristics of volume, velocity or variety
Note also that another outcome of this stage is the determination of the underlying budget required to carry out the analysis project
The processing engine enables data to be queried and manipulated in other ways, but to implement this type of functionality requires custom programming
Workflow Engine
any required purchase of tools, hardware, training, etc. need to be understood in advance, so that the anticipated investment can be weighed against the expected benefits of archieving the goals
initial iteration of the big data analysis lifecycle will require more up-front investment of Big Data technologies, products and training compared to later iterations where these earlier investment can be repeatedly leveraged
In the context of traditional data analysis, the ______ states that, starting with a reasonably large sample size, the value obtained from the analysis of additional data decreases as more data is successively added to the original sample
is dedicated to identify datasets (and their sources) required for the analysis project
identifying a wider variety of data sources may increase the probability of finding hidden patterns and correlations
Classification, outlier detection, filtering, natural language processing, text analytics and sentiment analysis can utilize ___________
Clustering
it can be beneficial to identify as many types of related data sources and insights as possible, especially when we dont know exactly what we're looking for
examples of appenden metadata can include dataset size and structure, source information, date and time of creation or collection, language-specific information, etc.
is generally used in data mining to get an understanding of the properties of a given dataset. After developing this understanding, classification can be used to make better predictions about similar, but new or unseen data
Internal Datasets
a list of available datasets from sources, such as data marts and operational systems, are typically compiled and matched against a predefined dataset specification
A distributed Big Data solution that needs to run on multiple servers relies on the coordination engine mechanism to ensure operational consistency across all of the participating servers
Subsequent to __________ being made available to business users to support business decision-making (such as via dashboard), there may be further oportunities to utilize the __________
In other areas such as the scientific domains, the objective may simply be to observe which version works better in order to improve a process or product
External Dataset
a list of possible third-party data providers (data markets and publicity available datasets), are generally compiled. Some forms of external data may be embedded within blogs or other types of content-based Websites, in which case they may need to be harvested via automated tools
Reconciling these differences can require complex logic that is executed automatically without the need for human intervention
A big data _________ utilizes a distributed parallel programming framework that enables it to process very large amounts of data distributed across multiple nodes
Data adquisition & filtering
the data is gathered from all of the data sources that were identified during the previous stage and is then subjected to the automated filtering of corrupt data or data that has been deemed to have no value to the analysis objectives
depending on the type of data source, data may come as a dump of files (such as data purchased from a third-party data provider), or may require API integration (such as with Twitter)
A ________ engine enables data to be moved in or out big data solution storage devices
A _________ comprises grouped read/writes, with a larger data footprint consisting of complex joins and high-latency responses
In many cases, especially where external, unstructured data is concerned, some or most of the acquired data may be irrelevant (noise) and can be discarded as par of the filtering process
data classified as "corrupt" can include records with missing or nonsensical values or invalid data types
it involves plotting entities as nodes and connections as edges between nodes
OLTP and operational systems (write-intensive) as well as operational BI and analytics (read-intensive), both fall within this category
data that is filtered out for one analysis may possibly be valuable for a different type of analysis
Coordination Engine
both internal and external data needs to be persisted once it gets generated or enters the enterprise boundary
for batch analytics, this data is persisted to disk prior to analysis
extracting text for text analytics, which requires scans of whole documents, will not be necessary if the underlying Big Data solution can already read the document in its native format directly
is dedicated to determining how and where processed analysis data can be further leveraged
in the case of realtime analytics, the data is analyzed first and then persisted to disk
metadata can be added via automation to data from both internal and external data sources to improve the classification and querying
Also known as offline processing, ________ processing involves processing data in batches and usually imposes delays (resulting in high-latency responses)
is generally applied via the following two approaches: collaborative ____________ and content-based ____________
it is vital that metadata be machine-readable and passed forward along subsequent analysis stages
The ability to analyze massive amounts of data and find useful insights carries little value if the only ones that can interpret the results are the analysis
both version are subjected to an experiment simultaneously, the observations are recorded to determine which version is more successful
metadata is added through an automated mechanism to data received from both internal and external data sources
Some of the data identified as input for the analysis may arrive in a format incompatible with the big data solution
the need to address disparate types of data is more likely with data from external sources
make it possible to develop highly reliable, highly available distributed big data solutions that can be deployed in a cluster
Data needs to be imported before it can be processed by the big data solution
is dedicated to extracting disparate data and transforming it into a format that the underlying big data solution can use for the purpose of the data analysis
the extend of extraction and transformation required depends on the types of analytics and capabilities of the big data solution
provenance can play an important role in determining the accuracy and quality of qustionable data
is closely related to parallel data processing in how the same principle of "divide-and-conquer" is applied
further transformation is needed in order to separate the data into two separate fields as required by the big data solution
it can also be used to make predictions about the values of the dependent variable while it is still unknown
However, ___________ is always archieved through physically separate machines that are networked together as a cluster
Invalid data can skew and falsify analysis results
Unlike traditional enterprise data where the data structure is pre-defined and data is pre-validated, data input into big data analyses can be unstructured without any indication of validity
More than one independent variable can be tested at the same time
The _______ essencially acts a resource arbitrator that manages and allocates available resources
its complexity can further make it difficult to arrive at a set of suitable validation constraints
is dedicated to establishing (often complex) validation rules and removing any known invalid data
is closely related to the concept of classificatopm and clustering, although its algorithms focus on finding abnormal values
for batch analytics, ______________ can be achieved via an offline ETL operation
Big data solutions often receive redundant data across different datasets
this redundancy can be exploited to explore interconnected datasets in order to assemble validation parameters and fill in missing valid data
A ___ represents a geographic measure by which different regions are color-coded according to a certain theme
A _________ is a file system that can store large files spread across a cluster
The presence of invalid data is resulting in spikes. Although the data appears abnormal, it may be indicative of a new pattern
for realtime analytics, a more complex in-memory system is required to validate and cleanse the data at the source
data that appears to be invalid may still be valuable in that it may possess hidden patterns and trends
No hypothesis or predetermined assumptions are generated
A _______ database is a non-relational database that is highly scalable, fault-tolerant and specifically designed to house unstructured data
Data may be spread across multiple datasets, requiring that datasets be joined together via common fields, in other cases, the same data fields may appear in multiple datasets
Either way, a method of data reconciliation is required or the dataset representing the correct value needs to be determined
Law of Diminishing Marginal Utility
A ______ can be in the form of a chart or a map
is dedicated to integrating multiple datasets together to arrive at a unified view
future dara analysis requirements need to be considered during this stage to help foster data reusability
The ________ mechanism can also be used for support distributed locks, support distributed queues, establish a highly available registry for obtaining configuration information, reliable asynchronous communication between processes that are running on different servers
essentially provides the ability to discover text rather than just search it
Data structure and Semantics
performing the stage of data aggregation & representation can become complicated because of differences in this
Within Big Data ________ can first be applied to discover if a relationship exists
Data structure
are an effective visual analysis technique for expressing patterns, data compositions via part-whole relations and geographic distributions of data
Hadoop's batch-based data processing fully lends itself to the pay -per-use model of __________, which can reduce operational costs since a typical Hadoop cluster size can range from a few to a few thousand nodes
Semantics
A value that is labelled differently in two different datasets may mean the same thing
Instead of hard-coding the required learning rules, either supervised or unsupervised machine learning is applied to develop the computer's understanding of the __________
Network Analysis
Data Aggregation
The large volumes processed by Big Data solutions can make____________ a time and effort-intensive operation
Whether _____________ is required or not, it is important to understand that the same data can be stored in many different forms. One form may be better suited for a particular type of analysis than another
require processing resources that they request from the resource manager
the data is then analyzed to prove or disprove the hypothesis and provide definitive answers to specific questions
Data structure standarized
can act as a common denominator that can be used for a range of analysis techniques and projects. This can require establishing a central, standard analysis repository, such as a NoSQL database
The _____ essencially acts a resource arbitrator that manages and allocates available resources
comprise random read/writes that involve fewer joins and require low-latency responses, with a smaller data footprint
is dedicated to carrying out the actual analysis task, which typically involves one or more types of analysis
this stage can be iterative in nature, especially if the _________________ is exploratory so that analysis is repeated until the appropiate pattern or correlation is uncovered
A _______ may internally use a processing engine to process multiple large datasets in parallel
the accuracy and applicability of the patterns and relationships that are found in a large dataset will be higher than that of a smaller dataset
the exploratory analysis approach is explained shortly, along with confirmatory analysis
depending on the type of analytics required, this stage can be as simple as querying a dataset to compute an aggregation for comparision
Correlation, regression, time series analysis, classification, clustering, outlier detection, filtering, natural language processing, text analytics and sentiment analysis are considered forms of ________
it can be as challenging as combining data mining and complex statistical analysis techniques to discover patterns and anomalies, or to generate a statistical or mathematical model to depict relationship between variables
The approach taken when carrying out this stage can be classified as confirmatory analysis or exploratory analysis (the latter is linked to data mining)
The results of completing the _______________ stage provide users with the ability to perform visual analysis, allowing for the discovery of answers to questions that users have not yet even formulated
A given ______ may support either data ingress or egress functions
Confirmatory Data Analysis
_____________ is a deductive approach where the cause of phenomenon being investigated is proposed beforehand
can be used to determine the number of entities that fall within a certain radius of another entity
hypothesis
The proposed cause or assumption is called a ____________
is an item filtering technique based on the collaboration (merging) of users' past behavior
This type of environment is provided by a platform that is comprised of a set of distributed storage and processing technologies
As the amount of digitized documents, e-mails, social media posts and log files increases, business have an increasing need to leverage any value that can be extracted from these forms of semi-structured and unstructured data
Data samples are typically used
this information can then be integrated into the decision-making process
Unexpected findings or anomalies are usually ignored since a predetermined cause was assumed
Exploratory Data Analysis
_____________ is an inductive approach that is closely associated to data mining
can be carried out via the use of supported by correlation, heat maps, time series analysis, network analysis, spatial data analysis, clustering, outlier detection, natural language processing and text analytics
Instead, the data is explored through analysis to develop an understanding of the cause of the phenomenon
Although it may not provide definitive answers, this method provides a general direction that can facilitate the discovery of patterns or anomalies
represents a constant rate of change
Large amounts of data and visual analysis are typically used
Data visualization
is dedicated to using __________________ techniques and tools to graphically communicate the analysis results for efective interpretarion by business users
is the process of finding data that is significantly different from or inconsistent with the rest of the data within a given dataset
Business users need to be able to understand the results in order to obtain value from the analysis and subsequently have the ability to provide feedback
is an item filtering technique focused on the similarity between users and items
The same results may be presented in a number of different ways, which can influence the interpretation of the results
Consequently, it is important to use the most suitable visualization technique by keeping the business domain in context
Another aspect to keep in mind is that providing a method of drilling down to comparatively simple statistics were generated
The objective is to use graphic representations to develop a deeper understanding of the data being analyzed. Specifically, it helps identify and highlight hiden patterns, correlations and anomalies
Analysis results
Natural Language Processing
A ___________ provides the ability to design and process a complex sequence of operations that can be triggered either at set time intervals or when data becomes available
includes both text and speech recognition
Utilization Analysis results
Depending on the nature of the analysis problems being addressed, it is possible for the analysis results to produce "models" that encapsulate new insights and understandings about the nature of the patterns and relationships that exist within the data that was just analyzed
Data Transfer Engine
A model look like a mathematical equation or a set of rules
Models can be used to improve business process logic, application system logic and can form the basis of a new system or software program
new models may be used to improve the programming logic within existing enterprise systems or may form the basis of new systems
Input for enterprise systems
Filtering
the data analysis results may be automatically (or manually) fed directly into enterprise systems to enhance and optimized their behavior and performance
Business Process Optimization
The identified patterns, correlations and anormalies discovered during the data analysis are used to refine business processes
models may also lead to opportunities to improve business process logic
is a computer's ability to comprehend human speech and text as naturally understood by humans
Alerts
Data analysis results can be used as input for existing _______ or may form the basis of new _______
Text Analytics
Recommender systems may also be based on a hybrid of both collaborative _______ and content-based _______ to fine-tune the accuracy and effectiveness of generated suggestions
Data Analysis Techniques
Statistical Analysis
Visual Analysis
Note that distributed file systems and databases are both on disk _________ mechanisms
Machine Learning
Semantic Analysis
Each node in the _____ has its own dedicated resources such as memory and hard drive and runs its own operating system just like a desktop computer
Correlation
Regression
Classification
The use of ________ can reduce development time and enables the manipulation of large datasets without the need to write complex programming logic
Outlier Detection
Some propietary ________ also provide specialized data analysis features, such as text analytics and machine log analysis processing
Heat Maps
Time series analysis
Spatial Data Analysis
it can be based on either supervised or unsupervised learning
suggest that there is no relationship at all between the two variables
Applications of __________ include operations and logistic optimization, environmental sciences and infrastructure planning
Sentiment Analysis
uses statistical methods based on mathematical formulas as a means for analyzing data
this type of analysis is commonly used to describe datasets via summarization, such as providing the mean, median or mode of statistics associated with the dataset
Spatial or geospatial data is commonly used to identify the geographic location of individual entities
it can also be used to infer patterns and relationships within the dataset, such as regression and correlation
is a supervised learning technique by which data is classified into relevant, previously learned categories
We may be further interested in discovering how closely Variables A and B are related, which means we may also want to analyze the extend to which Variable B increases in relation to Variable A's increase
also known as split or bucket testing compares two versions of an element to determine which version is superior based on a pre-defined metric
the element can be a range of things
is expressed as a decimal number between -1 to +1, which is known as the _____________ coefficient
the current version of the element is called the control version, whereas the modified version is called the treatment
Although __________ can be implemented in almost any domain, it is most often used in marketing
Generally, the objective is to gauge human behavior with the goal of increasing sales
This is a traditional data analysis principle that claims that data held in a reasonably sized dataset provides the maximum value
is an analysis technique used to determine whether two variables are related to each other
if they are found to be related, the next step is to determine what their relationship is
Query Engine
In general, the more learning data the computer has, the more correctly it can decipher human text and speech
The use of ________ helps to develop and understanding of a dataset and find relationships that can assist in explaining a phenomenon
Is therefore commonly used for data mining where the identification of relationships between variables in a dataset leads to the discovery of patterns and anomalies
This can reveal the nature of the dataset or the cause of a phenomenon
Migrating to the cloud is logical for enterprises planning to run analytics on datasets that are available via data markets, as most data markets store their data in the cloud such as Amazon S3
Big Data solutions require a distibuted processing environment that can accomodate large-scale data volumes, velocity and variety
Correlated
items can be ______ either based on a user's own behavior or by matching the behavior of multiple users
Note that a workflow engine may provide integration with a _______ to enable the automated import and export data
The degree of relationship changes from being strong to weak when moving from -1 to 0 or +1 to 0
For example, a __________ can execute logic that collects relational data from multiple databases at regular intervals via the data transfer engine mechanism, applies a set of ETL operations via the processing engine mechanism and finally persists the results to a NoSQL storage device
+1
suggest that there is a strong positive relationship between the two variables
When one variable increases, the other also increases and viceversa
typically involve large quantities of data with sequential read/writes, and comprises a group of read or write queries
0
when one increases, the other may stay the same, or increase or decrease arbitrarily
-1
suggest that there is a strong negative relationship between the two variables
when one variable increases, the other decreases and viceversa
Therefore, the value of each additional batch does not diminish value; rather, it provides more value
The analysis technique of _________ explores how a dependent variable is related to an independent variable within a dataset
As a sample scenario, __________ could help determine the type of relationship that exists between temperature (independent variable) and crop yield (dependent variable)
Applying this technique helps determine how the value of the dependent variable changes in relation to change in the value of the independent variable
When the independent variable increases, for example, does the dependent variable also increase? If yes, is the increase in a linear or non-linear proportion?
However, in such cases only one independent variable may change. The others are kept constant
can help enable a better undestanding of what a phenomenon is, and why it occurred
Users of Big Data solutions can make numerous data processing requests, each of which can have different processing workload requirements
Linear regression
the ______ states that the confidence with which predictions can be made increases as the size of the data that is being analyzed increases
Non-linear regression
A ________ is a method of storing and organizing data on a storage medium, such as hard drives, DVD´s, and flash drives
represents the variable rate of change
does not imply a causation. The change in the value of one variable may not be responsible for the change in the value of the second variable, although both may change at the same rate
assumes that both variables are independent
deal with already identified dependent and independent variables
implies that there is a degree of causation between the dependent and independent variables that may be direct or indirect
can then be applied to further explore the relationship and predict the values of the dependent variable, based on the known values of the independent variables
is a form of data analysis that involves the graphic representation of data to enable or enhance its visual perception
based on the premise that humans can understand and draw conclusions from graphics more quickly than from text, _______ act as a discovery tool in the field of Big Data
is also directly related to exploratory data analysis, as it encourages the formulation of questions from different angles
they also facilitate the identification of areas of interest and the discovery of extreme (high/low) values within a dataset
Each value is given a color according to its type, or the range that it falls under
Solely analyzing operational (structured) data may cause businesses to miss out on cost-saving or business expansion opportunities, especially those that are customer-focused
Instead of coloring the whole region, the map may be superimposed by a layer made up of collections of colored shapes representing various regions
chart
A _____ represents a matrix of values in which each cell is color-coded according to the value
map
NLP, Text analytics and sentiment analysis be used in support of __________
The data collected for _______ is always time-dependent
Named Entities(person, group, place, company), Pattern-Based Entities(social insurance number, zip code), Concepts (an abstract representation of a entity), Facts (relationship between entities)
Time series Analysis
is the analysis of data that is recorded over periodic intervals of time
this type of analysis makes use of _________, which is a time-ordered collections of values recorded over regular time intervals
helps to uncover patterns within data that are time-dependent. Once identified, the pattern can be extrapolated for future predictions.
are usually used for forecasting by identifying long-term trends, seasonal periodic patterns and irregular short-term variations in the dataset
Big Data solutions can be partially or fully deployed in clouds in order to leverage the storage and computing resources that are available from the cloud provider
Unlike other types of analyses, _________ always includes time as a comparision variable
is solely based on the similarity between users' behavior, and requires a large amount of user behavior data in order to accurately filter items
is the specialized analysis of text through the application of data mining, machine learning and natural language processing techniques to extract value out of unstructured text
Network
Within the context of visual analysis, a _______ is a interconnected collection of entities
Entity
An ____ can be a person, a group or some other business domain object such as a product
may be connected with another directly or indirectly
Also known as online processing, ____________ processing follows an approach whereby data is processed interactively, without delay (resulting in low-latency responses)
Some connections may only be one-way, so that traversal in the reverse direction is not possible
is a technique that focuses on analyzing relationships between entities within the network
provides analysis features more sophisticated than those of heat maps
There are specialized variations of __________ include route optimization, social network analysis and spread prediction
are based on predictive analytics techniques and therefore are associated with the same analysis techniques as predictive analytics. Additionally, _____ may utilize heat maps, network analysis and spatial data analysis to graphically show various outcomes
focused on analyzing location-based data in order to find different geographic relationship and patterns between entities
is manipulated through a geographical information system (GIS) that plots spatial data on a map generally using its longitude and latitude coordinates
With the ever-increasing availability of location-based data, _________ can be analyzed to gain location insights
Data used as input for_________ can either contain exact locations (longitude,latitude) or the information required to calculate locations (such as zip codes or IP addresses)
Law of large numbers
A target user´s past behavior (likes, rating, purchase history, etc.) is collaborated with the behavior of similar users
this means that the greater the amount of data available for analysis, the better we become at making correct decisions
Within computing, a ______ is a tightly coupled collection of servers, or nodes. These servers usually have the same hardware specifications and are connected together via network to work as a single unit
A _______ provides a logical view of the data stored on the storage medium as a tree structure of files and directories
The ____ does not apply to Big Data
The greater the volume and variety of data that Big Data solutions can process allows for each additional batch of data to carry greater potential of unearthing new patterns and anomalies
for speech recognition, the system attemps to comprehend the speech and then performs an action, such as transcribing text
the system is fed data that is already categorized or labeled, so that it can develop an understanding of the different categories
the system is fed unknown (but similar) data for classification, based on the understanding it developed
a common application of this technique is for the filtering of e-mail spam. Note that ___________ can be performed for two or more categories
in a simplified _____ process, the machine is fed labeled data during training that builds its understanding of the _______. The machine is then fed unlabeled data, which is classifies itself
A file is an atomic unit of storage used by the _________ to stored data. Files are organizated inside of a directory
is an unsupervised learning technique by which data is divided into different groups so that the data in each group has similar properties
There is no prior learning of categories required; intead, categories are implicity generated based on the data groupings
Applications include document classification and search, as well as builiding a 360-degree view of a customer by extracting information from a CRM system
How the data is grouped depends on the type of algorithm used. Each algorithm uses a different technique to identify ______
Is solely dedicated to individual user preferences and does not require data about other users
The machine learning technique is used to identify anomalies, abnormalities and desviation that can be advantageous (such as oportunities) or disadvantageous (such a risk)
involves the simultaneous execution of multiple sub-tasks that collectivelly comprise a larger task
To the client, a file appears local and can be accessed via multiple locations
is the automated process of finding relevant items from a pool of items
A common medium by which ________ is implemented is via the use of a recommender system
A ________ generally provides only one of the listed functions
Colaborative Filtering
Based on the similarity of the user´s behavior, items are filtered for the target user
Content-based Filtering
A user profile is created based on the user´s past behavior (likes, rating, purchase history, etc.)
Analytics Engine
The similarities identified between the user profile and the attributes of various items, lead to items being filtered for the user
A recommender system predicts user preferences and generate suggestions for the user accordingly
A recommender system typically uses either colaborative _____ or content-based _________ to generate suggestions
In order for the machines to extract valuable information, text and speech data needs to be understood by the machines in the same way as humans do. _____ represents practices for extracting meaningful information from textual and speech data
this allows computers to perform a variety of useful task, such as full-text searches
A _______ database generally provides an API-based query interface, rather than the SQL Interface
functionally can be further grouped into the following categories: event, file, relational
Useful insights from text-based data can be gained by helping businesses develop an understanding of the information that is contained within a large body of text
The basic tenet of ___________ is to turn unstructured text into data that can be searched and analyzed
Analysts working with big data solutions are not expected to know how to program processing engines
generally involves two steps: Parsing text within documents to extract, Categorization of documents using these extracted entities and facts
the extracted information can be used to perform a context-specific search on entities, based on the type of relationship that exists between the entities
Similarly, processed data may need to be exported to other systems before it can be used outside of the big data solution
Parsing text within documents to extract:
The ________ mechanisms is responsible for processing data (usually retrieved from storage devices) based on pre-defined logic, in order to produce a result
is a specialized form of text analysis that focuses on determining the bias or emotions of individuals
this form of analysis determines the attitude of the author (of the text) by analyzing the text within the context of the natural language
not only provides information about how individuals feel, but also the intensity of their feeling
Common applications for __________ include early identification of customer satisfaction or dissatisfaction, gauging product sucess or failure and spotting new trends
are usually divided into two types: Batch and Transactional
Quantitative Analysis
Correlation and regression are examples of ________. A/B testing can make use of ____________ techniques for results comparision.
Storage Device
Qualitative Analysis
Data Mining
Descriptive Analytics
A/B testing, heat maps and spatial data analysis are considered forms of ____________
The output of one workflow can become the input of another workflow
Strategic BI and analytics fall in this category, since they are highly read intensive task involving large volumes of data
Diagnostic Analytics
Correlation, regression, time series analysis, network analysis and spatial data analysis are considered forms of _________
The workflow logic processed by a _____________ mechanism can involve the participation of other big data mechanism
Predictive Analysis
Prescriptive Analytics
Supervised Learning
Unsupervised Learning
Clustering, outlier detection, filtering, natural language processing, text analytics and sentiment analysis can utilize ___________
Cluster
These engines may provide the agent-based processing of inflight data, which enables various data cleasing and transformation activities to be performed in realtime
In the diagram, a _____ is used to execute a task based on distributed / parallel data processing frameworks
File System
Operating systems employ ______ for data storage. Each operating system provides support for one or more ________, like NTFS for windows and ext for linux
Distributed File System
Examples include the Google File System and Hadoop ________
NoSQL
However, some _______ databases may also provide a SQL-like query interface
processing engine, storage device, resource manager
Parallel Data Processing
the premise is to reduce the execution time by dividing a single larger task into multiple smaller task
A _________ in Big Data os defined as the amount and nature of data that is processed within a certain amount of time
Although __________ can be archieved through multiple networked machines, it is more typically achieved within the confines of a single machine (multiple processors or cores)
Distributed Data Processing
This allows large amounts of data to be imported or exported within a short period of time
Processing workloads
Batch Workload
Queries can be complex and involve multiple joins
Transactional workload
involves small amounts of data with random read/writes
Although these workloads contain a mix of read/write queries, they are generally more write-intensive than read-intensive
Cloud Computing
is a specialized form of distibuted computing that introduce utilization models for remotely provisioning scalable and measured IT resources
It can also represent hierarchical values by using color-coded nested rectangles
the clustered processing resources required by Big Data solutions can benefit from the highly scalable and elastic IT resources available on cloud-based environments
In short _______ provides the three ingredients required for a big data solutions: input data, computing and storage
It makes sence from enterprises already using cloud computing to reuse the cloud from their Big Data initiatives, because:
IT already possesses the required cloud computing skills
the imput data already exists in the cloud
Big data Mechanisms
represents the primary, common components of big data solutions, regardless of the open source or vendor products used for implementation
Processing Engine
Resource Manager
At minimun, any given big data solution needs to contain the _____, ______ and _______ mechanism in order to effectively process large datasets in support of the big data analysis lifecycle
storage device, analytics engine, coordination engine
processing engine , query engine, data transfer engine
resource manager, analytics engine, workflow engine
___________ mechanisms provide the underlying data storage environment for persisting the datasets that are processed by big data solutions
A _______ can exists as a distibuted file system or a database
Distributed file systems can be used for persisting immutable data that is intended for streaming access or batch processing
Databases, such as NoSQL repositories, can be used for structured and unstructured storage and read/write data access
Any data processing that is requested by the big data solution is fulfilled by the __________
Batch Processing Engine
Provides support for batch data where processing tasks can take anywhere from minutes to hours to complete. This type of processing engine is considered to have high latency
Realtime Processing Engine
Provides support for realtime data with sub-seconds response times. This type of processing engine is considered to have low latency
Data that is held in storage can be processed in a variety of ways by a given Big Data solutions and all data processing requests require the allocation of processing resources
A _______ acts as a schedules and prioritizes processing requests according to individual processing workload requirements
Unlike other data processing systems where input data conforms to a schema and is mostly structured, data sources for a big data solution tend to include a mix of structured and unstructured data
Data Transfer ingress and egress
It is common for multiple diferent ________ to be part a big data solution to facilitate a range of import and export requirements for different types of data
Data Transfer Ingress Engine
Event-based __________ generally use a publish-subcribe model based on the use of a queue to ensure high reliability and availability
enable the substitution of data that is distributed across a range of sources residing in multiple systems outside of the big data solution
The _______ mechanism abstracts the processing engine from end-users by providing a front-end user-interface that can used to query underlying data, along with features for creating query execution plans
Languages that are more familiar and easier to work with (such as SQL) can be used by non-technical users to perform ETL tasks and run ad hoc queries for data analysis activities
Common processing functions performed by a ______ include sum,average, group by join and sort
Under the hood, the ________ seamlessly transforms user queries into the relevant low-level code that can be used by the processing engine
The ________ mechanism is able to process advanced statistical and machine learning algorithms in support of analytics processing requirements, including the identification of patterns and correlations
It generally uses the processing engine mechanism to run algorithms on large datasets.
The defined workflows are analogous to a flowchart with control logic (such as decisions, forks, joins) and generally rely on a batch-style processing engine for execution
