P1-Modulo 2 : Big data analysis y technology concepts

1

Big data analysis differs from trditional data analysis primary because

1

In big data analysis and analytics, a fundamental step-by-step process is needed to organize the task involved

1

which are 1,2 and 3 stages of bigData analysis lifecycle

1

which are 4,5 and 6 stages of bigData analysis lifecycle

1

which are 7,8 and 9 stages of bigData analysis lifecycle

1

The business case evaluation stage requires that a business case be , and ________ prior to proceeding with the actual hands-on analysis tasks.

1

An evaluation of a Big Data analysis bussines case helps decision-makers undertand

1

The KPIs is ussefull in Business Case Evaluation

1

based on the business requirements documented in the _______________________________ , it can be determined whether the business problems being addresed are really Big data problems

1

a bussines problem needs to be directly related to one or more of the big data characteristics

1

Another outcome in Business case evaluation is

1

The invesment can be weighed against the expected benefits of achieving the goals

1

Initial iterations of the big data analysis lifecycle will not required more up-front invesment of Big Data tecnologies, products and training compared to later iterations

1

The data identification stage is dedicated to identifiying the _____________ required for the analysis project

1

identifying a wider variety of data sources may increase the probability of finding

1

Can be beneficial to identify as many types of releated data sources and insights as possible, especilly when we don´t know exactly what we're looking for.

1

Depending on the business scope of analysis project and nature of business problems being adressed, the required dataset and their sourcescan be

1

Internal dataset

1

external datasets

1

data is gathered from all of data sources that were identified during the previous stage, and is then subjected to the automated filtering of corrupt data or data that has been deemed to have no value to the analysis obvjectives

1

Depending on the type of data source , data may come as a dump of files or may require API integration

1

in many cases , especially where external, unstructured data is concerned, some or most of the acquired data may be irrelevant (noise) and can be discarded as part of the filtering process

1

data classified as "corrupt" can include records with missing or nonsensical values or invalid data type

1

Data thah is filtered out for one analysis may not be valueable for a different type of analysis

1

it is advisable to store a verbatim copy of the original dataset proceeding with the filtering.

1

to be persisted once it gets generated or enters the enterprise boundary

1

The data is persisted to disk prior to analysis

1

The data is analyzed first and then persisted to disk

1

Can be added via automation to data from both internal and external data sources to improve the classification an querying

1

Metadata example can include

1

it is vital that metadata be machine-readable and passed forward along subsequent analysis stages

1

This helps to maintain data provenance throughout the Big Data analysis lifecycle, wich helps establish and preserve data accuracy and quality

1

Some of the data identified as input for the analysis may arrive in a format incompatible with the big data solution

1

the need to address disparate types of data is more likely with data from

1

is dedicated to extracting disparate data and transforming it into a format that the underliying Big Data solution can use for the purpose of the data analysis

1

The extent of extraction and transformation required depends

1

Estracting the required fields from delimited textual data (such as with web server log files) may not be necessary

1

example of document that not need further transformation

1

The invalid data can

1

data input into Big Data analyses can be unstructured without any indication of validity

1

the complexity can further make it easy to arrive at a set of suitable validation constraint

1

Is dedicated to establishing (often complex) validation rules and removing any know invalid data

1

Big Data solutions often receive redundant data across different datasets, this redundancy can be exploited to explore interconnected datasets in order to assemble validation parameters and fill in missing valid data

1

For Batch analytics, data validation and cleansing can be achieved via offline

1

Data input in Big Data can be unstructured without any indication of validity

1

provenance can play an important role in determining the accuracy and quality of questionable data

1

data that appears to be invalid may still be valuable in that it may posses

1

Data may be spread across multiple datasets, requiring that datasets be joined together via common files (date or ID)

1

either way a method of data ________________ is required or the dataset representing ther correct value needs to be determined

1

Dedicated to integrating multiple datasets together to arrive at a unified view

1

Can become complicated because od differences in : although the data format may be the same, the data model may be different

1

Can become complicated because od differences in : A valuethat is labelled differently in two different datasets may mean the same thing (surname and last name)

1

In data Aggregation & Representation reconciling the differences can required complex logic that is executed ___________________.

1

Future data analysis requirements need to be considered during the stage ___________________ to help foster data reusability

1

whether ___________________ is required or not, it is important to understand that the same data can be stored in many different forms. One form may be better suited for a particular type of analysis than another

1

A data structured standarized by the Big Data solution can require establishing a central, standard analysis repository, such as a

1

the data analysis stage is dedicated to carriying out the actual analysis task, which typically involves one or more types of analytics

1

This stage can be iterative in nature, because repeated until appropiated pattern or correlation is uncovered

1

The approach taken when carrying out this stage, data analysis, an be classified as ______________________________

1

___________________________ adata analysis is a deductive approach where the cause of the phenomenon being investigated is proposed beforehand

1

the proposed cause or assumption is called a

1

data samples are tipically used

1

unexpected findings or anomalies are usually ignored since a predetermined cause was assumed

1

is an inductive approach that is closely associated to data mining

1

this analysis provides a general direction that can facilitate the discovery of patterns or annomalies

1

Large amounts of data and visual analysis are typically used

1

is dedicated to using _____________________ techniques and tools to graphically communicate the analysis results for effective interpretation by bussines users

1

Bussines users needs to be able to understand the results in order to obtain value from analysis and subsequently have de ability to provide feedback from_ back to stage ____

1

the same results may be presented ina a number a number of different ways.

1

another aspect to keep in mind is that providing a method of drilling down to comparatively simple statistics is crucial, in order for users to understand how to statistics were generated

1

support businessdecission-making, there may be further opportunieties to utilize the analysis results

1

The utilization os analysis results is dedicated to determining how and where processed analysis data can be further leveraged

1

"models" that encapsulated new insights and understandings about the nature of the patterns and realationships that exist within data that was just analyzed

1

A "model" may look like a

1

Models can be used to improved bussines process logic

1

the data analysis results may be automatically or manually fed directly into enterprise systems to enhace and optimize their behavior and performance

1

The identiffied patterns correlations and anomalies discovered during the data analysis are used to refine business process

1

Data analysis results can be used as input for existing events that requires them to take corrective action

1

Big data nalysis concepts

1

statistical analysis

1

visual Analysis

1

machine learning

1

semantic analysis

1

use statistical methods based on mathematical formulas as means for analizing data

1

it can also be used to infer patterns ans relationships within the dataset, such as regression and correlation

1

also know as split or bucket testing, compares two versions of an element to determine wich version is superior based on a predefined metric

1

A/B testing: the current version of the element is called the __ version, whereas the modified version is called the

1

both version, are subjected to an experiment simultaneously. The observationsare recorded to determine wich version is more sccessful

1

Athough ________________________can be implemented in almost domain, it is most often used in marketing

1

Generally, the objective is to gauge human behavior with the goal of increasing sales (as per the example below)

1

is the new version of a drug better than the old one?

1

is an analysis tecnique used to determine whether two variables are related to each other

1

an example of a relationship between two variables:
The value of variable A increases whenever the value of variable B increases

1

Helps to develop an understanding of a dataset and find relationships that can assist in explaining a phenomenon

1

commonly used for data mining where the identification between variables in a dataset leads to the discovery of patterns ans anomalies

1

When two variables are considered to be correlated they are considered to be aligned based on a linear relationship

1

This mean that when one variable changes, the other variable also changes proportionally and constantly

1

______________________ is expresed a a decimal number between -1 to 1, which is know as the correlation coeficient

1

Correlation +1

1

0 Correlation

1

-1 Correlation

1

sample: "Do students who perform well at elementary school perform equally well at high school"

1

explores how a dependent variable is related to an independent variable within a dataset

1

Helpss determine how the value od dependent variable changes in relation to changes in the value of the independent varible

1

what the analysts discover is that 15% of additional stock in required for enery 5-degree increase in temperature

1

more than one independent variable can be tested at the same time

1

in such cases only one independent variable may change. The others are kept constants

1

can help enable a better understanding of what a phenomenin is and why it ocurred

1

represents a constant rate of change

1

Represents the variable rate of change

1

what will be the grades of a student studying at a high school based on her primary school grades

1

_________________does not imply a causation. The change in the value of one variable may not be responsible for the change in the value of the second variable. although both may change at the same rate

1

assumes that both variables are independent

1

Deals with already identified dependent and independent variables

1

_________________ can be applied to further explore the relationship and predict the values of the dependent variable, based on the know values of the independent variable

1

is a form of data analysis that involves the graphic representation of data to enable or enhace its visual perception

1

develop a deeper understanding of the data being analyzed. Specifically, it helps identify and highlight hidden patterns, correlations and anomalies.

1

visual analysis

1

Are an effective visual analysis technique for expressing patterns, data compositions via part-whole relations and geographic distribution of data

1

They also facilitate the identification of areas of interest ans the discovery of extreme (high/low) values wihin a dataset

1

___________ itself is a visual, color-coded representation of data values

1

A _______________ can be in the form of a chart or a map, as shown in the following pages

1

A___________ represents a matrix of values in which each cell is color-coded according to the value

1

A ___________ represents a geographic measure by wich different regions are color-code according to certain theme

1

How can i visually identify any patterns related to carbon emission across a large number of cities around the world

1

____________is the analysis of data that is recorded over periodic intervals of time

1

Helps to uncover patterns within data that are time-dependent. Once identified, the patterns can be axtrapollated for future predictions

1

time series analyses are usually used for forecasting by identifiying long-term trends. seasonal periodic patterns and irregular short-term variations in the dataset

1

always includes time as a comparision variable

1

is generally expressed using a line chart, with time plotted on the x-axis and the recorded data values plotted on the y-axis

1

how much yield should the farmer expect based on historical yield data

1

is an interconected collection of entities

1

An entity can be a person a group or some other business domain object such as a product

1

some conectios may only be one-way, so that transversal in the reverse direction is nor possible

1

is a techniquethat focuses on analizing relationships between entities within the network

1

There are specialized variations of network analysis

1

is used to find the shortest routes between the central warehouse and remote stores in order to minimize the durations of deliveries

1

How can identify interaction patterns among a very large number of protein-to-protein interactiona?

1

is focused on analizing location-based data in order to find different geographic relationships and patterns between entities

1

____________________________ is manipulated through a geographical information system (Gis) that plots spatial data on a map generally using its longitude and latitude coordinates

1

no two stores can be within a distance of 5 kilometers of each other to prevent the stores from competing with each other.

1

how far do customers have to commute in order to get to a supermartket?

1

if the human knowledge can be combined with the processing speed of machines, machines will be able to process large amounts of data without requiring much human intervention

1

machine learning

1

Two fundamental laws that pertain to machine learning

1

the law _____________________________states that the confidence with wich predictions can be made increases as the size of data that is being analyzed increases

1

in other words the accuracy and applicability of the patterns and relationshipsthat are found in a large dataset will be higher that of a smaller dataset

1

the greater the amount of data available for analysis, the better we become of making correct decisions

1

in the context of traditional data analysis, ___________________________ states that, starting with a reasonably large sample size, the value obtained from the analysis of additional data decreases as more data is successively added to the original sample

1

The law of dimishing marginal utility does not apply to big data

1

The greater the volume and variety of data that Big Data solutions can process allows for each additional batch of data to carry greater potential of unearthing new patterns and anomalies. Therefore, the value of each additional batch does not diminish value: rather, it provides more value

1

is a supervised learning technique by witch data is classified into relevant, previously learned categories

1

Step 1: The system is fed data that is already categorized or labeled, so that it can develop an understanding of different categories

1

step 2: The system is fed unknow (but similar) data for classification, based on the understanding it developed

1

A common application of this techniques is for the filtering of e-mail spam. Note that classification can be performed for two or more categories

1

Based on old data, a training dataset is compiled that contains tagged examples of customers that have or not previously defaulted

1

Does a fingerprint belong to a suspect based on a record of this previous fingerprints

1

Is an unsupervised learning technique by wich data is divided into different groups so that the data in each group has similar properties

1

There is no prior learning of categories required: instead categories are implicity generated based on the data groupings

1

Is generally used in data minig to get an understanding of properties of a given dataset. Afterdeveloping this understanding, classificatioin can be used to make better predictions about similar, but new or unseen data

1

In a bank each group is the introduced to one or more financial products most suitable to the characteristics of the overall profile of the group

1

How many different categories of elements are there in the periodic table

1

Detection is the process of finding data that is significantly different from or inconsistent with the rest of the data within a given dataset

1

this machine learning tecnique is used to identify anomalies, abnormalities and deviations that can be opportunities or risks

1

it can be bsaed on either supervised or unsupervised learning

1

include fraud detection, medical diagnosis, network data analysis and sensor data analysis

1

In order ti find if a transaction is likely to be fraudulent or not, the bank´s IT team builds a sustem emplying ____________________ technique that is based on supervised learning

1

are there any wrongly identified fruits and vegetables in the training dataset used for classification task

1

is the automated process of finding relevant items from a pool of items

1

items can be filtered either based on a users own behavior or by matching the behavior of multiple users

1

_________________ is generally applied viat the following two approaches

1

items can be filtered either based on a users own behavior or by matching the behavior of multiple users

1

A common medium by wich ________________is implemented is via the use of a recomender system

1

technique based on the collaboration of users past behavior

1

based on the similarityof users behavior, items are filtered for the target user

1

is solely based on the similarity between users behavior, and requires a large amount of user behavior data in order to accurately

1

collaborative filtering is an example of application of law of large numbers

1

technique focused on the similarity between users an items

1

A user profile is created based on the users past behavior (likes, ratings, purchase history, etc)

1

Contrary to collaborative filtering, content-based filtering is solely dedicated to individual user preferences and does not require data about other users

1

A recomender system predicts user preferences and generate suggestions for the user accordingly

1

suggestions commonly pertain to recomending items, such as movies, books, web pages, people etc

1

A recomender system typically uses either collaborative filtering or content-based filtering to generate suggestions

1

recommender system may also be based on a hybrid of both collaborative filtering and content-based filtering to fine-tune the accuracy and effectiveness of generated suggestions

1

Based on matches found between financial product purchased by customers and the properties of similar financial products, the recommnder system automates seggestion for potential financial products that customers may also be interested in

1

Wich holiday destinations can be recommended based on the travel history of a holiday makes?

1

A fragment of text or speech data can carry different meanings in different contexts, whereas a complete sentence may retain its meaning, even if structured in different ways. In order for the machines to extract valuable information, text and speech data needs to be understood by the machines in the same way as humans do. Semantic analysis represents practices for extracting meaningful information from textual and speech data

1

types of semantic analysis

1

Is a computers ability to comprehend human speech and text as naturally understood by humans

1

This allows computers to perfom a variety of useful task, such as full-text searches

1

instead of hard-coding the required learning rules, either supervised or unsupervised machine learning is applied to develop the computer undestanding of the natural language

1

in general the more learning data the computer has, the more correctly it can decipher human text and speech

1

Natural language processing includes both text and speech recognition

1

For speech recognition the system attempts to comprehend the speech and then performs an action, such as transcribing text

1

How can grammatical mistakes be automaticalle identified?

1

Unstructured text is generally much more difficult to analyze and search, compared to structured text

1

is the specialized analysis of text through the application of data mining, machine learning and natural language processing techniques to extract value out of unstructured text. Text analytics essentially provides the ability to discover text rather than just search it

1

useful insights from text-based data can be gained by helping business develop an understanding of the information that is contained within a large body of text

1

the basic tenet of text analytics is to turn unstructured text into data that can be searched and analyzed

1

As the amount of digitized documents, e-mail, social media posts and log files increases, businesses have an increasing need to leverage any value that can be extracted from these forms of semi-structured and unstructured data

modulo 2 Big Data