Data Preprocessing

Nota:

• Reduce representation of data set.
• Dimension Reduction Numerosity Reduction

Nota:

• Normalization,data discretization and concept hierarchy generation are forms of Data Transformation.

Nota:

• Integrating multiple databases,data cubes or files.

Nota:

• There are many possible reasons for inaccurate data. For example,the age of a person must be enter within 100.

Nota:

• Believability reflects how much the data are trusted by users.

Nota:

• Interpretability reflects how easy the data are understood to users.

Nota:

• To Clean the data by filling in missing values,smoothing noisy data,identifying or removing outliers, and resolving inconsistencies.
• 1. Missing Values. 2. Noisy Data. 3. Data Cleaning as a Process.

1. Missing Values

   Nota:

   • 1. Ignore the tuple.
   • 2. Fill in the missing value manually.
   • 3. Use a global constant to fill in the missing value.
   • 5. Use the attribute mean or median for all samples belonging to the same class as the given tuple.
   • 6. Use the most probable value to fill in the missing value.
2. Noisy Data

   Nota:

   • 1. Binning  -> Smoothing by bin means.-> Smoothing by bin medians.-> Smoothing by bin boundaries.
   • 2. Regression -> Linear Regression
3. Data Cleaning as a Process

   Nota:

   • Rules Used: -> Unique Rule. -> Consecutive Rule. -> Null Rule.
   • Tools Used: -> Data Subscribing Tools. -> Data Auditing Tools. -> Data Migration Tools.

Nota:

• To integrate multiple databases,data cubes, or files.

1. Redundancy and Correlation Analysis

   Nota:

   • Redundancy can be detected by Correlation Analysis.
   • 1. Correlation Test for Nominal Data. 2. Correlation Coefficient for Numeric Data. 3. Covariance of Numeric Data.
2. Entity Identification Problem
3. Tuple Duplication

   Nota:

   • Duplication should also be detected at tuple level.
4. Data Value Conflict Detection and Resolution

   Nota:

   • Data Integration also involves the detection and resolution of  Data Value Conflicts.

Nota:

• Normalization, data discretization, and concept hierarchy generation are forms of Data Transformation.

1. Data Transformation Strategies

   Nota:

   • 1. Smoothing. 2. Attribute Construction. 3. Aggregation. 4. Normalization. 5. Discretization. 6. Concept Hierarchy Generation for nominal data.
2. Data Transformation by Normalization

   Nota:

   • -> Min-Max Normalization. -> Z- Score Normalization. -> Decimal Scaling.
3. Discretization by Binning

   Nota:

   • Binning is a top-down splitting technique based on a specified number of bins.
4. Discretization by Histograms

   Nota:

   • Histogram analysis is an unsupervised discretization technique because it does not use class information.
5. Discretization by Cluster,Decision Tree and Correlation Analyses
6. Concept Hierarchy Generation for Nominal Data

   Nota:

   • 1. Specification of a partial ordering of attributes explicitly at the schema level by users or experts.
   • 2. Specification of a portion of a hierarchy by explicit data grouping.
   • 3. Specification of set of attributes,but not of their partial ordering.
   • 4. Specification of only a partial set of attributes.

Nota:

• To obtain the reduced representation of the data set.

1. Data Reduction Strategies

   Nota:

   • Dimensionality Reduction. Numerosity Reduction.
   • Data Compression. -> Lossless. -> Lossy.
2. Wavelet Transforms

   Nota:

   • Discrete Wavelet Transform.
3. Principal Components Analysis

   Nota:

   • Also called Karhunen - Loeve method.
4. Attribute Subset Selection

   Nota:

   • Reduces the data set size by removing irrelevant or redundant attributes.
   • -> Step Forward Selection. -> Step Backward Elimination. -> Combination of both. -> Decision Tree Induction.
5. Regression and Log - Linear Models

   Nota:

   • Linear Regression. Multiple Linear Regression. Log-Linear models.
6. Histograms

   Nota:

   • Use binning to approximate data distributions and are a popular form of data reduction.
   • -> Equal Width. -> Equal Frequency.
7. Clustering

   Nota:

   • Clustering consider data tuples as objects. Centroid distance is an alternative measure of cluster quality.
8. Sampling

   Nota:

   • Sampling can be used as a data reduction technique because it allows a large data set to be represented by a much smaller random data sample.