DATA PREPROCESSING

1. Missing Values

   Annotations:

   • Use the attributes mean to fill in the missing value
   • Use the attribute mean for all sample belonging to the same class as the given tupe
   • Use the most probable values to fill in the missing values
   1. Data Cleaning

      Annotations:

      • Fill in missing values & correct inconsistencies in the data
   2. Ignore the tuple

      Annotations:

      • Class label is missing & not effective
   3. Fill in the missing value manuvally

      Annotations:

      • time consuming & may not be feasible
   4. Use global constant to fill in the missing value

      Annotations:

      • replace values & same constant
2. Noisy Data
   1. Binning

      Annotations:

      • It smooth sorted data value by consulting its neighborhood
      • smoothing by bin means smoothing by bin medians smoothing by bin boundaries
   2. Clustering

      Annotations:

      • Similar values organized into groups & outliers may be detected by clustering
   3. Combained computer & human Inspection

      Annotations:

      • Outlier may be identified through a combination of computer and human inspection
   4. Regression

      Annotations:

      • Data can be smoothed by fitting the data to function
      • Linear regression
      • Multiple linear regression
3. Inconsistent Data

   Annotations:

   • Data inconsistencies may be corrected manually using external reference
   • Knowledge engineering tools may also be used to detect the violation of  known data constrains

Annotations:

• It can be applied to obtain a reduced representation of the data set yet closely maintains the integrity of the original data

1. Dimensionality Reduction
2. Data Compression
   1. wavelet transforms
   2. Principal components analysis
3. Numerosity Reduction
4. Data cube Aggregation
5. Strategies
   1. Data cube aggregation
   2. Dimension reduction

      Annotations:

      • step wise forward selection step wise backward elimination combination of  forward selection & backward elimination decision tree induction
   3. Numerosity reduction
      1. Histograms
      2. Clustering
      3. Sampling

         Annotations:

         • SRSWOR of size n SRSWR of size n Cluster sample Stratisfied sample
   4. Discretization & hierarchy Generation
      1. For numeric data
         1. Bining
         2. Histogram & Analysis
         3. Cluster Analysis
         4. Entropy based Discretization
         5. Segmantation by Natural Partitioning
      2. For categorical data
         1. Portion of a hierarchy by explicit data grouping
         2. Partial ordering of attributes explicity at the schema level
         3. Set of attributes,but not their partial orderies

1. For Categorical Data
2. For Numeric Data

1. Data Integration

   Annotations:

   • It can help improve accuracy & speed of the subsequent mining process
   • Reduce and avoid Redundancies & inconsistencies
   • Detection and resolution of data value conflicts
2. Data Transformation
   1. Smoothing

      Annotations:

      • Remove the noise from data
   2. Attribute construction

      Annotations:

      • new attributes are constructed and added to help the mining process
   3. Aggregation

      Annotations:

      • Aggregation operation are applied to the data
   4. Generlization

      Annotations:

      • Primitive data are replaced by high-level concept through the use of concept hierarchies
   5. Normalization

      Annotations:

      • Attribute data are scaled with in small specified range such as  -1.0 to 1.0 or 0.0 to 1.0

1. Ease of Mining Process
2. To Improve the Quality of Data