Comprehensive Guide to Data Cleaning and Preprocessing Techniques

Understanding the crucial concepts of data cleaning such as Garbage In, Garbage Out principle (GIGO), Non-Linear and Geographic data inspection, handling NaN values, feature scaling, PCA, correlations, and more. Explore the steps involved in cleaning and preprocessing data for data science and machine learning projects.

• Data cleaning
• Preprocessing
• GIGO principle
• Feature scaling
• PCA

paris Follow

Uploaded on Jul 22, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

Presentation Transcript

1. Data Data Cleaning by Gio 0

2. Basic Principle: Garbage in Garbage Out Garbage in Garbage out (GIGO) is the prevailing principle that flawed components of a data set can invalidates the practical use that data set in data science or machine learning Data Cleaning is the act of removing all flawed or irrelevant parts of data so that what remains is more suited to a particular goal; typically, data science or machine learning 1

3. NYC Taxi Data Set Data dictionary CSV and Data Frames Resources & Links 2

4. Preliminary Inspection Statistical breakdown NaN counts Visual inspection Notice: Non-Linear & Geographic columns 3

5. Non-Linear Column Value Engineering Categorical columns Binary and One-Hot Encoding Timestamping dates 4

6. Geographic Value Engineering Geo-Encoding API and GeoPandas Shape files European Petroleum Survey Group(EPSG) Frequency measurements 5

7. Middle Data Inspection Statistical breakdown NaN counts Random visual inspection Notice: NaN & total number of columns 6

8. Replace Not-a-Number(NaN) Values Approaches: drop rows, statistical replacement, etc. NaN distribution and random value generation 7

9. Feature Scaling, PCA & Correlations Feature Scaling is the process of normalizing a range of a variable to add context to the values within the data Principal Component Analysis is the process of utilizing the principal components of a data set to reduce the dimensionality(# of columns) of that data set Correlations are the statistical relationships between variables that can imply dependencies between those variables 8

10. Feature Scaling, PCA & Correlations Rescaling(Min-Max Normalization) Standardization Correlation Matrices, Tables and Lists 9

11. Final Data Inspection Compare with original data set Statistical breakdown Random visual inspection 10

12. Thank you! 11