Introduction to Data Mining: Concepts & Techniques
The evolution, importance, and techniques of data mining, and how it can be applied to various industries. Understand the challenges and benefits of data mining.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. Download presentation by click this link. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
Presentation Transcript
Data Mining : Introduction to Concepts and Techniques Dr. V. Shyamala Susan, Head, Asst. Professor, A.P.C. Mahalaxmi College for Women, Thoothukudi -628 002.
Module overview Evolution of Database Technology Why Data Mining Now ? What is Data Mining ? Evolution of Data Mining Why Data Mining is Important ? The Scope of Data Mining Tasks of Data Mining Major Components of KDD Process Data Mining : KDD Process Data Mining on What Kind of Data ? Data Mining as Business Intelligence Data Mining Techniques Data Mining Technologies Data Mining Issues Data Mining Challenges
Evolution of Database Technology Relational database This is a general database suitable for all enterprise applications. In-Memory Database This database is required for high volume applications. Web scale (noSQL) Internet services with billions of users created the first use case for vertical database technology Vertical Databases On-demand applications across multiple industries with specialized requirements
Why Data Mining Now ? The explosive growth of terabyte-to-petabyte data o Automated data collection tools and database technology Business: Web, transactions, e-commerce stocks, o Science: Remote sensing, scientific simulation, bioinformatics o Society and everyone: YouTube, news, digital cameras, o So we Drown in Data but Starve for Knowledge Information poor Data rich Manual Analysis In Spreadsheets Long time to insights In flexible, static reports Data rich by Business, Science, Society and IT
What is Data Mining ? Data Mining : It is the process of identifying anomalies, patterns and similarities within massive data sets. We can increase revenues, split costs, improve customer relationships, minimize risk and more utilising data mining techniques. Other Names of Data Mining : Statisticians - Data Fishing, Data Dredging (1960) AI, Machine Learning Community - Knowledge Discovery in Databases(1989) Business management term - Business Intelligence(1990) And also data archaeology, information discovery, information harvesting,, knowledge extraction, data/pattern analysis, etc.
Evolution of Data Mining Data Mining draws ideas from Statistics /AI , Pattern Recognition etc. Traditional techniques may be unsuitable due to huge, distributed and high dimensionality data Bayer s Theorem (1763) Regression (1805) Turing (1936) Neural Network (1943) Evolutionary Computation (1965) Databases (1970s) Genetic Algorithms (1975) KDD (1989) SVM (1992) Data Science (2001) Moneyball (2003) Big Data Widespread Adoption DJ patil (2015)
Why Data Mining is Important ? Data mining helps analysts understand the key facts, relationships, trends, patterns and anomalies that can sometimes go inaccurate Application of Data mining in different areas: Tele Communication Industry Retail & Marketing Applications Pharmaceutical Firms Medical Companies Credit Card Companies Insurance Companies
The Scope of Data Mining Makes it easier to identify important information regarding business in a huge database. By having certain skills, they generate new market opportunities: Automated prediction of trends and behaviors The information is projected automatically in large databases. Questions that historically involved extensive analysis can now be addressed directly from the data easily. Predictive major problems include predicting bankruptcy and other forms of default, and classifying segments of a population likely to return equally to events Automated discovery of previously unknown patterns - Data mining instruments identify previously hidden patterns in one step Data analyzes on retail sales are designed to identify seemingly unrelated products that are often purchased together. It also idealizes fraudulent transactions with credit cards and classifies anomalous data that may represent errors in keying data entry.
Tasks of Data Mining Data mining deals with the types of patterns that may be exploited. Association (discovers relationships between attributes) Descriptive (Ccharacterize the general properties of the data in the database) Clustering (Group similar items together into some clusters) Summarization (Maps data into subsets with associated simple descriptions) Data Mining tasks Classification (Assign data into predefined classes) Predictive (Perform inference on the current data in order to make predictions) Prediction (Predict a real value for a given data instance ) Regression (Maps data item to a real valued prediction variable)
Major Components of KDD Process Data mining systems' significant components are a data source, data mining engine, data warehouse server, the pattern evaluation module, graphical user interface, and knowledge base. The Graphical User Interface (GUI) module needs to communicate between the data mining system and the user. Graphical User Interface Pattern evaluation module is responsible for searching patterns Pattern Evaluation Data mining engine contains several modules for the operation of data mining tasks such as classification of associations, clustering, pr ediction, analysis of time series, etc Data Mining Engine Knowledge Base The database or data warehouse server is composed of the original data ready for processing. Database Or Data Warehouse Server Data Cleaning & Data Integration Knowledge base is useful for sharing knowledge and managing user interface Filtering Data Warehouses Databases
Data Mining : KDD Process Steps in Data Mining or Knowledge Discovery in Databases ( KDD) Knowledge Interpretation / Evaluation Data visualization and result interpretation Data mining Patterns Model with algorithms such as clustering, regression and classification Transformed data Dimension reduction. Factor analysis Transformation Pre - Processing Processed data Difference, or taking logarithm to be normalization Selection Target data Data collection and sampling. Correlation analysis
Data Mining : KDD Process Data cleaning and preprocessing: (may take 60 percent of the effort!) Removes noise and data Data integration Combines multiple sources of data Data selection Relevant information is collected from the database Data reduction and transformation: The Data is transformed into suitable forms for mining by selecting functions of data mining (summarization, regression, classification, association, clustering for instance Data mining: Search for patterns of interest Pattern evaluation and knowledge presentation visualization, transformation, removing redundant patterns, etc. Use of discovered knowledge
Data Mining on What Kind of Data ? Data Mining techniques can be applied to any kind of data repositories such as Relational Databases Data Warehouse Transactional Databases Advanced Systems Flat Files World Wide Web. Advanced database system object oriented and object relation databases specific application oriented databases such as spatial database, time series databases, text databases and multimedia databases.
Data Mining on What Kind of Data ? Relational Databases - A collection of tables that consists of a set of attributes Data warehouse - A repository that collects information from multiple sources and stored under a unified scheme residing on a single site. Transactional Databases- A collection of tables where each record represents a transaction. Advanced Databases Object oriented Databases - Databases constructed based on object - oriented programming concept. Object Relational Databases - Databases constructed based on an object relational data model. Spatial Databases Databases that stores a large amount of space - related data, such as maps, pre- processed remote sensing or medical imaging data and VLSI chip layout data. Temporal Databases and Time - Series Databases - Databases store both time and relation data. A temporal database usually stores relational data that include time - related attributes and a time series database stores sequence of values or events changing with time.
Data Mining on What Kind of Data ? Databases for Text and Multimedia Text databases store descriptions of words for objects such as long sentences or paragraphs, warnings, summary reports etc. A Multimedia database system stores a huge collection of multimedia content including data from audio, image , video, sequence, and hypertext. Heterogeneous Databases and Legacy Databases A legacy database is a group of heterogeneous databases which combine various types of data systems. The Internet Site The world wide web is today a popular and interactive medium for distributing information.
Data Mining as Business Intelligence Data Mining method identifies important patterns and knowledge from a large amount of information. In these steps, the data patterns are extracted using smart patterns. The data is depicted in the form of patterns and models are organized using clustering and classification techniques Project objectives Problem definition Data collection Insights Dataset construction Purpose of model Steps review Business issues Modeling techniques
Data Mining Techniques Data mining methods were introduced and used in huge data sets to find previously unknown, valid relationships and patterns.
Data Mining Techniques 1. Anomaly or Outlier Detection: - It detects the flaws of the data 2. Association Rule Learning: It finds the relations or interesting patterns from the large data set. 3. Clustering Analysis: It identifying similar data from the large data-set and grouping them 4. Classification Analysis: It obtains information about the data. 5. Regression Analysis: It helps to identify the relationship between the different variables. 6. Data Warehousing: It is the technique of storing large quantities of structured data securely 7. Visualization: Itis the process of tabulating data in the form of graphs, charts and diagrams and digital images. 8. Statistical Techniques: It calculates the mean, mode, and median of the data to predict future patterns. 9. Tracking Patterns: It identifies patterns from the current database. 10. Sequential Patterns: It identifies the sequence of the data.
Data Mining Technologies Data mining has integrated numerous techniques from other fields such as statistics, machine learning , pattern recognition, data system and data warehouse systems , information retrieval, visual analytics, algorithms, high-performance computing and many specific applications Machine Learning Visualization Statistics Data Mining Technologies Pattern recognition Information retrieval Data warehouse
Data Mining Issues Major issues in data mining are mining methodology, user interaction, performance, and diverse data types. Mining Methodology and User Interaction Issues Mining various methods and techniques in databases Interactive knowledge mining at numerous abstractional levels Adding background information Languages used in data mining and ad hoc data mining Presentation and visualisation of the results of data mining Handling incomplete or noisy data Evaluating patterns Performance Issues Data mining algorithms are efficient and scalable. Parallel and distributed mining algorithms, and incremental Issues Relating to the Diversity of Database Types Handling of the related and complex data types Mining heterogeneous databases and global information systems
Data Mining Challenges Data mining is hindered by the increasing quantity and complexity of big data, security, scalability, data quality and streaming data Big Data Data Mining Challeng es Privacy & Security Overfitting Cost of Scale
Summary Evolution of Database Technology Why Data Mining is important nowadays? Data Mining discovers interesting patterns from the data Evolution of Data Mining The important of data mining in different field The Scope of Data Mining The different tasks of Data Mining The significant components of data mining systems are a data source, data mining engine, data warehouse server, the pattern evaluation module, graphical user interface, and knowledge base. KDD Process to obtain knowledge from data What Kind of Data can be mined Data Mining as a Business Intelligence, process identifies interesting patterns and knowledge from a large amount of data. In these steps, intelligent patterns are applied to extract the data patterns. The data is represented in the form of patterns and models are structured using classification and clustering techniques Data mining techniques have been developed and used to find previously unknown, valid patterns and relationships in huge data sets. Data mining has incorporated many techniques from other domains such as statistics, machine learning, pattern recognition, database and data warehouse systems, information retrieval, visualization, algorithms, high performance computing, and many application domains Major issues in data mining are mining methodology, user interaction, performance, and diverse data types. Data mining is hindered by the increasing quantity and complexity of big data, security, scalability, data quality and streaming data
