Ce diaporama a bien été signalé.
Le téléchargement de votre SlideShare est en cours. ×

Lect 1 introduction

Chargement dans…3

Consultez-les par la suite

1 sur 39 Publicité

Plus De Contenu Connexe

Diaporamas pour vous (20)

Similaire à Lect 1 introduction (20)


Plus récents (20)


Lect 1 introduction

  1. 1. Dr. Hrudaya Kumar Tripathy Introduction
  2. 2.  Definition, motivation & application  Branches of data mining  Classification, clustering,Association rule mining  Some classification techniques
  3. 3. § There has been enormous d a t a g r o w t h i n b o t h commercial and scientific databases due to advances in data generation and collection technologies § New mantra § Gather whatever data you can whenever and wherever possible. § Expectations § Gathered data will have value either for the purpose collected or for a purpose not envisioned. Computational Simulations Social Networking: Twitter Sensor Networks Traffic Patterns Cyber Security E-Commerce
  4. 4.  Before 1600, empirical science  1600-1950s, theoretical science ◦ Each discipline has grown a theoretical component.Theoretical models often motivate experiments and generalize our understanding.  1950s-1990s, computational science ◦ Over the last 50 years, most disciplines have grown a third, computational branch (e.g. empirical, theoretical, and computational ecology, or physics, or linguistics.) ◦ Computational Science traditionally meant simulation. It grew out of our inability to find closed-form solutions for complex mathematical models.  1990-now, data science ◦ The flood of data from new scientific instruments and simulations ◦ The ability to economically store and manage petabytes of data online ◦ The Internet and computing Grid that makes all these archives universally accessible ◦ Scientific info. management, acquisition, organization, query, and visualization tasks scale almost linearly with data volumes. Data mining is a major new challenge!  Jim Gray and Alex Szalay, TheWorldWideTelescope:An Archetype for Online Science, Comm.ACM, 45(11): 50-54, Nov. 2002
  5. 5. • 1960s: Data collection, database creation, IMS and network DBMS • 1970s: Relational data model, relational DBMS implementation • 1980s: RDBMS, advanced data models (extended-relational, OO, deductive, etc.), Application-oriented DBMS (spatial, scientific, engineering, etc.) • 1990s: Data mining, data warehousing, multimedia databases, and Web databases • 2000s • Stream data management and mining • Data mining and its applications • Web technology (XML, data integration) and global information systems
  6. 6.  Lots of data is being collected and warehoused ◦ Web data  Yahoo has Peta Bytes of web data  Facebook has billions of active users ◦ e-commerce • Amazon handles millions of visits/day ◦ purchases at department/ grocery stores ◦ Bank/Credit Card transactions  Computers have become cheaper and more powerful  Competitive Pressure is Strong ◦ Provide better, customized services for an edge (e.g. in Customer Relationship Management)
  7. 7. Sky Survey Data  Data collected and stored at enormous speeds (GB/hour) ◦ remote sensors on a satellite  NASA EOSDIS archives over petabytes of earth science data / year ◦ telescopes scanning the skies  Sky survey data ◦ High-throughput biological data ◦ scientific simulations  terabytes of data generated in a few hours  Data mining helps scientists ◦ in automated analysis of massive datasets ◦ In hypothesis formation SurfaceTemperature of Earth Gene Expression Data
  8. 8. Improving health care and reducing costs Finding alternative/ green energy sources Predicting the impact of climate change Reducing hunger and poverty by increasing agriculture production
  9. 9.  Data explosion problem ◦ Automated data collection tools and mature database technology lead to tremendous amounts of data stored in databases, data warehouses and other information repositories  We are drowning in data, but starving for knowledge!  Solution: Data warehousing and data mining ◦ Data warehousing and on-line analytical processing ◦ Extraction of interesting knowledge (rules, regularities, patterns, constraints) from data in large databases
  10. 10.  Data mining (knowledge discovery in databases): ◦ Extraction of interesting (non-trivial, implicit, previously unknown and potentially useful) information or patterns from data in large databases.  Alternative names and their “inside stories”: ◦ Data mining: a misnomer? ◦ Knowledge discovery(mining) in databases (KDD), knowledge extraction, data/pattern analysis, data archeology, business intelligence, etc.
  11. 11.  Many Definitions ◦ Exploration & analysis, by automatic or semi-automatic means, of large quantities of data in order to discover meaningful patterns
  12. 12.  Finding hidden information in a database  Fit data to a model  Similar terms ◦ Exploratory data analysis ◦ Data driven discovery ◦ Deductive learning
  13. 13. l What is not Data Mining? – Look up phone number in phone directory – Q u e r y a W e b search engine for information about “Amazon” l What is Data Mining? – Certain names are more prevalent in c e r t a in U S locations (O’Brien, O’Rurke, O’Reilly… in Boston area) – G ro u p t o g e t h e r s i m i l a r documents returned by search engine according to their c o n t e x t ( e . g . A m a z o n rainforest,Amazon.com,)
  14. 14.  Draws ideas from machine learning/AI, pattern recognition, statistics, and database systems  Traditional techniques may be unsuitable due to data that is ◦ Large-scale ◦ High dimensional ◦ Heterogeneous ◦ Complex ◦ Distributed  A key component of the emerging field of data science and data-driven discovery
  15. 15. Data Mining Statistics OtherDiscipline Visualization Algorithm Pattern Recognition Machine Learning Database Technology
  16. 16.  Tremendous amount of data ◦ Algorithms must be highly scalable to handle such as tera-bytes of data  High-dimensionality of data ◦ Micro-array may have tens of thousands of dimensions  High complexity of data ◦ Data streams and sensor data ◦ Time-series data, temporal data, sequence data ◦ Structure data, graphs, social networks and multi-linked data ◦ Heterogeneous databases and legacy databases ◦ Spatial, spatiotemporal, multimedia, text and Web data ◦ Software programs, scientific simulations  New and sophisticated applications
  17. 17.  This is a view from typical database systems and data warehousing communities  Data mining plays an e s s e n t i a l ro l e i n t h e knowledge discover y process Data Cleaning Data Integration Databases Data Warehouse Task-relevant Data Selection Data Mining Pattern Evaluation
  18. 18. The knowledge discovery process is depicted in Figure as an iterative sequence of the following steps: 1. Data cleaning (to remove noise and inconsistent data) 2. Data integration (where multiple data sources may be combined) 3. Data selection (where data relevant to the analysis task are retrieved from the database) 4. Data transformation (where data are transformed and consolidated into forms appropriate for mining by performing summary or aggregation operations) 5. Data mining (an essential process where intelligent methods are applied in order to extract data patterns) 6. Pattern evaluation (to identify the truly interesting patterns representing knowledge based on interestingness measures) 7. Knowledge presentation (where visualization and knowledge representation techniques are used to present the mined knowledge to the user)
  19. 19. Increasing potential to support business decisions End User Business Analyst Data Analyst DBA Decision Making Data Presentation Visualization Techniques Data Mining Information Discovery Data Exploration Statistical Summary, Querying, and Reporting Data Preprocessing/Integration, Data Warehouses Data Sources Paper, Files, Web documents, Scientific experiments, Database Systems
  20. 20. 1. Identify the problem 2. Use data mining techniques to transform the data into information 3.Act on the information 4. Measure the results
  21. 21.  Query ◦ Well defined ◦ SQL  Query ◦ Poorly defined ◦ No precise query language  Data ◦ Operational data  Output ◦ Precise ◦ Subset of database  Data ◦ Not operational data  Output ◦ Fuzzy ◦ Not a subset of database
  22. 22.  Database  Data Mining – Find all customers who have purchased milk – Find all items which are frequently purchased with milk. (Association rules) – Find all credit applicants with last name of Smith. – Identify customers who have purchased more than $10,000 in the last month. – Find all credit applicants who are poor credit risks. (Classification) – Identify customers with similar buying habits. (Clustering)
  23. 23.  Data to be mined ◦ Database data (extended-relational, object-oriented, heterogeneous, legacy), data warehouse, transactional data, stream, spatiotemporal, time-series, sequence, text and web, multi-media, graphs & social and information networks  Knowledge to be mined (or: Data mining functions) ◦ Characterization, discrimination, association, classification, clustering, trend/deviation, outlier analysis, etc. ◦ Descriptive vs. predictive data mining ◦ Multiple/integrated functions and mining at multiple levels  Techniques utilized ◦ Data-intensive, data warehouse (OLAP), machine learning, statistics, pattern recognition, visualization, high-performance, etc.  Applications adapted ◦ Retail, telecommunication, banking, fraud analysis, bio-data mining, stock market analysis, text mining,Web mining, etc.
  24. 24.  Database-oriented data sets and applications ◦ Relational database, data warehouse, transactional database  Advanced data sets and advanced applications ◦ Data streams and sensor data ◦ Time-series data, temporal data, sequence data (incl. bio-sequences) ◦ Structure data, graphs, social networks and multi-linked data ◦ Object-relational databases ◦ Heterogeneous databases and legacy databases ◦ Spatial data and spatiotemporal data ◦ Multimedia database ◦ Text databases ◦ The World-Wide Web
  25. 25.  Decisions in data mining ◦ Kinds of databases to be mined ◦ Kinds of knowledge to be discovered ◦ Kinds of techniques utilized ◦ Kinds of applications adapted  Data mining tasks ◦ Descriptive data mining ◦ Predictive data mining
  26. 26.  PredictionTasks (Predictive) ◦ The objective of these tasks is to predict the value of a particular attribute based on the values of other attributes. ◦ The attribute to be predicted is commonly known as the target or dependent variable, while the attributes used for making the prediction are known as the explanatory or independent variables.  DescriptionTasks (Descriptive) ◦ Find human-interpretable patterns that describe the data. • Common data mining tasks  Classification [Predictive]  Clustering [Descriptive]  Association Rule Discovery [Descriptive]  Sequential Pattern Discovery [Descriptive]  Regression [Predictive]  Deviation Detection [Predictive]
  27. 27. Tid Refund Marital Status Taxable Income Cheat 1 Yes Single 125K No 2 No Married 100K No 3 No Single 70K No 4 Yes Married 120K No 5 No Divorced 95K Yes 6 No Married 60K No 7 Yes Divorced 220K No 8 No Single 85K Yes 9 No Married 75K No 10 No Single 90K Yes 11 No Married 60K No 12 Yes Divorced 220K No 13 No Single 85K Yes 14 No Married 75K No 15 No Single 90K Yes 10 Predictive M odeling Clustering Association Rules Anom aly Detection Milk Data Data Mining Tasks …
  28. 28. 29  Information integration and data warehouse construction ◦ Data cleaning, transformation, integration, and multidimensional data model  Data cube technology ◦ Scalable methods for computing (i.e., materializing) multidimensional aggregates ◦ OLAP (online analytical processing)  Multidimensional concept description: Characterization and discrimination ◦ Generalize, summarize, and contrast data characteristics, e.g., dry vs. wet region
  29. 29. 30  Frequent patterns (or frequent itemsets) ◦ What items are frequently purchased together in your Walmart?  Association, correlation vs. causality ◦ A typical association rule  Diaper  Beer [0.5%, 75%] (support, confidence)  Tea  Sugar [0.5%, 75%] (support, confidence) ◦ Are strongly associated items also strongly correlated? ◦ How to mine such patterns and rules efficiently in large datasets? ◦ How to use such patterns for classification, clustering, and other applications?
  30. 30. 31  Classification and label prediction ◦ Construct models (functions) based on some training examples ◦ Describe and distinguish classes or concepts for future prediction  E.g., classify countries based on (climate), or classify cars based on (gas mileage) ◦ Predict some unknown class labels  Typical methods ◦ Decision trees, naïve Bayesian classification, support vector machines, neural networks, rule-based classification, pattern- based classification, logistic regression, …  Typical applications: ◦ Credit card fraud detection, direct marketing, classifying stars, diseases, web-pages, …
  31. 31. 32  Unsupervised learning (i.e., Class label is unknown)  Group data to form new categories (i.e., clusters), e.g., cluster houses to find distribution patterns  Principle: Maximizing intra-class similarity & minimizing interclass similarity  Many methods and applications
  32. 32. 33  Outlier analysis ◦ Outlier:A data object that does not comply with the general behavior of the data ◦ Noise or exception? ― One person’s garbage could be another person’s treasure ◦ Methods: by product of clustering or regression analysis, … ◦ Useful in fraud detection, rare events analysis
  33. 33. 34  Sequence, trend and evolution analysis ◦ Trend, time-series, and deviation analysis: e.g., regression and value prediction ◦ Sequential pattern mining  e.g., first buy digital camera, then buy large SD memory cards ◦ Periodicity analysis ◦ Motifs and biological sequence analysis  Approximate and consecutive motifs ◦ Similarity-based analysis  Mining data streams ◦ Ordered, time-varying, potentially infinite, data streams
  34. 34. 35  Graph mining ◦ Finding frequent subgraphs (e.g., chemical compounds), trees (XML), substructures (web fragments)  Information network analysis ◦ Social networks: actors (objects, nodes) and relationships (edges)  e.g., author networks in CS, terrorist networks ◦ Multiple heterogeneous networks  A person could be multiple information networks: friends, family, classmates, … ◦ Links carry a lot of semantic information: Link mining  Web mining ◦ Web is a big information network: from PageRank to Google ◦ Analysis of Web information networks  Web community discovery, opinion mining, usage mining, …
  35. 35. 36  Are all mined knowledge interesting? ◦ One can mine tremendous amount of “patterns” and knowledge ◦ Some may fit only certain dimension space (time, location, …) ◦ Some may not be representative, may be transient, …  Evaluation of mined knowledge → directly mine only interesting knowledge? ◦ Descriptive vs. predictive ◦ Coverage ◦ Typicality vs. novelty ◦ Accuracy ◦ Timeliness ◦ …
  36. 36.  Mining Methodology ◦ Mining various and new kinds of knowledge ◦ Mining knowledge in multi-dimensional space ◦ Data mining:An interdisciplinary effort ◦ Boosting the power of discovery in a networked environment ◦ Handling noise, uncertainty, and incompleteness of data ◦ Pattern evaluation and pattern- or constraint-guided mining  User Interaction ◦ Interactive mining ◦ Incorporation of background knowledge ◦ Presentation and visualization of data mining results
  37. 37.  Efficiency and Scalability ‒ Efficiency and scalability of data mining algorithms ‒ Parallel, distributed, stream, and incremental mining methods  Diversity of data types ‒ Handling complex types of data ‒ Mining dynamic, networked, and global data repositories  Data mining and society ‒ Social impacts of data mining ‒ Privacy-preserving data mining ‒ Invisible data mining
  38. 38.  Web page analysis: from web page classification, clustering to PageRank & HITS algorithms  Collaborative analysis & recommender systems  Basket data analysis to targeted marketing  Biological and medical data analysis: classification, cluster analysis (microarray data analysis), biological sequence analysis, biological network analysis  Data mining and software engineering (e.g., IEEE Computer,Aug. 2009 issue)  From major dedicated data mining systems/tools (e.g., SAS, MS SQL- Server Analysis Manager, Oracle Data Mining Tools) to invisible data mining