走进数据科学-英文版-课件.pptx

1、Data Mining:Theory&AlgorithmsMining?Warehousing?5Technology Advancement6Technology Advancement7The World of Data8Data Rich,Information Poor910Learning Resources11International Conference on Data MiningInternational Conference on Data EngineeringInternational Conference on Machine LearningInternation

2、al Joint Conference on Artificial IntelligencePacific-Asia Conference on Knowledge Discovery and Data MiningACM SIGKDD Conference on Knowledge Discovery and Data MiningLearning Resources12Learning Resources13Xindong WuZhihua ZhouJiawei HanJian PeiQiang YangChih-Jen LinPhilip S.Yu Changshui ZhangLear

3、ning Resources14Interdisciplinary15Data MiningMachine LearningPattern RecognitionStatisticsArtificial IntelligenceUbiquitous16Data MiningBusiness IntelligenceData AnalyticsBig DataDecision SupportCustomer Relationship ManagementComprehensive Learning17Class Teaching Thinking DiscussionReading Materi

4、als Extension InspirationPractice Techniques ApplicationsLearning Listening1820DatavDefinition“Data are pieces of information that represent the qualitative or quantitative attributes of a variable or set of variables.Data are often viewed as the lowest level of abstraction from which information an

5、d knowledge are derived.”vData TypesContinuous,BinaryDiscrete,StringSymbolicvStoragePhysicalLogicalvMajor IssuesTransformationErrors and Corruption 21What is Big Data?v“Big data is high-volume,high-velocity and high-variety information assets that demand cost-effective,innovative forms of informatio

6、n processing for enhanced insight and decision making.”Gartnerv“Big data refers to datasets whose size is beyond the ability of typical database software tools to capture,store,manage,and analyze.”Mckinsey&Company22Big Data23Public Security24Health Care Application25Effectiveness ResearchPersonalize

7、d MedicineLocation Data:Urban Planning26Location Data:Mobile User27Location Data:Shopper28Retail Data:Targeted Marketing29Retail Data:Sentiment Analysis30Social Networks31Sports32Attractiveness Mining3334Open Datav Technically Open:available in a machine-readable standard format,which means it can b

8、e retrieved and meaningfully processed by a computer application.v Legally Open:explicitly licensed in a way that permits commercial and non-commercial use without restrictions.35Where to find data?36Open Government Data37Data Miningv People have been analysing and investigating data for centuries.v

9、 StatisticsMean,Variance,Correlation,Distribution v In modern days,data are often far beyond human comprehension.Diversity,Volume,Dimensionalityv DefinitionData Mining is the process of automatically extracting interesting and useful hidden patterns from usually massive,incomplete and noisy data.v N

10、ot a fully automatic processHuman interventions are often inevitable.Domain KnowledgeData Collection and Pre-processingv Synonym:Knowledge Discovery38v“If you are looking for a career where your services will be in high demand,you should find something where you provide a scarce,complementary servic

11、e to something that is getting ubiquitous and cheap.So whats getting ubiquitous and cheap?Data.And what is complementary to data?Analysis.So my recommendation is to take lots of courses about how to manipulate and analyze data:databases,machine learning,econometrics,statistics,visualization,and so o

12、n.”An interview with Google Chief Economist Hal Varian from the New York TimesIs DM really important?3940Business Intelligence41From Data To Intelligence42Decision ModelsData MiningPreprocessingDatabaseDecision SupportKnowledgeInformationDataData Integration&Analysis43The Process of Data Mining44454

13、647DM Techniques-Classificationv“Classification is a procedure in which individual items are placed into groups based on quantitative information on one or more characteristics(referred to as variables)and based on a training set of previously labeled items.”v Given a training set:(x1,y1),(xn,yn),pr

14、oduce a classifier(function)that maps any unknown object xi to its class label yi.v AlgorithmsDecision TreesK-Nearest NeighboursNeural NetworksSupport Vector Machinesv ApplicationsChurn PredictionMedical Diagnosis48XYClassification Boundaries49?XYIncomeSavingsLow RiskHigh RiskOverfitting Classificat

15、ion50Cross Validation51DataTraining SetTest SetEvaluationGenerated ModelsConfusion Matrix52Confusion Matrix Actual ValuePositiveNegativeTotalPredictedValuePositiveTruePositiveFalsePositivePNegativeFalseNegativeTrueNegativeNTotalPNTPR=TP/(TP+FN)TNR=TN/(TN+FP)Accuracy=(TP+TN)/(P+N)Receiver Operating C

16、haracteristic53Very small thresholdVery large thresholdRandom guessCost Sensitive Learning 54Lift Analysis 5556DM Techniques-Clusteringv“Clustering is the assignment of a set of observations into subsets(called clusters)so that observations in the same cluster are similar in some sense.”vDistance Me

17、tricsEuclidean DistanceManhattan DistanceMahalanobis DistancevAlgorithmsK-MeansSequential LeaderAffinity PropagationvApplicationsMarket ResearchImage SegmentationSocial Network Analysis57What is the difference between classification and clustering?Hierarchical Clustering58DM Techniques Association R

18、ule59Association Rule60Transaction IDMilkBreadButterBeer1110020110300014111050100ButterBread Milk,DM Techniques Regression61,XfY kkzkkxxzeyxxyxxyxy110110221010,11Overfitting Regression62yxSeeing is Knowing63Performance Dashboard6465Data PreprocessingvReal data are often surprisingly dirty.A Major Ch

19、allenge for Data MiningvTypical IssuesMissing Attribute ValuesDifferent Coding/Naming SchemesInfeasible ValuesInconsistent DataOutliersvData QualityAccuracyCompletenessConsistencyInterpretabilityCredibilityTimeliness66Data PreprocessingvData CleaningFill in missing values.Correct inconsistent data.I

20、dentify outliers and noisy data.vData IntegrationCombine data from different sources.vData TransformationNormalizationAggregationType ConversionvData ReductionFeature SelectionSampling6768Internet Privacy6970Privacy Protectionv Data:A Double-Edged Sword People can benefit greatly from data analysis.

21、The consequence of information leakage can be catastrophic.v People may be reluctant to give sensitive information due to privacy concerns.Drug,Tax,Sexuality v How to find out the percentage of people with a certain attribute?The interviewer should not know the true answer of each respondent.v Rando

22、mized Response Used in structured survey research.Can maintain the confidentiality of respondents.71Privacy Protectionv Two questions are presented:Q1:I have the attribute A.Q2:I do not have the attribute A.v The respondent uses a random device to:Answer Q1 with probability p.Answer Q2 with probabil

23、ity 1-p.The interviewer has no idea about which question is answered.72121)()()()1()()(*ppTruePTruePFalsePpTruePpTrueP5.0pCloud Computing73Cloud ComputingvPay As You GovSoftware as a Service vPlatform as a ServicevInfrastructure as a Service74Parallel Computing75Parallel Computing7677Mobile Supercom

24、puting78Intel MIC79The Big Picture80v Why bother so many different algorithms?v No algorithm is always superior to others.v No parameter setting is optimal over all problems.v Look for the best match between problem and algorithm.Experience Trial and Errorv Factors to consider:Applicability Computat

25、ional Complexity Interpretabilityv Always start with simple ones.No Free Lunch8182Just in Case Someone Asks 83Just in Case Someone Asks 84Grouping85XYGroup AGroup BViolent Crime vs.Video Game86Tricky?v Is there correlation between height and business success?v Average American Male is 59.v Only 3.9%adult American men are taller than 62”.v Around 30%CEOs of Fortune 500 are taller than 62.87Survivorship Bias88这是真的这是真的吗？吗？89时间去时间去哪儿了？哪儿了？90感谢您的聆听！