03Preprocessing.pptVIP免费

1DataMining:ConceptsandTechniques(3rded.)—Chapter3—JiaweiHan,MichelineKamber,andJianPeiUniversityofIllinoisatUrbana-Champaign&SimonFraserUniversity©2011Han,Kamber&Pei.Allrightsreserved.01/26/2523Chapter3:DataPreprocessingDataPreprocessing:AnOverviewDataQualityMajorTasksinDataPreprocessingDataCleaningDataIntegrationDataReductionDataTransformationandDataDiscretizationSummary4DataQuality:WhyPreprocesstheData?Measuresfordataquality:AmultidimensionalviewAccuracy:correctorwrong,accurateornotCompleteness:notrecorded,unavailable,…Consistency:somemodifiedbutsomenot,dangling,…Timeliness:timelyupdate?Believability:howtrustablethedataarecorrect?Interpretability:howeasilythedatacanbeunderstood?5MajorTasksinDataPreprocessingDatacleaningFillinmissingvalues,smoothnoisydata,identifyorremoveoutliers,andresolveinconsistenciesDataintegrationIntegrationofmultipledatabases,datacubes,orfilesDatareductionDimensionalityreductionNumerosityreductionDatacompressionDatatransformationanddatadiscretizationNormalizationConcepthierarchygeneration6Chapter3:DataPreprocessingDataPreprocessing:AnOverviewDataQualityMajorTasksinDataPreprocessingDataCleaningDataIntegrationDataReductionDataTransformationandDataDiscretizationSummary7DataCleaningDataintheRealWorldIsDirty:Lotsofpotentiallyincorrectdata,e.g.,instrumentfaulty,humanorcomputererror,transmissionerrorincomplete:lackingattributevalues,lackingcertainattributesofinterest,orcontainingonlyaggregatedatae.g.,Occupation=“”(missingdata)noisy:containingnoise,errors,oroutlierse.g.,Salary=“−10”(anerror)inconsistent:containingdiscrepanciesincodesornames,e.g.,Age=“42”,Birthday=“03/07/2010”Wasrating“1,2,3”,nowrating“A,B,C”discrepancybetweenduplicaterecordsIntentional(e.g.,disguisedmissingdata)Jan.1aseveryone’sbirthday?8Incomplete(Missing)DataDataisnotalwaysavailableE.g.,manytupleshavenorecordedvalueforseveralattributes,suchascustomerincomeinsalesdat...