數(shù)據(jù)挖掘ppt

1、智能信息處理32學時,費高雷fgl@uestc.edu.cn電子科技大學 通信與信息工程學院,教師信息,費高雷電話：61830209郵箱：fgl@uestc.edu.cn地址：科研樓B325研究方向：網絡層析成像反演理論與方法復雜多維信息處理,提綱,引言數(shù)據(jù)挖掘概念及必要性數(shù)據(jù)挖掘的主要任務案列分析,Teaching Material,數(shù)據(jù)挖掘：概念與技術（原書第3版）[美]Jiawei Han ，

2、等 著范明 ，孟小峰 譯 機械工業(yè)出版社 特點：大量的圖解、實例和練習參考：第2版,Reference Books,考核方式,成績構成：期末70%、期中5%、平時25%（平時 ＝ 實驗15% + 考勤10%）考試方式：期中隨堂開卷、期末開卷實驗成績：結果分析50%、報告50%課堂講述、課后完成,Content,1、引言 2、認識數(shù)據(jù) 3、數(shù)據(jù)預處理 4、數(shù)據(jù)倉庫與聯(lián)機分析

3、處理（自學）5、數(shù)據(jù)立方體技術 （自學）6、挖掘關聯(lián)規(guī)則（重點）,7、高級模式挖掘（自學）8、分類：基本概念（重點）9、分類：高級方法（自學）10、聚類分析：基本概念和方法（重點）11、高級聚類分析（自學）12、離群點檢測13、智能信息處理技術前沿,數(shù)據(jù)挖掘技術不斷進步、新方法不斷涌現(xiàn)，本課程注重基本概念和基本方法,開課必要性：工業(yè)界的角度,,,開課必要性：學術的角度,,提綱,引言數(shù)據(jù)挖掘概念及必要性數(shù)據(jù)挖掘的主要任

4、務案列分析,Trends leading to Data Flood,More data is generated:Bank, telecom, other business transactions ...Scientific Data: astronomy, biology, etcWeb, text, and e-commerce More data is captured:Storage technology fas

5、ter and cheaperDBMS capable of handling bigger DB,舉例,Europe's Very Long Baseline Interferometry (VLBI) has 16 telescopes, each of which produces 1 Gigabit/second of astronomical data over a 25-day observation sessio

6、n storage and analysis a big problemWalmart reported to have 24 Tera-byte DB AT&T handles billions of calls per daydata cannot be stored -- analysis is done on the fly,Growth Trends,Moore’s lawComputer Speed dou

7、bles every 18 monthsStorage lawtotal storage doubles every 9 monthsConsequencevery little data will ever be looked at by a humanKnowledge Discovery is NEEDED to make sense and use of data.,智能信息處理,為適應信息時代信息處理的要求，當前信息

8、處理技術向智能化發(fā)展，廣泛的模擬人的智能來處理各種信息智能信息處理是計算機科學中的交叉學科，其目標是利用計算機技術處理海量復雜信息，研究新的、先進的理論技術,數(shù)據(jù)挖掘（Data Mining）?智能信息處理,,,,,,涉及的相關領域,,Statistics,MachineLearning,Databases,Visualization,Data Mining,,,,,,,,,,,,,Transformed Data,Pattern

9、sand Rules,Target Data,RawData,,,,Knowledge,Data Mining,Transformation,Interpretation& Evaluation,Selection& Cleaning,Integration,Understanding,數(shù)據(jù)挖掘的重要性,DATAWarehouse,Knowledge,提綱,引言數(shù)據(jù)挖掘概念及必要性數(shù)據(jù)挖掘的主要任務案

10、列分析,Data Mining Tasks: Classification,,,,,,,,,,,,,,,,,,,,,,Learn a method for predicting the instance class from pre-labeled (classified) instances,Many approaches: Statistics, Decision Trees, Neural Networks, ...,,,,

11、,,,,,,,Classification: Decision Trees,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,X,Y,,,,,,,if X > 5 then blueelse if Y > 3 then blueelse if X > 2 then greenelse blue,5,2,3,,,Classification: Neural Nets,Can select more comp

12、lex regionsCan be more accurateAlso can overfit the data – find patterns in random noise,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Data Mining Tasks: Clustering,,,,,cluster,,outlier,Linear Regression,,,,,,,,,,,,,,,,x,y,y = x + 1,,

13、,X1,Y1,(salary),(age),Example of linear regression,Major Data Mining Tasks,Classification: predicting an item classClustering: finding clusters in dataAssociations: e.g. A & B & C occur frequentlyVisualization

14、: to facilitate human discoveryEstimation: predicting a continuous valueDeviation Detection: finding changesLink Analysis: finding relationships…,實驗工具,Weka主頁：http://www.cs.waikato.ac.nz/ml/weka/,提綱,引言數(shù)據(jù)挖掘概念及必要性數(shù)據(jù)挖

15、掘的主要任務案列分析,OD flow I-b,,案列：IP網絡流量異常行為檢測,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,OD flow I-b,Fri,S

16、at,Sun,案列：IP網絡流量分類(一),?,?,?,?,?,HTTP,Bittorrent,SMTP,,?,Traffic classification (TC),?,FeaturesReacting on application development,案列：IP網絡流量分類(二),29,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,Training instances for class ATraining i