异常值（outlier）如何处理？

在数据挖掘的过程中，我们可能会经常遇到一些偏离于预测趋势之外的数据，通常我们称之为异常值。
我们通常将这样的一些数据的出现归为误差，各种各样的误差，也许是仪器、也许是测量人员、也许是其他的一些什么原因……
基于这样的原因，我们更多的时候将异常值弃之不理，可是这样的处理真的正确吗？

基本的想法：

1. 丢弃：
    如果主要考虑的是统计学上的分布情况，我们更为在意的是群体的统计学的特点，那么，这少数数据表现出来的异常一般认为是可以丢弃的；

2. 重现：
    如果说这样的数据出现一次，我们将这归为误差，那么重现无疑是最好的证明方式，不论是证实还是证伪；

3. 找到原因：
    一切现象的出现都是有其原因的，所不同的只是原因是什么。如果能够找到出现异常的原因，也许比目标的结论更有意义。

所以处理的一般建议：


     ->N->暂时丢弃（但记录在案，这很重要）

重现 ->Y-> 找到原因 -> Y->成功
 
                  -> N ->交给合适的人
（也许你没有能力找到原因，但是你已经发现了她的踪影）



关于outlier的简要介绍：


更多极可能有用的书：

1.Geographic data mining and knowledge discovery [Miller, Harvey J.]
2.Identification of outliers [Hawkins, Douglas M.]
3.Identification of outliers in large statistical data [Hadi]
4.Numerical computer methods. Part C [Johnson, Michael L., 1947-]
5.Outliers in statistical data [Barnett, Vic.1994]





一本也许有趣的书

Outliers: The Story of Success

Details


Synopsis
Cultural critic Malcolm Gladwell applies his best trend watching insights and peripatetic methodology to the biq question of why some people achieve success and others do not. These Outliers, as he calls them--successful people who stand out from the rest of the crowd--may be business people, artists, athletes, entertainers, scientists, or work in other fields. In a variation of the nature versus nurture debates, Gladwell seeks answers in culture, environment, and upbringing. Without seeking simple formulas, he examines a range of artifacts--IQ tests, for example, and birth-date data--as he digs for answers in a book that draws on psychology, business, and sociology, and which explains difficult concepts in an accessible manner. THE OUTLIERS confirms that Malcolm Gladwell, here a public intellectual, is himself an Outlier.

Size
Length:	320 pages
Height:	8.3 in
Width:	5.8 in
Thickness:	1.0 in
Weight:	14.4 oz

Industry Reviews
"Malcolm Gladwell has a rare ability: he can transform academic research into engaging fables spotlighting real people. His new book...is beautifully written."
(11/02/2008)

"Buoyed by two runaway bestsellers, BLINK and THE TIPPING POINT, Gladwell has positioned himself as a roving ambassador between cultural and corporate America, penetrating boardrooms and living rooms, providing bullet points for cocktail parties and management seminars, and changing not just the things we talk about but the way we talk about them."
(11/17/2008)

"OUTLIERS has much in common with Gladwell's earlier work. It is a pleasure to read and leaves you mulling over its inventive theories for days afterward."
(11/30/2008)