Chinaunix首页 | 论坛 | 博客
  • 博客访问: 568665
  • 博文数量: 86
  • 博客积分: 2581
  • 博客等级: 少校
  • 技术积分: 793
  • 用 户 组: 普通用户
  • 注册时间: 2009-01-05 20:09
文章分类

全部博文(86)

文章存档

2009年(86)

我的朋友

分类: Mysql/postgreSQL

2009-08-30 21:23:49

I ran into this little problem today; when dividing two whole number the result MySQL gave me was nowhere near as precise as I needed it to be. The values were truncated which caused me some errors in my application.

Example:
mysql> select 1*0.00001;
+-----------+
| 1*0.00001 |
+-----------+
| 0.00001 |
+-----------+

Ok, that looks fine. Let's do the same thing using division:

mysql> select 1/100000;
+----------+
| 1/100000 |
+----------+
| 0.0000 |
+----------+

Oops! What happened?

The issue is that the maximum precision of the result value depends on the number of decimal places in the arguments. Since the second version uses two whole numbers, the result uses the default number of decimal places, which is 4. So you really have to take care to make sure you're getting the precision you want out of your math operations!

To MySQL's credit,
the behavior is clearly documented (however, I maintain it's still a bit troublesome because how often would you expect the division operator to have a list of instructions and caveats?)

There are at least a couple simple solutions:
mysql> SELECT CAST(1/100000 AS DECIMAL(8,5) );
+---------------------------------+
| CAST(1/100000 AS DECIMAL(8,5) ) |
+---------------------------------+
| 0.00001 |
+---------------------------------+

Or

mysql> SET div_precision_increment=5;
mysql> SELECT 1/100000;
+----------+
| 1/100000 |
+----------+
| 0.00001 |
+----------+

阅读(8791) | 评论(0) | 转发(0) |
给主人留下些什么吧!~~