5. 50个并发2000地址，邮件只包含信头，每个连接3封信的结果：
11:17,3613,1447,0,1849,0
11:18,3876,1547,0,1930,0
11:19,4088,1634,0,2080,0
11:20,3920,1566,0,1977,0
11:21,4057,1621,0,2021,0
11:22,3946,1573,0,1960,0
11:23,3316,1324,0,1695,0
11:24,2576,1030,0,1295,0
11:25,2889,1155,0,1439,0
^^^^^^^^^^^^^^^^^^^^^^*
11:26,3984,1596,0,1990,0
11:27,3986,1588,0,1993,0
11:28,4164,1662,0,2064,0
11:29,3973,1591,0,1986,0
11:30,3991,1592,0,1982,0
11:31,3521,1406,0,1758,0
11:32,2590,1035,0,1297,0
^^^^^^^^^^^^^^^^^^^^^^*
11:33,2655,1058,0,1322,0
11:34,2745,1097,0,1348,0
注意^^*所标记的时间段，SMTP注入性能大幅度下降。检查/var/spool/postfix目录，发现qmgr已达到处理上限了，该上限由qmgr_message_active_limit 参数设置。故此incoming开始增长很多，Postfix开始放慢处理速度。
以上所有测试都是在default_destination_concurrency_limit = 10的条件下完成的，所以在virtual进行本地mailbox投递时，并发数很少，而active目录增长得很快，没有足够多的virtual进行投递，结果是无法快速地将邮件写到硬盘。
6.提升default_destination_concurrency_limit：
default_destination_concurrency_limit = 1000
此时出现另外一个问题：active队列大概只维持在200-300百封信的样子，但defer/defered目录开始有所增长，并且SMTP注入速度下降，大概只有1850封信/分钟的速度。此时virtual的并发数目达到了400个！
可以推断，这是因为virtual投递速度和数量大幅度增加，致使磁盘写(write)操作过于频繁，使磁盘I/O性能吃紧，导致性能下降。
解决办法：使用分布式存储，利用mailswitch这一类技术，将最终的邮件投递由存储机器上的软件完成。
将这个default_destination_concurrency_limit改成100，速度提升了一些，但很多出现了很多lookup failure：
time,messages,data(K),errors,connections,SSL connections
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
11:57,2773,1124,2,1459,0
11:58,2302,920,0,1156,0
11:59,2492,996,0,1264,0
12:00,2204,879,0,1087,0
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
12:01,2311,924,2,1146,0
Server error:451 : Temporary lookup failure
.
12:02,2487,993,1,1265,0
12:03,2284,912,0,1112,0
12:04,1515,602,81,808,0
^^ 81个错误
12:05,1969,787,11,1012,0
12:06,2139,858,0,1073,0
12:07,2174,868,0,1095,0
12:08,2214,884,0,1093,0
12:09,2252,900,0,1118,0
Server error:451 : Temporary lookup failure
并发降至50：
default_destination_concurrency_limit = 50
测试的log：
Server error:451 : Temporary lookup failure
.
12:21,2645,1064,2,1379,0
12:22,3033,1211,0,1503,0
12:23,2996,1206,0,1472,0
Server error:451 : Temporary lookup failure
.
12:24,2911,1152,1,1455,0
12:25,2716,1093,0,1359,0
12:26,2844,1131,0,1434,0
12:27,2748,1096,0,1360,0
12:28,2738,1107,0,1396,0
12:29,2652,1051,0,1352,0
12:30,2759,1108,0,1355,0
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
12:31,2472,987,13,1253,0
Postal测试时打开100个并发,每个邮件最大10k,每个连接1封,发5000封，产生了非常多的failure，不过速度还不错。
接下来Postal 使用50并发，每并发1个邮件，打开了openLDAP cache支持：
time,messages,data(K),errors,connections,SSL connections
23:03,3540,19196,46,3685,0
23:04,4246,23136,0,4247,0
23:05,4182,22873,19,4194,0
^^19个错误
23:06,3736,19922,74,3808,0
^^ 70多个错误!
23:33,3683,3359,0,3733,0
23:34,3935,3632,0,3935,0
23:35,4041,3664,0,4041,0
23:36,3990,3636,0,3989,0
23:37,4164,3798,0,4165,0
23:38,3872,3512,0,3872,0
23:39,3210,2945,0,3209,0
23:40,2624,2400,0,2625,0
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
23:41,2502,2286,3,2505,0
23:42,2760,2508,0,2759,0
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
23:43,2528,2305,2,2531,0
23:44,2078,1972,0,2078,0
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
23:45,1520,1387,4,1524,0
23:46,1715,1536,0,1715,0
23:47,1475,1362,0,1475,0
23:48,1508,1371,0,1507,0
可见active目录的上限已经达到了，所以SMTP注入速度开始下降，邮件已经开始处理不过来了。
调整Postal 参数，使用50并发，每个并发1个msg，尽最大能力发送，打开openLDAP cache模式，qmgr_messages_active_limit=50000，测试持续了近一个小时，日志如下：
[root@ns1 postal-0.61]# ./do.sh
time,messages,data(K),errors,connections,SSL connections
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
23:52,3663,3353,3,3714,0
23:53,3784,3454,0,3786,0
23:54,3747,3409,0,3747,0
23:55,3861,3498,0,3861,0
23:56,3922,3577,0,3922,0
23:57,3822,3491,0,3822,0
23:58,3748,3429,0,3747,0
23:59,3608,3264,0,3609,0
00:00,3320,3027,0,3319,0
00:01,3410,3117,0,3411,0
00:02,3317,3042,0,3316,0
00:03,3580,3286,0,3581,0
00:04,3507,3205,0,3507,0
00:05,3475,3114,0,3475,0
00:06,3259,2995,0,3258,0
00:07,3186,2912,0,3186,0
00:08,3552,3216,0,3553,0
00:09,2887,2635,0,2887,0
00:10,2517,2292,0,2517,0
00:11,2721,2505,0,2721,0
00:12,2574,2317,0,2574,0
00:13,2587,2357,0,2587,0
00:14,2573,2342,0,2573,0
00:15,2704,2463,0,2704,0
00:16,2647,2405,0,2647,0
00:17,2585,2383,0,2585,0
00:18,2421,2235,0,2421,0
00:19,2573,2349,0,2572,0
00:20,2378,2170,0,2378,0
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
...... 期间Temporary lookup failure有12个
00:21,2330,2144,15,2346,0
00:22,2610,2376,0,2610,0
00:23,2551,2317,0,2550,0
00:24,2586,2380,0,2587,0
00:25,2530,2282,0,2528,0
00:26,2558,2306,0,2560,0
00:27,2444,2232,0,2444,0
00:28,2509,2250,0,2508,0
00:29,2523,2293,0,2524,0
00:30,2445,2210,0,2445,0
00:31,2555,2345,0,2554,0
00:32,2455,2225,0,2456,0
00:33,2477,2266,0,2476,0
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
00:34,2236,2038,2,2238,0
00:35,2245,2061,0,2245,0
00:36,2536,2311,0,2537,0
00:37,2210,2004,0,2210,0
00:38,2385,2150,0,2385,0
00:39,2463,2242,0,2463,0
00:40,2496,2291,0,2495,0
00:41,2471,2261,0,2471,0
00:42,2422,2208,0,2422,0
00:43,2370,2159,0,2371,0
00:44,2387,2197,0,2386,0
00:45,2203,2028,0,2203,0
00:46,2494,2301,0,2494,0
00:47,2313,2133,0,2314,0
00:48,2255,2016,0,2255,0
00:49,2428,2236,0,2428,0
00:50,2327,2113,0,2326,0
00:51,2436,2244,0,2437,0
00:52,2334,2127,0,2334,0
00:53,2399,2202,0,2398,0
00:54,2271,2078,0,2272,0
00:55,2255,2046,0,2255,0
00:56,2397,2205,0,2397,0
00:57,2392,2183,0,2392,0
00:58,2364,2161,0,2364,0
00:59,2241,2031,0,2240,0
01:00,2354,2137,0,2355,0
01:01,2223,2000,0,2223,0
01:02,2326,2118,0,2325,0
01:03,2343,2155,0,2344,0
01:04,2388,2209,0,2388,0
队列里积压了13万封信，70多分钟内注入了22万封信，正确投递并写入用户$HOME目录的有95000多封，由于人为限制了投递的并发数，因此这个结果也在意料之内了。
[root@ns2 postfix]# postsuper -d ALL
postsuper: Deleted: 137410 messages
[root@ns2 postfix]# du -sk /home/domains/bigmail.hzqbbc.com/
392124 /home/domains/bigmail.hzqbbc.com
[root@ns2 postfix]# find /home/domains/bigmail.hzqbbc.com/ | wc -l
95349
6. 新一轮测试
Postal 使用50/100个并发，1-3封邮件/每连接，将Postfix升级到Snapshot 2.0.2，并打开proxymap daemon，做如下的测试：
[root@ns1-bjcnc postal-0.61]# ./do.sh
time,messages,data(K),errors,connections,SSL connections
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
14:02,3743,3366,3,1880,0
14:03,4244,3809,0,2129,0
14:04,4345,3915,0,2204,0
14:05,4187,3790,0,2114,0
[root@ns1-bjcnc postal-0.61]# ./do.sh
time,messages,data(K),errors,connections,SSL connections
14:10,4380,3973,5,2289,0
14:11,4481,4065,0,2215,0
14:12,4541,4083,0,2260,0
.
14:13,2741,2455,14,1389,0
14:14,3927,3549,0,1949,0
14:15,3611,3235,0,1807,0
14:16,3634,3304,0,1796,0
14:17,3889,3475,0,1933,0
[root@ns1-bjcnc postal-0.61]# ./do.sh
time,messages,data(K),errors,connections,SSL connections
14:22,1838,1644,0,961,0
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
14:23,1910,1721,2,959,0
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
14:24,1989,1769,2,1007,0
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
14:25,1828,1652,2,915,0
[root@ns1-bjcnc postal-0.61]# ./do.sh
Can't open config file "conver". Doing no expansion.
time,messages,data(K),errors,connections,SSL connections
Server error:451 : Temporary lookup failure
.
14:27,4110,3721,1,2105,0
14:28,289,221,0,149,0
[root@ns1-bjcnc postal-0.61]# cat do.sh
#!/bin/sh
MAXMSGSIZE=1
PROC=50
MSGPERCONN=3
MSGPERMIN=5000
SSLPERCENT=0
SMTP_HOST=210.82.193.91
./postal -m $MAXMSGSIZE -p $PROC -c $MSGPERCONN -r $MSGPERMIN -a \
-b netscape $SMTP_HOST myuser conver
[root@ns1-bjcnc postal-0.61]# ./do.sh
time,messages,data(K),errors,connections,SSL connections
.
14:38,4213,3793,10,2176,0
14:39,4308,3869,0,2174,0
14:40,4296,3835,0,2160,0
14:41,4261,3850,0,2136,0
14:42,4342,3930,0,2168,0
14:43,4258,3843,0,2139,0
14:44,4172,3769,0,2067,0
14:45,4111,3684,0,2052,0
14:46,3920,3577,0,1961,0
14:47,3987,3590,0,1987,0
14:48,3729,3365,0,1851,0
14:49,3702,3318,0,1842,0
14:50,3745,3371,0,1879,0
14:51,3961,3558,0,1969,0
14:52,3753,3415,0,1876,0
14:53,3498,3139,0,1781,0
14:54,3337,3001,0,1661,0
14:55,2671,2398,0,1341,0
14:56,2722,2453,0,1365,0
14:57,2648,2388,0,1342,0
14:58,2466,2199,7,1227,0
14:59,2601,2344,0,1299,0
通过上述多次测试表明，在打开proxymap daemon后，SMTP注入速度明显提高，结果是提升了10%-15%。
接下来将virtual投递并发限制
[root@ns1-bjcnc postal-0.61]# ./do.sh
time,messages,data(K),errors,connections,SSL connections
20:44,4289,3902,0,2166,0
20:45,4329,3874,0,2169,0
Server error:451 : Temporary lookup failure
.
期间产生12个Temporary lookup failure
20:46,3993,3594,13,1983,0
20:47,4422,4018,0,2216,0
20:48,4639,4163,0,2323,0
以上部分是final_destination 限制在20个并发virtual
========================================
接着改成10个，速度提升。说明bottleneck仍然在disk i/o（投递多了就明显影响i/o）ldap仍然有潜力可挖掘。
[root@ns1-bjcnc postal-0.61]# ./do.sh
Can't open config file "conver". Doing no expansion.
time,messages,data(K),errors,connections,SSL connections
20:53,5172,4690,0,2636,0
20:54,5074,4530,0,2529,0
20:55,4988,4529,0,2494,0
20:56,4856,4376,0,2456,0
20:57,5114,4602,0,2530,0
20:58,4720,4251,0,2345,0
20:59,4490,4038,0,2230,0
21:00,4716,4234,0,2363,0
21:01,4355,3931,0,2188,0
21:02,4488,4046,0,2241,0
21:03,4495,4062,0,2271,0
21:04,4680,4224,0,2363,0
21:05,4295,3882,0,2141,0
21:06,3822,3436,0,1921,0
21:07,2899,2597,0,1425,0
21:08,2863,2587,0,1447,0
完整结果如下：
[root@ns1-bjcnc postal-0.61]# ./do.sh
Can't open config file "conver". Doing no expansion.
time,messages,data(K),errors,connections,SSL connections
20:53,5172,4690,0,2636,0
20:54,5074,4530,0,2529,0
20:55,4988,4529,0,2494,0
20:56,4856,4376,0,2456,0
20:57,5114,4602,0,2530,0
20:58,4720,4251,0,2345,0
20:59,4490,4038,0,2230,0
21:00,4716,4234,0,2363,0
21:01,4355,3931,0,2188,0
21:02,4488,4046,0,2241,0
21:03,4495,4062,0,2271,0
21:04,4680,4224,0,2363,0
21:05,4295,3882,0,2141,0
21:06,3822,3436,0,1921,0
21:07,2899,2597,0,1425,0
21:08,2863,2587,0,1447,0
21:09,2887,2600,0,1473,0
21:10,2772,2494,0,1372,0
21:11,2106,1896,0,1033,0
21:12,1188,1079,0,584,0
21:13,1493,1355,0,742,0
21:14,1696,1540,0,828,0
21:15,1860,1677,0,920,0
21:16,1581,1428,0,784,0
21:17,2022,1837,0,1018,0
21:18,1477,1317,0,735,0
21:19,1511,1375,0,760,0
21:20,1877,1691,0,950,0
21:21,1459,1314,0,725,0
21:22,1494,1321,0,740,0
21:23,1526,1386,0,756,0
21:24,1309,1188,0,643,0
21:25,1446,1288,0,751,0
21:26,1396,1252,0,722,0
21:27,1431,1260,0,713,0
21:28,1312,1199,0,663,0
21:29,1290,1149,0,658,0
21:30,1190,1078,0,608,0
21:31,949,864,0,464,0
21:32,969,875,0,484,0
21:33,1210,1095,0,625,0
21:34,1384,1255,0,681,0
21:35,1332,1207,0,645,0
21:36,1018,928,0,503,0
21:37,882,795,0,442,0
21:38,1222,1095,0,613,0
21:39,1171,1051,0,581,0
21:40,1222,1123,0,609,0
21:41,1223,1088,0,606,0
21:42,1220,1086,0,607,0
21:43,1328,1208,0,665,0
21:44,1409,1263,0,719,0
21:45,1488,1349,0,723,0
21:46,1237,1115,0,628,0
21:47,1160,1035,0,586,0
21:48,1270,1146,0,626,0
21:49,1400,1261,0,708,0
到了21:10开始，actvie已接近上限，处理速度开始来不及了。此外，postal里限制了一分钟只发5000封信，所以本次测试的峰值51xx信/分未必可信，应能提高更多。
提高注入速度的新测试
笔者根据在IBM developerWorks上得到的灵感，将Postfix queue的所在的磁盘分区挂接参数(mount)改为noatime，这样实际减少了磁盘的写入操作（noatime表示对于访问操作，不更新文件的atime记录，这可减少磁盘的write操作），提高了5%的性能。
通过elvtune调整了磁盘write操作的延时，可明显提高速度。
注意使用linux的iostat及相关的I/O监视工具。。以下是调整后的数值：
time,messages,data(K),errors,connections,SSL connections
23:57,5325,4804,0,2703,0
23:58,5480,4905,0,2702,0
23:59,5160,4624,0,2573,0
00:00,5211,4689,0,2609,0
00:01,4918,4441,0,2453,0
00:02,4872,4377,0,2395,0
00:03,5039,4564,0,2492,0
00:04,4694,4223,0,2308,0
效果有了明显进展。而再进行多次测试发现，有些时段可达到峰值63xx封/分，估计这因为是Postfix的Multiplex技术及磁盘I/O的cache所带来的收益，multiplex可复用已打开的smtp通道，减少了进程的创建带来的开销。
[root@ns1-bjcnc postal-0.61]# ./do.sh
time,messages,data(K),errors,connections,SSL connections
22:23,6330,5721,0,3213,0
22:24,5592,5038,0,2783,0
22:25,5069,4555,0,2509,0
22:26,4624,4203,0,2333,0
22:27,4778,4311,0,2438,0
22:28,4396,3943,0,2215,0
[root@ns1-bjcnc postal-0.61]# ./do.sh
Can't open config file "conver". Doing no expansion.
time,messages,data(K),errors,connections,SSL connections
22:32,5121,4615,0,2611,0
22:33,4696,4263,0,2352,0
22:34,4532,4094,0,2277,0
22:35,4622,4138,0,2297,0
22:36,5006,4515,0,2493,0
22:37,4487,4052,0,2284,0
22:38,4595,4121,0,2300,0
22:39,4669,4221,0,2300,0
22:40,4381,3961,0,2149,0
22:41,4217,3781,0,2100,0
[root@ns1-bjcnc postal-0.61]# ./do.sh
time,messages,data(K),errors,connections,SSL connections
22:44,5148,4642,0,2629,0
22:45,4868,4390,0,2414,0
22:46,4890,4392,0,2451,0
22:47,4803,4306,0,2419,0
22:48,5005,4552,0,2493,0
22:49,4435,3976,0,2221,0
22:50,4515,4055,0,2246,0
22:51,4666,4187,0,2336,0
22:52,4556,4105,0,2283,0
22:53,4472,3979,0,2239,0
22:54,4136,3749,0,2057,0
22:55,4649,4172,0,2355,0
22:56,4304,3895,0,2142,0
22:57,4055,3607,0,2008,0
22:58,2692,2425,0,1335,0
22:59,2690,2402,0,1346,0
23:00,2796,2533,0,1379,0
23:01,2738,2448,0,1396,0
23:02,2522,2287,0,1252,0
23:03,2574,2337,0,1286,0
[root@ns1-bjcnc postal-0.61]# ./do.sh
time,messages,data(K),errors,connections,SSL connections
23:08,6084,5525,0,3068,0
23:09,5834,5254,0,2912,0
23:10,4683,4205,0,2303,0
23:11,4859,4376,0,2453,0
23:12,4265,3814,0,2142,0
23:13,4595,4179,0,2282,0
23:14,4466,4002,0,2217,0
23:15,4438,3982,0,2201,0
23:16,4463,4033,0,2242,0
23:17,4592,4123,0,2327,0
23:18,4234,3802,0,2097,0
23:19,4545,4049,0,2265,0
23:20,3853,3437,0,1925,0
23:21,2708,2417,0,1349,0
23:22,2730,2452,0,1363,0
[root@ns1-bjcnc postal-0.61]# ./do.sh
Can't open config file "conver". Doing no expansion.
time,messages,data(K),errors,connections,SSL connections
23:42,5327,4852,0,2721,0
23:43,5201,4653,0,2631,0
23:44,5377,4815,0,2705,0
23:45,4899,4431,0,2398,0
23:46,4863,4378,0,2421,0
23:47,4885,4382,0,2460,0
23:48,5012,4523,0,2496,0
23:49,5038,4501,0,2561,0
23:50,4898,4427,0,2431,0
23:51,4888,4400,0,2494,0
23:52,4856,4350,0,2398,0
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
Server error:451 : Temporary lookup failure
.
23:53,4584,4115,4,2306,0
23:54,4693,4243,0,2336,0
接下来，将保存对列的磁盘sda3进行调整:
(queue) r_d w_d from 4096/8192 -> 4096/2048
并清理队列，重新测试，注入速度快了不少。
没调整磁盘R/W延迟之前的结果（r/w -> 8192/4096) ：
time,messages,data(K),errors,connections,SSL connections
00:08,5280,4790,0,2704,0
00:09,5135,4570,0,2546,0
00:10,5280,4810,0,2635,0
00:11,5082,4572,0,2547,0
00:12,5195,4648,0,2596,0
00:13,5053,4591,0,2549,0
00:14,5089,4568,0,2583,0
00:15,4937,4438,0,2485,0
00:16,4727,4312,0,2340,0
00:17,4396,3958,0,2224,0
00:18,4860,4353,0,2413,0
00:19,4451,3994,0,2227,0
00:20,4582,4132,0,2300,0
00:21,2809,2486,0,1439,0
调整为r/w -> 4096/2048，结果连续出现了6次6xxx/分:
[root@ns1-bjcnc postal-0.61]# ./do.sh
time,messages,data(K),errors,connections,SSL connections
00:23,6247,5597,0,3174,0
00:24,6581,5891,0,3280,0
00:25,6544,5898,0,3236,0
00:26,6457,5737,0,3235,0
00:27,6283,5690,0,3137,0
最后结论
LDAP的速度不容质疑，在硬件足够好的前提下，系统的瓶颈一般在磁盘的I/O上，LDAP一般较磁盘I/O要快。而如何调整Postfix的注入并发数、系统总进程数、投递的并发数等，需要大量的测试，找到在具体平台、具体硬件下所能达到的最高效能，继而定出合理的数值。
作为一个繁忙的邮件系统，磁盘的I/O能力，尤其是队列所在的磁盘I/O能力要求非常高。在不增加硬件投入的前提下，善于利用iostat, vmstat及相关I/O分析工具，可以找到磁盘的限制在哪里，利用elvtune调整磁盘的读/写时延(delay)来减少这种限制所带来的性能劣化。
如果有足够的经济实力，可以使用带NVRAM的磁盘系统，使用Ultra 320的磁盘，甚至组成RAID0+1的系统来构成队列。这样可有效的提高速度。有报道称，国外有公司使用Sun的存储设备，在Solaris下实现了sendmail 287封邮件/秒 的处理速度。换言之相当于17220封/分的速度。这得益于它们使用了Sun的高级存储设备，配备了NVRAM的Cache，非常高速的硬盘，相当好的性能调整和优化。
通过这个国外的实例可以说明一点，Sendmail其实性能也可以相当好，关键是看系统的设计和部署人员的能力和经验。
而本文所实现的结果，也反映了Postfix在低端硬件（U160 SCSI磁盘，PIII 933的普通CPU）下效能相当不俗。如果按正常的4000封/分钟的处理速度，那么一天相当于处理576万封普通大小的电子邮件。
