Python内置marshal, cPickle等序列化库，但cPickle效率不太理想，marshal文档也说不保证版本兼容性。今天在列表中看到几个第三方库，故自己测试下：
测试脚本：
#!/usr/bin/env python
import sys,os,time
import cPickle
import marshal
import shelve
import tnetstring
import msgpack
def get_dict():
d = {}
for i in xrange(500000):
d[i] = "abcd" * 10
return d
import time, functools
def timeit(func):
@functools.wraps(func) #let __do__ has marshal_dump's __name__, __doc__
def __do__(*args,**kwargs):
start = time.time()
result= func(*args,**kwargs)
print '%s used time %ss'%(func.__name__,time.time()-start)
return result
return __do__
@timeit
def marshal_dump(d):
return marshal.dumps(d)
@timeit
def marshal_load(s):
return marshal.loads(s)
def marshal_test(d):
s = marshal_dump(d)
marshal_load(s)
@timeit
def cPickle_dump(d):
return cPickle.dumps(d, cPickle.HIGHEST_PROTOCOL)
@timeit
def cPickle_load(s):
return cPickle.loads(s)
def cPickle_test(d):
s = cPickle_dump(d)
cPickle_load(s)
@timeit
def tnetstring_dump(d):
return tnetstring.dumps(d)
@timeit
def tnetstring_load(s):
return tnetstring.loads(s)
def tnetstring_test(d):
s = tnetstring_dump(d)
tnetstring_load(s)
@timeit
def msgpack_dump(d):
return msgpack.packb(d)
@timeit
def msgpack_load(s):
return msgpack.unpackb(s)
def msgpack_test(d):
s= msgpack_dump(d)
msgpack_load(s)
def main():
d = get_dict()
marshal_test(d)
cPickle_test(d)
tnetstring_test(d)
msgpack_test(d)
if __name__ == "__main__":
main()
在一台Ubuntu 10.04 32bit，双核Intel(R) Xeon(TM) CPU 3.06GHz机器上，有如下成绩：
marshal_dump used time 0.18213891983s
marshal_load used time 0.335217952728s
cPickle_dump used time 2.73869895935s
cPickle_load used time 0.558264017105s
tnetstring_dump used time 0.622045040131s
tnetstring_load used time 0.311910152435s
msgpack_dump used time 1.07127404213s
msgpack_load used time 0.27454996109s
结论
-
cPickle最慢，基本上可以抛弃
-
marshal虽然有不兼容隐患，但如果能保证相同的Python版本，还是可以一用
-
tnetstring无论序列化或反序列化速度都比较中庸
-
msgpack反序列化最快，如果读多写少的应用场景是最佳选择。而且msgpack支持多语言。
在Intel Core i5 2.4GHz，8G 内存上，
python2.7.9发现cPickle比marshal的dump快，load稍慢,
marshal_dump used time 0.0894310474396s
marshal_load used time 0.110545158386s
cPickle_dump used time 0.0701198577881s
cPickle_load used time 0.133043050766s
python3 中pickle和cPickle合并（import pickle as cPickle），而性能很差：
marshal_dump used time 0.09095096588134766s
marshal_load used time 0.13521504402160645s
cPickle_dump used time 0.2016129493713379s
cPickle_load used time 0.15890908241271973s
