1.把mongo读写封装成api
2.在api初始化时保持数据库长链接;并且用线程每2分钟遍历一次所有的表并count一次
import sys
import time
import pymongo
import json
import log
import traceback
import threading
//库名test,表名test_table
server_list = ['test-mongos.all.serv:6365' for i in range(8)]
conn_str = "mongodb://test:123456@" + ",".join(server_list) + "/test"
print conn_str
conn = pymongo.MongoClient(conn_str)
test = conn.test
def load_cache(){
#load cache
try:
table_names_cache = test.collection_names()
for tb in table_names_cache:
print tb, test[tb].count()
except:
traceback.print_exc()
log.warn("acl data load failed")
# threading reflash
bg_job = threading.Timer(120, load_acl_data, ()) #120 seconds
bg_job.start()
}
#query
start_time = time.time()
condition = {"id": "abc"}
doc_iter = test['test_table'].find(condition, {"_id": 0}).batch_size(15).limit(1)
docs = list(doc_iter.max_time_ms(200000)) #200000ms
end_time = time.time()
print "use time:" + str(end_time - start_time)
你会发现使用主键或唯一索引进行增删改查时,调这个api的平响会是你直接调原生lib读写耗时的1/10 – 1/20左右(平响20-50ms=>3-5ms),原理应该就是count请求导致了主键和唯一索引短时间内被cache,此时增删改定位数据位置时用到了内存中的索引cache所以速度变快。
(注:mongo集群需要有比较充沛的内存)
yan 20170815 01:33