让python调用mongo读写速度加速10倍的方法

1.把mongo读写封装成api

2.在api初始化时保持数据库长链接;并且用线程每2分钟遍历一次所有的表并count一次

import sys
import time
import pymongo
import json
import log
import traceback
import threading

//库名test,表名test_table
server_list = ['test-mongos.all.serv:6365' for i in range(8)]
conn_str = "mongodb://test:123456@" + ",".join(server_list) + "/test"
print conn_str
conn = pymongo.MongoClient(conn_str)
test = conn.test

def load_cache(){
    #load cache
    try:
        table_names_cache = test.collection_names()
        for tb in table_names_cache:
            print tb, test[tb].count()
    except:
        traceback.print_exc()
        log.warn("acl data load failed")

    # threading reflash
    bg_job = threading.Timer(120, load_acl_data, ())  #120 seconds
    bg_job.start()
}

#query
start_time = time.time()
condition = {"id": "abc"}
doc_iter = test['test_table'].find(condition, {"_id": 0}).batch_size(15).limit(1)
docs = list(doc_iter.max_time_ms(200000))    #200000ms
end_time = time.time()
print "use time:" + str(end_time - start_time)

你会发现调这个api的平响会是你直接去查耗时的1/10 – 1/20左右,原理应该就是count请求导致了全表扫描,而全表扫描的结果短时间内被cache,此时查询时用到了内存中的cache所以速度变快。(mongo集群需要有非常充沛的内存)

yan 20170815 01:33

发表评论

电子邮件地址不会被公开。