mongodb在建立一个T级别的数据库时,进程挂掉的如何解决方法

有那么一个人出现在你的生命里,其他人都会变成风景,只有那个人是他心口的朱砂,放在心尖上的人,再也放不下。

mongod进程退出, 退出时数据库中记录大概1.2亿,数据库小大600G

错误信息如下:

Sat May 12 08:37:18 Invalid access at address: 0x7ef6036a9000

Sat May 12 08:37:19 Got signal: 7 (Bus error).

Sat May 12 08:37:19 [conn4878] moveChunk data transfer progress: { active: true, ns: "genbank.main", from: "shard1/192.168.136.72:29022,192.168.136.73:29023,192.168.136.79:29021", min: { _id: ObjectId('4fabda86be960438ed0099d2') }, max: { _id: ObjectId('4fabdaa1be960439500040b1') }, state: "clone", counts: { cloned: 8138, clonedBytes: 50270960, catchup: 0, steady: 0 }, ok: 1.0 } my mem used: 0
Sat May 12 08:37:19 Backtrace:
0xa90999 0xa90f70 0x7f3c7ffcc060 0x7f3c7f5b3873 0xa48342 0xa48c76 0x97cd04 0x97e2ff 0x940ca5 0x944574 0x8866b7 0x88dc29 0xaa33f6 0x637407 0x7f3c7ffc3efc 0x7f3c7f55d89d
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo10abruptQuitEi+0x399) [0xa90999]
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo24abruptQuitWithAddrSignalEiP7siginfoPv+0x220) [0xa90f70]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x10060) [0x7f3c7ffcc060]
 /lib/x86_64-linux-gnu/libc.so.6(+0x137873) [0x7f3c7f5b3873]
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo17MigrateFromStatus5cloneERSsRNS_14BSONObjBuilderE+0x3d2) [0xa48342]
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo19InitialCloneCommand3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x16) [0xa48c76]
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo11execCommandEPNS_7CommandERNS_6ClientEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x6a4) [0x97cd04]
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo12_runCommandsEPKcRNS_7BSONObjERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x6ff) [0x97e2ff]
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo11runCommandsEPKcRNS_7BSONObjERNS_5CurOpERNS_11_BufBuilderINS_16TrivialAllocatorEEERNS_14BSONObjBuilderEbi+0x35) [0x940ca5]
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo8runQueryERNS_7MessageERNS_12QueryMessageERNS_5CurOpES1_+0x1724) [0x944574]
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod() [0x8866b7]
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo16assembleResponseERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0x559) [0x88dc29]
/usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x76) [0xaa33f6]
 /usr/local/mongodb-linux-x86_64-2.0.4/bin/mongod(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x287) [0x637407]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x7efc) [0x7f3c7ffc3efc]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f3c7f55d89d]

Logstream::get called in uninitialized state
Sat May 12 08:37:19 ERROR: Client::~Client _context should be null but is not; client:conn
Logstream::get called in uninitialized state
Sat May 12 08:37:19 ERROR: Client::shutdown not called: conn

问题已经解决。有用机器是numa架构,因此在启动mongod进程的时候加上:numactl --interleave=all
mongodb官网上有如下说明,可供参考:
Linux, NUMA and MongoDB tend not to work well together. If you are running MongoDB on numa hardware, we recommend turning it off (running with an interleave memory policy). Problems will manifest in strange ways, such as massive slow downs for periods of time or high system cpu time.

numactl
To turn off NUMA, start mongod with

numactl --interleave=all ${MONGODB_HOME}/bin/mongod --config conf/mongodb.conf

proc settings

echo 0 > /proc/sys/vm/zone_reclaim_mode

You can change zone_reclaim_mode without restarting mongod. For more information on this setting seehttp://www.kernel.org/doc/Documentation/sysctl/vm.txt.

本文mongodb在建立一个T级别的数据库时,进程挂掉的如何解决方法到此结束。过去的习惯,决定今日的你,所以,过去的懒惰,决定你今日的一败涂地。小编再次感谢大家对我们的支持!

您可能有感兴趣的文章
mongos崩溃后无法重启问题的如何解决方法

MongoDB游标超时问题的4种如何解决方法

MongoDB磁盘IO问题的3种如何解决方法

MongoDB最大连接数设置失效的异常分析过程与如何解决方法

MongoDB诡异问题之sh.stopBalancer卡住的如何解决方法