2014/10/25 14:46:36.253271 [ 1370]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/25 14:46:36.254020 [ 1370]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/25 14:46:36.254038 [ 1370]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/25 16:20:44.553577 [ 1404]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/25 16:20:44.553694 [ 1404]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/25 16:20:44.553707 [ 1404]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/25 17:48:34.171983 [ 1582]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/25 17:48:34.172075 [ 1582]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/25 17:48:34.172088 [ 1582]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/25 18:19:11.715748 [ 1559]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/25 18:19:11.715876 [ 1559]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/25 18:19:11.715890 [ 1559]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/25 20:38:12.602655 [ 1357]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/25 20:38:12.602748 [ 1357]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/25 20:38:12.602762 [ 1357]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/27 11:09:01.636613 [ 1366]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/27 11:09:01.636716 [ 1366]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/27 11:09:01.636730 [ 1366]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/27 11:41:25.818698 [ 1376]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/27 11:41:25.818793 [ 1376]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/27 11:41:25.818807 [ 1376]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/27 14:33:48.484936 [ 8746]: Starting CTDBD (Version 2.5.3) as PID: 8746 2014/10/27 14:33:48.757756 [ 8746]: Freeze priority 1 2014/10/27 14:33:48.775278 [ 8746]: Freeze priority 2 2014/10/27 14:33:48.775676 [ 8746]: Freeze priority 3 2014/10/27 14:33:48.803510 [ 8746]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/27 14:33:48.807169 [ 8746]: 00.ctdb: Set RecoverTimeout to 60 2014/10/27 14:33:48.810707 [ 8746]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/27 14:33:48.933404 [ 8746]: Freeze priority 1 2014/10/27 14:33:48.933477 [ 8746]: Freeze priority 2 2014/10/27 14:33:48.933531 [ 8746]: Freeze priority 3 2014/10/27 14:33:49.306475 [ 8746]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/27 14:33:52.438828 [recoverd: 8928]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/27 14:33:52.438926 [ 8746]: Freeze priority 1 2014/10/27 14:33:52.439000 [ 8746]: Freeze priority 2 2014/10/27 14:33:52.439065 [ 8746]: Freeze priority 3 2014/10/27 14:33:57.664734 [ 8746]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/27 14:33:57.680496 [ 8746]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/27 14:33:57.696523 [ 8746]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/27 14:33:57.711980 [ 8746]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/27 14:33:57.727396 [ 8746]: Vacuuming is disabled for persistent database registry.tdb 2014/10/27 14:33:57.760824 [ 8746]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/27 14:33:57.778460 [ 8746]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/27 14:33:57.788110 [ 8746]: Freeze priority 1 2014/10/27 14:33:57.789290 [ 8746]: Freeze priority 2 2014/10/27 14:33:57.790215 [ 8746]: Freeze priority 3 2014/10/27 14:33:57.950336 [ 8746]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/27 14:33:59.296884 [ 8746]: Thawing priority 1 2014/10/27 14:33:59.296919 [ 8746]: Release freeze handler for prio 1 2014/10/27 14:33:59.296961 [ 8746]: Thawing priority 2 2014/10/27 14:33:59.296981 [ 8746]: Release freeze handler for prio 2 2014/10/27 14:33:59.297009 [ 8746]: Thawing priority 3 2014/10/27 14:33:59.297027 [ 8746]: Release freeze handler for prio 3 2014/10/27 14:34:17.186801 [ 8746]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/27 14:34:18.419310 [ 8746]: Freeze priority 1 2014/10/27 14:34:18.420561 [ 8746]: Freeze priority 2 2014/10/27 14:34:18.421533 [ 8746]: Freeze priority 3 2014/10/27 14:35:08.978918 [ 8746]: DB Attach to database ctdb.tdb deferred for client with pid:11885 since node is in recovery mode. 2014/10/27 14:35:13.299547 [ 8746]: Event script '00.ctdb startup ' timed out after 60.0s, count: 0, pid: 10371 2014/10/27 14:35:13.299605 [ 8746]: startup event failed 2014/10/27 14:35:15.847649 [ 8746]: pnn 2 Invalid reqid 107 in ctdb_reply_control 2014/10/27 14:35:15.847688 [ 8746]: pnn 2 Invalid reqid 117 in ctdb_reply_control 2014/10/27 14:35:18.300311 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:35:18.300362 [ 8746]: Unable to launch startup event script 2014/10/27 14:35:23.300633 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:35:23.300687 [ 8746]: Unable to launch startup event script 2014/10/27 14:35:27.134015 [ 8746]: pnn 2 Invalid reqid 135 in ctdb_reply_control 2014/10/27 14:35:28.301700 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:35:28.301738 [ 8746]: Unable to launch startup event script 2014/10/27 14:35:33.301813 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:35:33.301862 [ 8746]: Unable to launch startup event script 2014/10/27 14:35:38.302097 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:35:38.302148 [ 8746]: Unable to launch startup event script 2014/10/27 14:35:43.303245 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:35:43.303289 [ 8746]: Unable to launch startup event script 2014/10/27 14:35:46.670637 [ 8746]: pnn 2 Invalid reqid 157 in ctdb_reply_control 2014/10/27 14:35:48.303745 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:35:48.303787 [ 8746]: Unable to launch startup event script 2014/10/27 14:35:53.304773 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:35:53.304837 [ 8746]: Unable to launch startup event script 2014/10/27 14:35:58.305306 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:35:58.305350 [ 8746]: Unable to launch startup event script 2014/10/27 14:36:03.305562 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:36:03.305599 [ 8746]: Unable to launch startup event script 2014/10/27 14:36:05.617439 [ 8746]: Freeze priority 1 2014/10/27 14:36:05.618316 [ 8746]: Freeze priority 2 2014/10/27 14:36:05.619245 [ 8746]: Freeze priority 3 2014/10/27 14:36:08.306461 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:36:08.306511 [ 8746]: Unable to launch startup event script 2014/10/27 14:36:13.300559 [ 8746]: ctdb_kill: trying to kill(10371, 9) a process that does not exist 2014/10/27 14:36:13.306683 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:36:13.306723 [ 8746]: Unable to launch startup event script 2014/10/27 14:36:18.307569 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:36:18.307622 [ 8746]: Unable to launch startup event script 2014/10/27 14:36:19.312060 [recoverd: 8928]: ctdb_control error: 'node is disconnected' 2014/10/27 14:36:19.312100 [recoverd: 8928]: ctdb_control error: 'node is disconnected' 2014/10/27 14:36:19.312135 [recoverd: 8928]: Async operation failed with ret=-1 res=-1 opcode=80 2014/10/27 14:36:19.312149 [recoverd: 8928]: Async wait failed - fail_count=1 2014/10/27 14:36:19.312162 [recoverd: 8928]: server/ctdb_recoverd.c:345 Failed to read node capabilities. 2014/10/27 14:36:19.312208 [recoverd: 8928]: server/ctdb_recoverd.c:3678 Unable to update node capabilities. 2014/10/27 14:36:23.308311 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:36:23.308359 [ 8746]: Unable to launch startup event script 2014/10/27 14:36:28.308846 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:36:28.308900 [ 8746]: Unable to launch startup event script 2014/10/27 14:36:33.309870 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:36:33.309911 [ 8746]: Unable to launch startup event script 2014/10/27 14:36:38.310508 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:36:38.310550 [ 8746]: Unable to launch startup event script 2014/10/27 14:36:43.311223 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:36:43.311266 [ 8746]: Unable to launch startup event script 2014/10/27 14:36:44.999377 [ 8746]: pnn 2 Invalid reqid 183 in ctdb_reply_control 2014/10/27 14:36:48.312297 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:36:48.312338 [ 8746]: Unable to launch startup event script 2014/10/27 14:36:53.313194 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:36:53.313231 [ 8746]: Unable to launch startup event script 2014/10/27 14:36:58.314054 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:36:58.314094 [ 8746]: Unable to launch startup event script 2014/10/27 14:37:03.314426 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:37:03.314463 [ 8746]: Unable to launch startup event script 2014/10/27 14:37:05.621025 [ 8746]: Event script '01.reclock startrecovery ' timed out after 10.2s, count: 0, pid: 15118 2014/10/27 14:37:05.621067 [ 8746]: Ignoring hung script for call 3 2014/10/27 14:37:05.637662 [ 8746]: Freeze priority 1 2014/10/27 14:37:05.638022 [ 8746]: Freeze priority 2 2014/10/27 14:37:05.638354 [ 8746]: Freeze priority 3 2014/10/27 14:37:08.315363 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:37:08.315406 [ 8746]: Unable to launch startup event script 2014/10/27 14:37:13.315531 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:37:13.315581 [ 8746]: Unable to launch startup event script 2014/10/27 14:37:18.316524 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:37:18.316565 [ 8746]: Unable to launch startup event script 2014/10/27 14:37:23.316954 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:37:23.317007 [ 8746]: Unable to launch startup event script 2014/10/27 14:37:28.317789 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:37:28.317859 [ 8746]: Unable to launch startup event script 2014/10/27 14:37:33.318425 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:37:33.318461 [ 8746]: Unable to launch startup event script 2014/10/27 14:37:38.319129 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:37:38.319193 [ 8746]: Unable to launch startup event script 2014/10/27 14:37:43.319607 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:37:43.319668 [ 8746]: Unable to launch startup event script 2014/10/27 14:37:47.996188 [ 8746]: Hung-script: ===== Start of hung script debug for PID="15118", event="startrecovery" ===== 2014/10/27 14:37:47.996231 [ 8746]: Hung-script: pstree -p -a 15118: 2014/10/27 14:37:48.016425 [ 8746]: Hung-script: 2014/10/27 14:37:48.018469 [ 8746]: Hung-script: ---- ctdb scriptstatus startrecovery: ---- 2014/10/27 14:37:48.019791 [ 8746]: Hung-script: 2 scripts were executed last startrecovery cycle 2014/10/27 14:37:48.019869 [ 8746]: Hung-script: 00.ctdb Status:OK Duration:49.798 Mon Oct 27 14:36:05 2014 2014/10/27 14:37:48.019903 [ 8746]: Hung-script: 01.reclock Status:OK Duration:-1414391815.419 Mon Oct 27 14:36:55 2014 2014/10/27 14:37:48.020084 [ 8746]: Hung-script: ===== End of hung script debug for PID="15118", event="startrecovery" ===== 2014/10/27 14:37:48.020563 [ 8746]: ctdb_kill: trying to kill(15118, 9) a process that does not exist 2014/10/27 14:37:48.144677 [ 8746]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/27 14:37:48.320081 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:37:48.320124 [ 8746]: Unable to launch startup event script 2014/10/27 14:37:49.272439 [ 8746]: Thawing priority 1 2014/10/27 14:37:49.272496 [ 8746]: Release freeze handler for prio 1 2014/10/27 14:37:49.272539 [ 8746]: Thawing priority 2 2014/10/27 14:37:49.272556 [ 8746]: Release freeze handler for prio 2 2014/10/27 14:37:49.272593 [ 8746]: Thawing priority 3 2014/10/27 14:37:49.272604 [ 8746]: Release freeze handler for prio 3 2014/10/27 14:37:57.358439 [ 8746]: Recovery daemon ping timeout. Count : 0 2014/10/27 14:37:57.359647 [recoverd: 8928]: ctdb_control error: 'ctdb_control timed out' 2014/10/27 14:37:57.359693 [recoverd: 8928]: ctdb_control error: 'ctdb_control timed out' 2014/10/27 14:37:57.359707 [recoverd: 8928]: Async operation failed with ret=-1 res=-1 opcode=80 2014/10/27 14:37:57.359717 [recoverd: 8928]: Async wait failed - fail_count=1 2014/10/27 14:37:57.359727 [recoverd: 8928]: server/ctdb_recoverd.c:345 Failed to read node capabilities. 2014/10/27 14:37:57.359737 [recoverd: 8928]: server/ctdb_recoverd.c:3678 Unable to update node capabilities. 2014/10/27 14:38:14.325504 [recoverd: 8928]: ctdb_control error: 'node is disconnected' 2014/10/27 14:38:14.325562 [recoverd: 8928]: ctdb_control error: 'node is disconnected' 2014/10/27 14:38:14.325586 [recoverd: 8928]: Async operation failed with ret=-1 res=-1 opcode=80 2014/10/27 14:38:14.325599 [recoverd: 8928]: Async wait failed - fail_count=1 2014/10/27 14:38:14.325611 [recoverd: 8928]: server/ctdb_recoverd.c:345 Failed to read node capabilities. 2014/10/27 14:38:14.325624 [recoverd: 8928]: server/ctdb_recoverd.c:3678 Unable to update node capabilities. 2014/10/27 14:38:47.373307 [ 8746]: pnn 2 Invalid reqid 553 in ctdb_reply_control 2014/10/27 14:38:53.320762 [ 8746]: Event script '01.reclock startup ' timed out after 19.7s, count: 0, pid: 18127 2014/10/27 14:38:53.320814 [ 8746]: startup event failed 2014/10/27 14:39:07.211179 [ 8746]: Event script '00.ctdb ipreallocated ' timed out after 60.0s, count: 0, pid: 17319 2014/10/27 14:39:07.211234 [ 8746]: "ipreallocated" event script failed (status -62) 2014/10/27 14:39:07.211247 [ 8746]: Banning this node for 30 seconds 2014/10/27 14:39:07.211259 [ 8746]: Freeze priority 1 2014/10/27 14:39:07.211395 [ 8746]: Freeze priority 2 2014/10/27 14:39:07.211578 [ 8746]: Freeze priority 3 2014/10/27 14:39:27.374013 [ 8746]: Event script '00.ctdb recovered ' timed out after 60.0s, count: 0, pid: 17950 2014/10/27 14:39:27.374058 [ 8746]: Ignoring hung script for call 4 2014/10/27 14:39:37.211760 [ 8746]: Banning timedout 2014/10/27 14:39:45.530034 [ 8746]: Hung-script: ===== Start of hung script debug for PID="18127", event="startup" ===== 2014/10/27 14:39:45.530082 [ 8746]: Hung-script: pstree -p -a 18127: 2014/10/27 14:39:45.531213 [ 8746]: DB Attach to database ctdb.tdb deferred for client with pid:20292 since node is in recovery mode. 2014/10/27 14:39:45.551972 [ 8746]: Hung-script: 2014/10/27 14:39:45.553712 [ 8746]: Hung-script: ---- ctdb scriptstatus startup: ---- 2014/10/27 14:39:45.554805 [ 8746]: Hung-script: 2 scripts were executed last startup cycle 2014/10/27 14:39:45.554860 [ 8746]: Hung-script: 00.ctdb Status:OK Duration:40.325 Mon Oct 27 14:37:53 2014 2014/10/27 14:39:45.554873 [ 8746]: Hung-script: 01.reclock Status:TIMEDOUT Mon Oct 27 14:38:33 2014 2014/10/27 14:39:45.554882 [ 8746]: Hung-script: OUTPUT: 2014/10/27 14:39:45.555015 [ 8746]: Hung-script: ===== End of hung script debug for PID="18127", event="startup" ===== 2014/10/27 14:39:45.555469 [ 8746]: ctdb_kill: trying to kill(18127, 9) a process that does not exist 2014/10/27 14:39:58.321988 [ 8746]: Event script '00.ctdb startup ' timed out after 60.0s, count: 0, pid: 18798 2014/10/27 14:39:58.322028 [ 8746]: startup event failed 2014/10/27 14:40:03.322925 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:40:03.322964 [ 8746]: Unable to launch startup event script 2014/10/27 14:40:04.489585 [ 8746]: Hung-script: ===== Start of hung script debug for PID="17950", event="recovered" ===== 2014/10/27 14:40:04.489632 [ 8746]: Hung-script: pstree -p -a 17950: 2014/10/27 14:40:04.509927 [ 8746]: Hung-script: 2014/10/27 14:40:04.511773 [ 8746]: Hung-script: ---- ctdb scriptstatus recovered: ---- 2014/10/27 14:40:04.512873 [ 8746]: Hung-script: 1 scripts were executed last recovered cycle 2014/10/27 14:40:04.512972 [ 8746]: Hung-script: 00.ctdb Status:OK Duration:-1414391907.374 Mon Oct 27 14:38:27 2014 2014/10/27 14:40:04.513138 [ 8746]: Hung-script: ===== End of hung script debug for PID="17950", event="recovered" ===== 2014/10/27 14:40:04.513529 [ 8746]: ctdb_kill: trying to kill(17950, 9) a process that does not exist 2014/10/27 14:40:07.212833 [ 8746]: ctdb_kill: trying to kill(17319, 9) a process that does not exist 2014/10/27 14:40:08.323350 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:40:08.323389 [ 8746]: Unable to launch startup event script 2014/10/27 14:40:13.323485 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:40:13.323526 [ 8746]: Unable to launch startup event script 2014/10/27 14:40:17.408840 [ 8746]: Freeze priority 1 2014/10/27 14:40:17.409180 [ 8746]: Freeze priority 2 2014/10/27 14:40:17.409499 [ 8746]: Freeze priority 3 2014/10/27 14:40:18.324194 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:40:18.324242 [ 8746]: Unable to launch startup event script 2014/10/27 14:40:23.324713 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:40:23.324754 [ 8746]: Unable to launch startup event script 2014/10/27 14:40:28.325330 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:40:28.325383 [ 8746]: Unable to launch startup event script 2014/10/27 14:40:29.341740 [recoverd: 8928]: ctdb_control error: 'node is disconnected' 2014/10/27 14:40:29.341784 [recoverd: 8928]: ctdb_control error: 'node is disconnected' 2014/10/27 14:40:29.341833 [recoverd: 8928]: Async operation failed with ret=-1 res=-1 opcode=80 2014/10/27 14:40:29.341853 [recoverd: 8928]: Async wait failed - fail_count=1 2014/10/27 14:40:29.341885 [recoverd: 8928]: server/ctdb_recoverd.c:345 Failed to read node capabilities. 2014/10/27 14:40:29.341897 [recoverd: 8928]: server/ctdb_recoverd.c:3678 Unable to update node capabilities. 2014/10/27 14:40:33.325580 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:40:33.325619 [ 8746]: Unable to launch startup event script 2014/10/27 14:40:36.370006 [ 8746]: Hung-script: ===== Start of hung script debug for PID="18798", event="startup" ===== 2014/10/27 14:40:36.370073 [ 8746]: Hung-script: pstree -p -a 18798: 2014/10/27 14:40:36.389563 [ 8746]: Hung-script: ctdb_event_help,18798 47 44 /etc/ctdb/events.d/00.ctdb startup 2014/10/27 14:40:36.389600 [ 8746]: Hung-script: `-00.ctdb,18799 /etc/ctdb/events.d/00.ctdb startup 2014/10/27 14:40:36.389620 [ 8746]: Hung-script: `-ctdb,20292 attach ctdb.tdb persistent 2014/10/27 14:40:36.391450 [ 8746]: Hung-script: ---- ctdb scriptstatus startup: ---- 2014/10/27 14:40:36.392503 [ 8746]: Hung-script: 1 scripts were executed last startup cycle 2014/10/27 14:40:36.392582 [ 8746]: Hung-script: 00.ctdb Status:TIMEDOUT Mon Oct 27 14:38:58 2014 2014/10/27 14:40:36.392612 [ 8746]: Hung-script: OUTPUT: 2014/10/27 14:40:36.392757 [ 8746]: Hung-script: ===== End of hung script debug for PID="18798", event="startup" ===== 2014/10/27 14:40:38.326545 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:40:38.326606 [ 8746]: Unable to launch startup event script 2014/10/27 14:40:43.327570 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:40:43.327629 [ 8746]: Unable to launch startup event script 2014/10/27 14:40:48.328321 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:40:48.328376 [ 8746]: Unable to launch startup event script 2014/10/27 14:40:53.328507 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:40:53.328544 [ 8746]: Unable to launch startup event script 2014/10/27 14:40:55.951831 [ 8746]: pnn 2 Invalid reqid 946 in ctdb_reply_control 2014/10/27 14:40:58.328733 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:40:58.328786 [ 8746]: Unable to launch startup event script 2014/10/27 14:41:03.328929 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:41:03.328984 [ 8746]: Unable to launch startup event script 2014/10/27 14:41:08.329575 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:41:08.329635 [ 8746]: Unable to launch startup event script 2014/10/27 14:41:13.330085 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:41:13.330128 [ 8746]: Unable to launch startup event script 2014/10/27 14:41:17.410721 [ 8746]: Event script '01.reclock startrecovery ' timed out after 19.0s, count: 0, pid: 22264 2014/10/27 14:41:17.410762 [ 8746]: Ignoring hung script for call 3 2014/10/27 14:41:18.331065 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:41:18.331101 [ 8746]: Unable to launch startup event script 2014/10/27 14:41:23.332182 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:41:23.332220 [ 8746]: Unable to launch startup event script 2014/10/27 14:41:28.332477 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:41:28.332514 [ 8746]: Unable to launch startup event script 2014/10/27 14:41:33.332972 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:41:33.333012 [ 8746]: Unable to launch startup event script 2014/10/27 14:41:34.731784 [ 8746]: Hung-script: ===== Start of hung script debug for PID="22264", event="startrecovery" ===== 2014/10/27 14:41:34.731864 [ 8746]: Hung-script: pstree -p -a 22264: 2014/10/27 14:41:34.754781 [ 8746]: Hung-script: ctdb_event_help,22264 43 40 /etc/ctdb/events.d/01.reclock startrecovery 2014/10/27 14:41:34.754834 [ 8746]: Hung-script: `-01.reclock,22265 /etc/ctdb/events.d/01.reclock startrecovery 2014/10/27 14:41:34.756684 [ 8746]: Hung-script: ---- ctdb scriptstatus startrecovery: ---- 2014/10/27 14:41:34.757792 [ 8746]: Hung-script: 2 scripts were executed last startrecovery cycle 2014/10/27 14:41:34.757895 [ 8746]: Hung-script: 00.ctdb Status:OK Duration:41.033 Mon Oct 27 14:40:17 2014 2014/10/27 14:41:34.757930 [ 8746]: Hung-script: 01.reclock Status:OK Duration:-1414392058.443 Mon Oct 27 14:40:58 2014 2014/10/27 14:41:34.758094 [ 8746]: Hung-script: ===== End of hung script debug for PID="22264", event="startrecovery" ===== 2014/10/27 14:41:38.333598 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:41:38.333646 [ 8746]: Unable to launch startup event script 2014/10/27 14:41:43.334282 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:41:43.334326 [ 8746]: Unable to launch startup event script 2014/10/27 14:41:48.334960 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:41:48.335002 [ 8746]: Unable to launch startup event script 2014/10/27 14:41:53.335097 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:41:53.335135 [ 8746]: Unable to launch startup event script 2014/10/27 14:41:58.335444 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:41:58.335494 [ 8746]: Unable to launch startup event script 2014/10/27 14:41:59.473019 [ 8746]: Recovery daemon ping timeout. Count : 0 2014/10/27 14:42:03.336053 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:42:03.336092 [ 8746]: Unable to launch startup event script 2014/10/27 14:42:08.336534 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:42:08.336576 [ 8746]: Unable to launch startup event script 2014/10/27 14:42:08.840861 [recoverd: 8928]: ctdb_control error: 'ctdb_control timed out' 2014/10/27 14:42:08.840924 [recoverd: 8928]: client/ctdb_client.c:1535 ctdb_control for getnodes failed ret:-1 res:-1 2014/10/27 14:42:08.840940 [recoverd: 8928]: server/ctdb_recoverd.c:3739 Unable to get nodemap from recovery master 0 2014/10/27 14:42:13.336880 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:42:13.336918 [ 8746]: Unable to launch startup event script 2014/10/27 14:42:14.354230 [recoverd: 8928]: ctdb_control error: 'node is disconnected' 2014/10/27 14:42:14.354272 [recoverd: 8928]: ctdb_control error: 'node is disconnected' 2014/10/27 14:42:14.354288 [recoverd: 8928]: Async operation failed with ret=-1 res=-1 opcode=80 2014/10/27 14:42:14.354313 [recoverd: 8928]: Async wait failed - fail_count=1 2014/10/27 14:42:14.354333 [recoverd: 8928]: server/ctdb_recoverd.c:345 Failed to read node capabilities. 2014/10/27 14:42:14.354348 [recoverd: 8928]: server/ctdb_recoverd.c:3678 Unable to update node capabilities. 2014/10/27 14:42:17.410733 [ 8746]: server/ctdb_recover.c:562 Been in recovery mode for too long. Dropping all IPS 2014/10/27 14:42:18.337586 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:42:18.337618 [ 8746]: Unable to launch startup event script 2014/10/27 14:42:23.338484 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:42:23.338522 [ 8746]: Unable to launch startup event script 2014/10/27 14:42:27.160216 [ 8746]: Freeze priority 1 2014/10/27 14:42:27.161408 [ 8746]: Freeze priority 2 2014/10/27 14:42:27.162366 [ 8746]: Freeze priority 3 2014/10/27 14:42:28.339564 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:42:28.339609 [ 8746]: Unable to launch startup event script 2014/10/27 14:42:33.340401 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:42:33.340438 [ 8746]: Unable to launch startup event script 2014/10/27 14:42:37.194220 [ 8746]: Freeze priority 1 2014/10/27 14:42:37.195016 [ 8746]: Freeze priority 2 2014/10/27 14:42:37.195707 [ 8746]: Freeze priority 3 2014/10/27 14:42:38.340718 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:42:38.340753 [ 8746]: Unable to launch startup event script 2014/10/27 14:42:43.341599 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:42:43.341638 [ 8746]: Unable to launch startup event script 2014/10/27 14:42:48.341980 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:42:48.342035 [ 8746]: Unable to launch startup event script 2014/10/27 14:42:53.342879 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:42:53.342921 [ 8746]: Unable to launch startup event script 2014/10/27 14:42:58.343809 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:42:58.343859 [ 8746]: Unable to launch startup event script 2014/10/27 14:43:03.344517 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:43:03.344557 [ 8746]: Unable to launch startup event script 2014/10/27 14:43:08.344781 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:43:08.344833 [ 8746]: Unable to launch startup event script 2014/10/27 14:43:13.345570 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:43:13.345607 [ 8746]: Unable to launch startup event script 2014/10/27 14:43:14.359872 [recoverd: 8928]: ctdb_control error: 'node is disconnected' 2014/10/27 14:43:14.359912 [recoverd: 8928]: ctdb_control error: 'node is disconnected' 2014/10/27 14:43:14.359927 [recoverd: 8928]: Async operation failed with ret=-1 res=-1 opcode=80 2014/10/27 14:43:14.359938 [recoverd: 8928]: Async wait failed - fail_count=1 2014/10/27 14:43:14.359949 [recoverd: 8928]: server/ctdb_recoverd.c:345 Failed to read node capabilities. 2014/10/27 14:43:14.359960 [recoverd: 8928]: server/ctdb_recoverd.c:3678 Unable to update node capabilities. 2014/10/27 14:43:18.346502 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:43:18.346540 [ 8746]: Unable to launch startup event script 2014/10/27 14:43:19.360178 [recoverd: 8928]: ctdb_control error: 'node is disconnected' 2014/10/27 14:43:19.360216 [recoverd: 8928]: ctdb_control error: 'node is disconnected' 2014/10/27 14:43:19.360230 [recoverd: 8928]: Async operation failed with ret=-1 res=-1 opcode=80 2014/10/27 14:43:19.360241 [recoverd: 8928]: Async wait failed - fail_count=1 2014/10/27 14:43:19.360250 [recoverd: 8928]: server/ctdb_recoverd.c:345 Failed to read node capabilities. 2014/10/27 14:43:19.360275 [recoverd: 8928]: server/ctdb_recoverd.c:3678 Unable to update node capabilities. 2014/10/27 14:43:19.361452 [ 8746]: Freeze priority 1 2014/10/27 14:43:19.361641 [ 8746]: Freeze priority 2 2014/10/27 14:43:19.361753 [ 8746]: Freeze priority 3 2014/10/27 14:43:22.366203 [recoverd: 8928]: Taking out recovery lock from recovery daemon 2014/10/27 14:43:22.366244 [recoverd: 8928]: Take the recovery lock 2014/10/27 14:43:23.347255 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:43:23.347295 [ 8746]: Unable to launch startup event script 2014/10/27 14:43:28.348177 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:43:28.348215 [ 8746]: Unable to launch startup event script 2014/10/27 14:43:33.348573 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:43:33.348605 [ 8746]: Unable to launch startup event script 2014/10/27 14:43:37.197173 [ 8746]: Event script '00.ctdb startrecovery ' timed out after 60.0s, count: 0, pid: 25254 2014/10/27 14:43:37.197212 [ 8746]: Ignoring hung script for call 3 2014/10/27 14:43:38.349397 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:43:38.349437 [ 8746]: Unable to launch startup event script 2014/10/27 14:43:43.349727 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:43:43.349765 [ 8746]: Unable to launch startup event script 2014/10/27 14:43:45.977333 [ 8746]: pnn 2 Invalid reqid 1267 in ctdb_reply_control 2014/10/27 14:43:48.350156 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:43:48.350191 [ 8746]: Unable to launch startup event script 2014/10/27 14:43:52.092445 [ 8746]: Freeze priority 1 2014/10/27 14:43:52.093804 [ 8746]: Freeze priority 2 2014/10/27 14:43:52.094877 [ 8746]: Freeze priority 3 2014/10/27 14:43:53.350335 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:43:53.350376 [ 8746]: Unable to launch startup event script 2014/10/27 14:43:55.101949 [ 8746]: Freeze priority 1 2014/10/27 14:43:55.102316 [ 8746]: Freeze priority 2 2014/10/27 14:43:55.102751 [ 8746]: Freeze priority 3 2014/10/27 14:43:58.109464 [ 8746]: Freeze priority 1 2014/10/27 14:43:58.110147 [ 8746]: Freeze priority 2 2014/10/27 14:43:58.110693 [ 8746]: Freeze priority 3 2014/10/27 14:43:58.351027 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:43:58.351073 [ 8746]: Unable to launch startup event script 2014/10/27 14:44:03.033628 [ 8746]: High RECLOCK latency 40.667294s for operation recd reclock 2014/10/27 14:44:03.036598 [ 8746]: Hung-script: ===== Start of hung script debug for PID="25254", event="startrecovery" ===== 2014/10/27 14:44:03.036644 [ 8746]: Hung-script: pstree -p -a 25254: 2014/10/27 14:44:03.037394 [ 8746]: Freeze priority 1 2014/10/27 14:44:03.037496 [ 8746]: Freeze priority 2 2014/10/27 14:44:03.037577 [ 8746]: Freeze priority 3 2014/10/27 14:44:03.059250 [ 8746]: Hung-script: 2014/10/27 14:44:03.061101 [ 8746]: Hung-script: ---- ctdb scriptstatus startrecovery: ---- 2014/10/27 14:44:03.062361 [ 8746]: Hung-script: 1 scripts were executed last startrecovery cycle 2014/10/27 14:44:03.062459 [ 8746]: Hung-script: 00.ctdb Status:OK Duration:-1414392157.197 Mon Oct 27 14:42:37 2014 2014/10/27 14:44:03.062670 [ 8746]: Hung-script: ===== End of hung script debug for PID="25254", event="startrecovery" ===== 2014/10/27 14:44:03.063163 [ 8746]: ctdb_kill: trying to kill(25254, 9) a process that does not exist 2014/10/27 14:44:03.351886 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:44:03.351914 [ 8746]: Unable to launch startup event script 2014/10/27 14:44:08.352174 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:44:08.352218 [ 8746]: Unable to launch startup event script 2014/10/27 14:44:13.352731 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:44:13.352779 [ 8746]: Unable to launch startup event script 2014/10/27 14:44:18.353248 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:44:18.353295 [ 8746]: Unable to launch startup event script 2014/10/27 14:44:22.365471 [ 8746]: Recovery daemon ping timeout. Count : 0 2014/10/27 14:44:23.353654 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:44:23.353695 [ 8746]: Unable to launch startup event script 2014/10/27 14:44:28.353847 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:44:28.353904 [ 8746]: Unable to launch startup event script 2014/10/27 14:44:30.727645 [ 8746]: pnn 2 Invalid reqid 1080 in ctdb_reply_control 2014/10/27 14:44:30.727740 [ 8746]: pnn 2 Invalid reqid 1112 in ctdb_reply_control 2014/10/27 14:44:30.727794 [ 8746]: pnn 2 Invalid reqid 1123 in ctdb_reply_control 2014/10/27 14:44:30.727834 [ 8746]: pnn 2 Invalid reqid 1139 in ctdb_reply_control 2014/10/27 14:44:30.727852 [ 8746]: pnn 2 Invalid reqid 1154 in ctdb_reply_control 2014/10/27 14:44:33.354099 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:44:33.354134 [ 8746]: Unable to launch startup event script 2014/10/27 14:44:38.354974 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:44:38.355041 [ 8746]: Unable to launch startup event script 2014/10/27 14:44:43.355657 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:44:43.355707 [ 8746]: Unable to launch startup event script 2014/10/27 14:44:48.355858 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:44:48.355917 [ 8746]: Unable to launch startup event script 2014/10/27 14:44:53.356397 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:44:53.356440 [ 8746]: Unable to launch startup event script 2014/10/27 14:44:58.357058 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:44:58.357111 [ 8746]: Unable to launch startup event script 2014/10/27 14:44:59.380501 [recoverd: 8928]: ctdb_control error: 'node is disconnected' 2014/10/27 14:44:59.380546 [recoverd: 8928]: ctdb_control error: 'node is disconnected' 2014/10/27 14:44:59.380563 [recoverd: 8928]: Async operation failed with ret=-1 res=-1 opcode=52 2014/10/27 14:44:59.380576 [recoverd: 8928]: Async wait failed - fail_count=1 2014/10/27 14:44:59.380588 [recoverd: 8928]: client/ctdb_client.c:3027 Unable to update nodeflags on remote nodes 2014/10/27 14:44:59.380606 [recoverd: 8928]: server/ctdb_recoverd.c:873 Unable to update nodeflags on remote nodes 2014/10/27 14:44:59.380618 [recoverd: 8928]: server/ctdb_recoverd.c:1857 Unable to update flags on all nodes for node 2 2014/10/27 14:44:59.381616 [ 8746]: Freeze priority 1 2014/10/27 14:44:59.381787 [ 8746]: Freeze priority 2 2014/10/27 14:44:59.381985 [ 8746]: Freeze priority 3 2014/10/27 14:45:02.384545 [recoverd: 8928]: Taking out recovery lock from recovery daemon 2014/10/27 14:45:02.384589 [recoverd: 8928]: Take the recovery lock 2014/10/27 14:45:03.358068 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:45:03.358105 [ 8746]: Unable to launch startup event script 2014/10/27 14:45:08.358215 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:45:08.358255 [ 8746]: Unable to launch startup event script 2014/10/27 14:45:13.358454 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:45:13.358499 [ 8746]: Unable to launch startup event script 2014/10/27 14:45:18.359185 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:45:18.359223 [ 8746]: Unable to launch startup event script 2014/10/27 14:45:23.359378 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:45:23.359411 [ 8746]: Unable to launch startup event script 2014/10/27 14:45:28.360026 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:45:28.360078 [ 8746]: Unable to launch startup event script 2014/10/27 14:45:33.361105 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:45:33.361142 [ 8746]: Unable to launch startup event script 2014/10/27 14:45:38.362182 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:45:38.362232 [ 8746]: Unable to launch startup event script 2014/10/27 14:45:43.362338 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:45:43.362407 [ 8746]: Unable to launch startup event script 2014/10/27 14:45:45.068022 [recoverd: 8928]: ctdb_recovery_lock: Failed to get recovery lock on '/mnt/lock/lockfile' 2014/10/27 14:45:45.068077 [recoverd: 8928]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds 2014/10/27 14:45:45.068166 [ 8746]: Banning this node for 30 seconds 2014/10/27 14:45:48.363137 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:45:48.363188 [ 8746]: Unable to launch startup event script 2014/10/27 14:45:53.363291 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:45:53.363333 [ 8746]: Unable to launch startup event script 2014/10/27 14:45:58.363651 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:45:58.363702 [ 8746]: Unable to launch startup event script 2014/10/27 14:46:03.364135 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:46:03.364166 [ 8746]: Unable to launch startup event script 2014/10/27 14:46:08.364364 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:46:08.364417 [ 8746]: Unable to launch startup event script 2014/10/27 14:46:13.365225 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:46:13.365270 [ 8746]: Unable to launch startup event script 2014/10/27 14:46:15.068727 [ 8746]: Banning timedout 2014/10/27 14:46:15.108171 [recoverd: 8928]: Taking out recovery lock from recovery daemon 2014/10/27 14:46:15.108213 [recoverd: 8928]: Take the recovery lock 2014/10/27 14:46:15.110044 [recoverd: 8928]: ctdb_recovery_lock: Failed to get recovery lock on '/mnt/lock/lockfile' 2014/10/27 14:46:15.110078 [recoverd: 8928]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds 2014/10/27 14:46:15.110145 [ 8746]: Banning this node for 30 seconds 2014/10/27 14:46:18.366197 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:46:18.366239 [ 8746]: Unable to launch startup event script 2014/10/27 14:46:23.366844 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:46:23.366893 [ 8746]: Unable to launch startup event script 2014/10/27 14:46:25.268013 [ 8746]: tcp/tcp_connect.c:211Failed to bind socket Cannot assign requested address(99) 2014/10/27 14:46:25.268112 [ 8746]: tcp/tcp_connect.c:211Failed to bind socket Cannot assign requested address(99) 2014/10/27 14:46:28.367972 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:46:28.368009 [ 8746]: Unable to launch startup event script 2014/10/27 14:46:33.368470 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:46:33.368499 [ 8746]: Unable to launch startup event script 2014/10/27 14:46:38.369315 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:46:38.369370 [ 8746]: Unable to launch startup event script 2014/10/27 14:46:43.370283 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:46:43.370333 [ 8746]: Unable to launch startup event script 2014/10/27 14:46:45.110631 [ 8746]: Banning timedout 2014/10/27 14:46:45.150500 [recoverd: 8928]: Taking out recovery lock from recovery daemon 2014/10/27 14:46:45.150541 [recoverd: 8928]: Take the recovery lock 2014/10/27 14:46:48.371229 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:46:48.371273 [ 8746]: Unable to launch startup event script 2014/10/27 14:46:53.371388 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:46:53.371435 [ 8746]: Unable to launch startup event script 2014/10/27 14:46:58.371644 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:46:58.371695 [ 8746]: Unable to launch startup event script 2014/10/27 14:46:59.382834 [ 8746]: server/ctdb_recover.c:562 Been in recovery mode for too long. Dropping all IPS 2014/10/27 14:47:03.372386 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:47:03.372423 [ 8746]: Unable to launch startup event script 2014/10/27 14:47:08.372660 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:47:08.372714 [ 8746]: Unable to launch startup event script 2014/10/27 14:47:09.289023 [recoverd: 8928]: ctdb_recovery_lock: Unable to open /mnt/lock/lockfile - (Transport endpoint is not connected) 2014/10/27 14:47:09.289061 [recoverd: 8928]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds 2014/10/27 14:47:09.289152 [ 8746]: Banning this node for 30 seconds 2014/10/27 14:47:13.373525 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:47:13.373568 [ 8746]: Unable to launch startup event script 2014/10/27 14:47:18.373767 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:47:18.373828 [ 8746]: Unable to launch startup event script 2014/10/27 14:47:23.374751 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:47:23.374793 [ 8746]: Unable to launch startup event script 2014/10/27 14:47:28.375507 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:47:28.375555 [ 8746]: Unable to launch startup event script 2014/10/27 14:47:33.376581 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:47:33.376628 [ 8746]: Unable to launch startup event script 2014/10/27 14:47:38.377172 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:47:38.377236 [ 8746]: Unable to launch startup event script 2014/10/27 14:47:39.289419 [ 8746]: Banning timedout 2014/10/27 14:47:39.336834 [recoverd: 8928]: Taking out recovery lock from recovery daemon 2014/10/27 14:47:39.336883 [recoverd: 8928]: Take the recovery lock 2014/10/27 14:47:39.337112 [recoverd: 8928]: ctdb_recovery_lock: Unable to open /mnt/lock/lockfile - (Transport endpoint is not connected) 2014/10/27 14:47:39.337127 [recoverd: 8928]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds 2014/10/27 14:47:39.337157 [ 8746]: Banning this node for 30 seconds 2014/10/27 14:47:43.377914 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:47:43.377983 [ 8746]: Unable to launch startup event script 2014/10/27 14:47:48.378941 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:47:48.378987 [ 8746]: Unable to launch startup event script 2014/10/27 14:47:53.379370 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:47:53.379410 [ 8746]: Unable to launch startup event script 2014/10/27 14:47:58.379968 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:47:58.380005 [ 8746]: Unable to launch startup event script 2014/10/27 14:48:03.380810 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:48:03.380862 [ 8746]: Unable to launch startup event script 2014/10/27 14:48:08.381875 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:48:08.381921 [ 8746]: Unable to launch startup event script 2014/10/27 14:48:09.338166 [ 8746]: Banning timedout 2014/10/27 14:48:09.381036 [recoverd: 8928]: Taking out recovery lock from recovery daemon 2014/10/27 14:48:09.381066 [recoverd: 8928]: Take the recovery lock 2014/10/27 14:48:09.381290 [recoverd: 8928]: ctdb_recovery_lock: Unable to open /mnt/lock/lockfile - (Transport endpoint is not connected) 2014/10/27 14:48:09.381314 [recoverd: 8928]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds 2014/10/27 14:48:09.381339 [ 8746]: Banning this node for 30 seconds 2014/10/27 14:48:13.382172 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:48:13.382232 [ 8746]: Unable to launch startup event script 2014/10/27 14:48:18.382447 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:48:18.382495 [ 8746]: Unable to launch startup event script 2014/10/27 14:48:23.383002 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:48:23.383046 [ 8746]: Unable to launch startup event script 2014/10/27 14:48:28.383141 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:48:28.383180 [ 8746]: Unable to launch startup event script 2014/10/27 14:48:33.384171 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:48:33.384219 [ 8746]: Unable to launch startup event script 2014/10/27 14:48:38.384919 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:48:38.384976 [ 8746]: Unable to launch startup event script 2014/10/27 14:48:39.382289 [ 8746]: Banning timedout 2014/10/27 14:48:39.423250 [recoverd: 8928]: Taking out recovery lock from recovery daemon 2014/10/27 14:48:39.423280 [recoverd: 8928]: Take the recovery lock 2014/10/27 14:48:39.423557 [recoverd: 8928]: ctdb_recovery_lock: Unable to open /mnt/lock/lockfile - (Transport endpoint is not connected) 2014/10/27 14:48:39.423573 [recoverd: 8928]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds 2014/10/27 14:48:39.423630 [ 8746]: Banning this node for 30 seconds 2014/10/27 14:48:43.385296 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:48:43.385346 [ 8746]: Unable to launch startup event script 2014/10/27 14:48:48.385973 [ 8746]: Refusing to run event scripts call 'startup' while in recovery 2014/10/27 14:48:48.386017 [ 8746]: Unable to launch startup event script 2014/10/27 14:50:43.926813 [ 1649]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/27 14:50:43.926913 [ 1649]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/27 14:50:43.926927 [ 1649]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/27 15:05:41.521877 [ 1686]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/27 15:05:41.521977 [ 1686]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/27 15:05:41.521992 [ 1686]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/27 16:17:47.005708 [ 1701]: Starting CTDBD (Version 2.5.3) as PID: 1701 2014/10/27 16:17:47.729112 [ 1701]: Vacuuming is disabled for persistent database registry.tdb 2014/10/27 16:17:47.744014 [ 1701]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/27 16:17:47.759088 [ 1701]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/27 16:17:47.774176 [ 1701]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/27 16:17:47.788893 [ 1701]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/27 16:17:47.803150 [ 1701]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/27 16:17:47.817911 [ 1701]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/27 16:17:47.817981 [ 1701]: Freeze priority 1 2014/10/27 16:17:47.832442 [ 1701]: Freeze priority 2 2014/10/27 16:17:47.832827 [ 1701]: Freeze priority 3 2014/10/27 16:17:47.922136 [ 1701]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/27 16:17:47.926218 [ 1701]: 00.ctdb: Set RecoverTimeout to 60 2014/10/27 16:17:47.929747 [ 1701]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/27 16:17:48.073929 [ 1701]: Freeze priority 1 2014/10/27 16:17:48.074022 [ 1701]: Freeze priority 2 2014/10/27 16:17:48.074079 [ 1701]: Freeze priority 3 2014/10/27 16:17:51.578941 [recoverd: 1929]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/27 16:17:51.579031 [ 1701]: Freeze priority 1 2014/10/27 16:17:51.579095 [ 1701]: Freeze priority 2 2014/10/27 16:17:51.579149 [ 1701]: Freeze priority 3 2014/10/27 16:17:55.078318 [ 1701]: Freeze priority 1 2014/10/27 16:17:55.079577 [ 1701]: Freeze priority 2 2014/10/27 16:17:55.080468 [ 1701]: Freeze priority 3 2014/10/27 16:18:55.095036 [ 1701]: Freeze priority 1 2014/10/27 16:18:55.095408 [ 1701]: Freeze priority 2 2014/10/27 16:18:55.095945 [ 1701]: Freeze priority 3 2014/10/27 16:18:59.902815 [ 1701]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/27 16:19:00.120810 [ 1701]: Freeze priority 1 2014/10/27 16:19:00.121516 [ 1701]: Freeze priority 2 2014/10/27 16:19:00.122113 [ 1701]: Freeze priority 3 2014/10/27 16:19:00.267888 [ 1701]: server/ctdb_freeze.c:344 recovery transaction cancelled called 2014/10/27 16:19:01.179939 [ 1701]: Freeze priority 1 2014/10/27 16:19:01.180091 [ 1701]: Freeze priority 2 2014/10/27 16:19:01.180195 [ 1701]: Freeze priority 3 2014/10/27 16:19:04.185123 [recoverd: 1929]: Taking out recovery lock from recovery daemon 2014/10/27 16:19:04.185181 [recoverd: 1929]: Take the recovery lock 2014/10/27 16:19:04.189852 [ 1701]: Freeze priority 1 2014/10/27 16:19:04.189923 [ 1701]: Freeze priority 2 2014/10/27 16:19:04.189971 [ 1701]: Freeze priority 3 2014/10/27 16:19:05.462859 [ 1701]: Thawing priority 1 2014/10/27 16:19:05.462919 [ 1701]: Release freeze handler for prio 1 2014/10/27 16:19:05.462948 [ 1701]: Thawing priority 2 2014/10/27 16:19:05.462960 [ 1701]: Release freeze handler for prio 2 2014/10/27 16:19:05.462979 [ 1701]: Thawing priority 3 2014/10/27 16:19:05.462990 [ 1701]: Release freeze handler for prio 3 2014/10/27 16:19:05.797116 [recoverd: 1929]: Resetting ban count to 0 for all nodes 2014/10/27 16:19:19.767718 [ 1701]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/27 16:19:19.812846 [recoverd: 1929]: Trigger takeoverrun 2014/10/27 16:19:20.056781 [ 1701]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/27 16:19:20.068021 [ 1701]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/27 16:19:20.089715 [ 1701]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/27 16:19:20.284702 [ 1701]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/27 16:19:22.616634 [ 1701]: Node became HEALTHY. Ask recovery master 2 to perform ip reallocation 2014/10/27 16:19:22.816971 [recoverd: 1929]: Public IP '10.10.10.184' is not assigned and we could serve it 2014/10/27 16:19:22.817032 [recoverd: 1929]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/27 16:19:22.817043 [recoverd: 1929]: Public IP '10.10.10.182' is not assigned and we could serve it 2014/10/27 16:19:22.817123 [recoverd: 1929]: Trigger takeoverrun 2014/10/27 16:19:23.136642 [ 1701]: 60.nfs: Reconfiguring service "nfs"... 2014/10/27 16:19:31.153481 [ 1701]: Freeze priority 1 2014/10/27 16:19:31.154651 [ 1701]: Freeze priority 2 2014/10/27 16:19:31.155601 [ 1701]: Freeze priority 3 2014/10/27 16:19:34.171634 [ 1701]: Freeze priority 1 2014/10/27 16:19:34.171931 [ 1701]: Freeze priority 2 2014/10/27 16:19:34.172190 [ 1701]: Freeze priority 3 2014/10/27 16:19:35.433529 [ 1701]: Thawing priority 1 2014/10/27 16:19:35.433605 [ 1701]: Release freeze handler for prio 1 2014/10/27 16:19:35.433651 [ 1701]: Thawing priority 2 2014/10/27 16:19:35.433669 [ 1701]: Release freeze handler for prio 2 2014/10/27 16:19:35.433695 [ 1701]: Thawing priority 3 2014/10/27 16:19:35.433710 [ 1701]: Release freeze handler for prio 3 2014/10/27 16:19:35.933309 [ 1701]: 60.nfs: Reconfiguring service "nfs"... 2014/10/27 16:19:57.391601 [ 1701]: Freeze priority 1 2014/10/27 16:19:57.392859 [ 1701]: Freeze priority 2 2014/10/27 16:19:57.393904 [ 1701]: Freeze priority 3 2014/10/27 16:19:57.395119 [ 1701]: Monitoring event was cancelled 2014/10/27 16:19:57.395162 [ 1701]: server/eventscript.c:569 Sending SIGTERM to child pid:9150 2014/10/27 16:20:19.795485 [ 1701]: pnn 2 Invalid reqid 2224 in ctdb_reply_control 2014/10/27 16:20:19.795614 [ 1701]: pnn 2 Invalid reqid 2238 in ctdb_reply_control 2014/10/27 16:20:43.251237 [ 1701]: pnn 2 Invalid reqid 2248 in ctdb_reply_control 2014/10/27 16:20:59.603323 [ 1701]: pnn 2 Invalid reqid 2268 in ctdb_reply_control 2014/10/27 16:20:59.603442 [ 1701]: pnn 2 Invalid reqid 2295 in ctdb_reply_control 2014/10/27 16:21:43.252883 [ 1701]: Recovery daemon ping timeout. Count : 0 2014/10/27 16:21:57.395393 [ 1701]: server/ctdb_recover.c:562 Been in recovery mode for too long. Dropping all IPS 2014/10/27 16:21:58.099995 [recoverd: 1929]: ctdb_control error: 'node is disconnected' 2014/10/27 16:21:58.100135 [recoverd: 1929]: client/ctdb_client.c:1535 ctdb_control for getnodes failed ret:-1 res:-1 2014/10/27 16:21:58.100154 [recoverd: 1929]: server/ctdb_recoverd.c:1183 Unable to get nodemap from remote node 1 2014/10/27 16:21:58.100169 [recoverd: 1929]: Unable to update local flags 2014/10/27 16:21:58.102467 [recoverd: 1929]: Taking out recovery lock from recovery daemon 2014/10/27 16:21:58.102501 [recoverd: 1929]: Take the recovery lock 2014/10/27 16:22:27.003298 [ 1701]: High RECLOCK latency 28.900705s for operation recd reclock 2014/10/27 16:22:27.009836 [ 1701]: Freeze priority 1 2014/10/27 16:22:27.010121 [ 1701]: Freeze priority 2 2014/10/27 16:22:27.010331 [ 1701]: Freeze priority 3 2014/10/27 16:22:27.041138 [ 1701]: 10.interface: Re-adding secondary address 10.10.10.182/24 to dev bond1 2014/10/27 16:22:58.100362 [ 1701]: Recovery daemon ping timeout. Count : 0 2014/10/27 16:23:27.192843 [recoverd: 1929]: client/ctdb_client.c:1015 control timed out. reqid:1572 opcode:52 dstnode:1 2014/10/27 16:23:27.192928 [recoverd: 1929]: client/ctdb_client.c:1126 ctdb_control_recv failed 2014/10/27 16:23:27.192941 [recoverd: 1929]: Async operation failed with state 3, opcode:52 2014/10/27 16:23:27.192958 [recoverd: 1929]: Async wait failed - fail_count=1 2014/10/27 16:23:27.192968 [recoverd: 1929]: client/ctdb_client.c:3027 Unable to update nodeflags on remote nodes 2014/10/27 16:23:27.192977 [recoverd: 1929]: server/ctdb_recoverd.c:873 Unable to update nodeflags on remote nodes 2014/10/27 16:23:27.193000 [recoverd: 1929]: server/ctdb_recoverd.c:1857 Unable to update flags on all nodes for node 0 2014/10/27 16:23:27.193039 [recoverd: 1929]: client/ctdb_client.c:962 reqid 1572 not found 2014/10/27 16:23:33.109264 [recoverd: 1929]: ctdb_control error: 'node is disconnected' 2014/10/27 16:23:33.109308 [recoverd: 1929]: ctdb_control error: 'node is disconnected' 2014/10/27 16:23:33.109322 [recoverd: 1929]: Async operation failed with ret=-1 res=-1 opcode=80 2014/10/27 16:23:33.109331 [recoverd: 1929]: Async wait failed - fail_count=1 2014/10/27 16:23:33.109340 [recoverd: 1929]: server/ctdb_recoverd.c:345 Failed to read node capabilities. 2014/10/27 16:23:33.109349 [recoverd: 1929]: server/ctdb_recoverd.c:3678 Unable to update node capabilities. 2014/10/27 16:23:33.110597 [recoverd: 1929]: Taking out recovery lock from recovery daemon 2014/10/27 16:23:33.110612 [recoverd: 1929]: Take the recovery lock 2014/10/27 16:23:33.119358 [ 1701]: Freeze priority 1 2014/10/27 16:23:33.119656 [ 1701]: Freeze priority 2 2014/10/27 16:23:33.119911 [ 1701]: Freeze priority 3 2014/10/27 16:23:34.385979 [ 1701]: Thawing priority 1 2014/10/27 16:23:34.386033 [ 1701]: Release freeze handler for prio 1 2014/10/27 16:23:34.386067 [ 1701]: Thawing priority 2 2014/10/27 16:23:34.386096 [ 1701]: Release freeze handler for prio 2 2014/10/27 16:23:34.386120 [ 1701]: Thawing priority 3 2014/10/27 16:23:34.386134 [ 1701]: Release freeze handler for prio 3 2014/10/27 16:23:34.394937 [recoverd: 1929]: Inconsistent IP allocation - node 0 thinks 10.10.10.183 is held by node 0 while it is assigned to node 2 2014/10/27 16:23:34.394984 [recoverd: 1929]: Trigger IP reallocation 2014/10/27 16:23:34.395239 [recoverd: 1929]: Inconsistent IP allocation - node 2 thinks 10.10.10.183 is held by node 0 while it is assigned to node 2 2014/10/27 16:23:34.395268 [recoverd: 1929]: Trigger IP reallocation 2014/10/27 16:23:34.690047 [ 1701]: 60.nfs: Reconfiguring service "nfs"... 2014/10/27 16:23:34.963319 [recoverd: 1929]: Resetting ban count to 0 for all nodes 2014/10/27 16:23:58.004451 [ 1701]: High RECLOCK latency 1.151293s for operation recd reclock 2014/10/27 16:24:43.116578 [recoverd: 1929]: ctdb_control error: 'node is disconnected' 2014/10/27 16:24:43.116674 [recoverd: 1929]: ctdb_control error: 'node is disconnected' 2014/10/27 16:24:43.116689 [recoverd: 1929]: Async operation failed with ret=-1 res=-1 opcode=80 2014/10/27 16:24:43.116698 [recoverd: 1929]: Async wait failed - fail_count=1 2014/10/27 16:24:43.116707 [recoverd: 1929]: server/ctdb_recoverd.c:345 Failed to read node capabilities. 2014/10/27 16:24:43.116730 [recoverd: 1929]: server/ctdb_recoverd.c:3678 Unable to update node capabilities. 2014/10/27 16:24:45.331407 [ 1701]: pnn 2 Invalid reqid 2979 in ctdb_reply_control 2014/10/27 16:25:43.124079 [recoverd: 1929]: ctdb_control error: 'node is disconnected' 2014/10/27 16:25:43.124167 [recoverd: 1929]: ctdb_control error: 'node is disconnected' 2014/10/27 16:25:43.124183 [recoverd: 1929]: Async operation failed with ret=-1 res=-1 opcode=80 2014/10/27 16:25:43.124207 [recoverd: 1929]: Async wait failed - fail_count=1 2014/10/27 16:25:43.124216 [recoverd: 1929]: server/ctdb_recoverd.c:345 Failed to read node capabilities. 2014/10/27 16:25:43.124227 [recoverd: 1929]: server/ctdb_recoverd.c:3678 Unable to update node capabilities. 2014/10/27 16:25:45.875583 [ 1701]: pnn 2 Invalid reqid 3497 in ctdb_reply_control 2014/10/27 16:25:46.427968 [ 1701]: pnn 2 Invalid reqid 3402 in ctdb_reply_control 2014/10/27 16:25:53.998288 [ 1701]: Event script '00.ctdb monitor ' timed out after 60.0s, count: 0, pid: 20796 2014/10/27 16:25:57.155173 [ 1701]: pnn 2 Invalid reqid 3761 in ctdb_reply_control 2014/10/27 16:25:59.092489 [ 1701]: High RECLOCK latency 1.936301s for operation recd reclock 2014/10/27 16:26:59.093872 [ 1701]: Recovery daemon ping timeout. Count : 0 2014/10/27 16:27:08.999500 [ 1701]: Event script '10.interface monitor ' timed out after 23.1s, count: 1, pid: 25018 2014/10/27 16:27:33.269145 [recoverd: 1929]: ctdb_control error: 'node is disconnected' 2014/10/27 16:27:33.269233 [recoverd: 1929]: client/ctdb_client.c:1535 ctdb_control for getnodes failed ret:-1 res:-1 2014/10/27 16:27:33.269249 [recoverd: 1929]: server/ctdb_recoverd.c:1183 Unable to get nodemap from remote node 0 2014/10/27 16:27:33.269264 [recoverd: 1929]: Unable to update local flags 2014/10/27 16:27:33.272001 [recoverd: 1929]: Taking out recovery lock from recovery daemon 2014/10/27 16:27:33.272032 [recoverd: 1929]: Take the recovery lock 2014/10/27 16:28:09.000169 [ 1701]: ctdb_kill: trying to kill(25018, 9) a process that does not exist 2014/10/27 16:28:09.267471 [ 1701]: pnn 2 Invalid reqid 4444 in ctdb_reply_control 2014/10/27 16:28:24.000749 [ 1701]: Event script '00.ctdb monitor ' timed out after 60.0s, count: 2, pid: 26144 2014/10/27 16:28:33.270336 [ 1701]: Recovery daemon ping timeout. Count : 0 2014/10/27 16:29:24.001811 [ 1701]: ctdb_kill: trying to kill(26144, 9) a process that does not exist 2014/10/27 16:29:27.827980 [recoverd: 1929]: ctdb_recovery_lock: Failed to get recovery lock on '/mnt/lock/lockfile' 2014/10/27 16:29:27.828038 [recoverd: 1929]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds 2014/10/27 16:29:27.828116 [ 1701]: Banning this node for 30 seconds 2014/10/27 16:29:27.828153 [ 1701]: Freeze priority 1 2014/10/27 16:29:27.828324 [ 1701]: Freeze priority 2 2014/10/27 16:29:27.828451 [ 1701]: Freeze priority 3 2014/10/27 16:29:27.828600 [ 1701]: Monitoring event was cancelled 2014/10/27 16:29:27.828630 [ 1701]: server/eventscript.c:569 Sending SIGTERM to child pid:29275 2014/10/27 16:29:33.270926 [ 1701]: Recovery daemon ping timeout. Count : 1 2014/10/27 16:29:42.829357 [ 1701]: Skip monitoring since databases are frozen 2014/10/27 16:29:57.828410 [ 1701]: Banning timedout 2014/10/27 16:29:57.829484 [ 1701]: Skip monitoring since databases are frozen 2014/10/27 16:30:12.830161 [ 1701]: Skip monitoring since databases are frozen 2014/10/27 16:30:15.074050 [ 1701]: Freeze priority 1 2014/10/27 16:30:15.075124 [ 1701]: Freeze priority 2 2014/10/27 16:30:15.076194 [ 1701]: Freeze priority 3 2014/10/27 16:30:18.091334 [ 1701]: Freeze priority 1 2014/10/27 16:30:18.091621 [ 1701]: Freeze priority 2 2014/10/27 16:30:18.091884 [ 1701]: Freeze priority 3 2014/10/27 16:30:27.828367 [recoverd: 1929]: ctdb_control error: 'ctdb_control timed out' 2014/10/27 16:30:27.828440 [recoverd: 1929]: client/ctdb_client.c:4643 ctdb_control for set ban state failed 2014/10/27 16:30:27.828451 [recoverd: 1929]: server/ctdb_recoverd.c:176 Failed to ban node 2 2014/10/27 16:30:27.829074 [ 1701]: Event script '00.ctdb releaseip bond1 10.10.10.184 24' timed out after 60.0s, count: 3, pid: 30855 2014/10/27 16:30:27.829105 [ 1701]: Ignoring hung script for bond1 10.10.10.184 24 call 6 2014/10/27 16:31:13.391289 [recoverd: 1929]: ctdb_control error: 'node is disconnected' 2014/10/27 16:31:13.391371 [recoverd: 1929]: ctdb_control error: 'node is disconnected' 2014/10/27 16:31:13.391386 [recoverd: 1929]: Async operation failed with ret=-1 res=-1 opcode=80 2014/10/27 16:31:13.391397 [recoverd: 1929]: Async wait failed - fail_count=1 2014/10/27 16:31:13.391418 [recoverd: 1929]: server/ctdb_recoverd.c:345 Failed to read node capabilities. 2014/10/27 16:31:13.391428 [recoverd: 1929]: server/ctdb_recoverd.c:3678 Unable to update node capabilities. 2014/10/27 16:31:18.092892 [ 1701]: Event script '00.ctdb startrecovery ' timed out after 60.0s, count: 3, pid: 32555 2014/10/27 16:31:18.092986 [ 1701]: Ignoring hung script for call 3 2014/10/27 16:31:27.829584 [ 1701]: ctdb_kill: trying to kill(30855, 9) a process that does not exist 2014/10/27 16:31:27.829722 [ 1701]: Event script '00.ctdb releaseip bond1 10.10.10.182 24' timed out after 60.0s, count: 3, pid: 369 2014/10/27 16:31:27.829737 [ 1701]: Ignoring hung script for bond1 10.10.10.182 24 call 6 2014/10/27 16:31:27.829895 [ 1701]: pnn 2 Invalid reqid 4948 in ctdb_reply_control 2014/10/27 16:32:00.659414 [ 1701]: pnn 2 Invalid reqid 4069 in ctdb_reply_control 2014/10/27 16:32:00.659482 [ 1701]: pnn 2 Invalid reqid 4075 in ctdb_reply_control 2014/10/27 16:32:00.659509 [ 1701]: pnn 2 Invalid reqid 4129 in ctdb_reply_control 2014/10/27 16:32:00.659521 [ 1701]: pnn 2 Invalid reqid 4189 in ctdb_reply_control 2014/10/27 16:32:13.392539 [ 1701]: Recovery daemon ping timeout. Count : 0 2014/10/27 16:32:13.392682 [recoverd: 1929]: ctdb_control error: 'ctdb_control timed out' 2014/10/27 16:32:13.392722 [recoverd: 1929]: ctdb_control error: 'ctdb_control timed out' 2014/10/27 16:32:13.392743 [recoverd: 1929]: Async operation failed with ret=-1 res=-1 opcode=80 2014/10/27 16:32:13.392759 [recoverd: 1929]: Async wait failed - fail_count=1 2014/10/27 16:32:13.392784 [recoverd: 1929]: server/ctdb_recoverd.c:345 Failed to read node capabilities. 2014/10/27 16:32:13.392804 [recoverd: 1929]: server/ctdb_recoverd.c:3678 Unable to update node capabilities. 2014/10/27 16:32:18.092956 [ 1701]: server/ctdb_recover.c:562 Been in recovery mode for too long. Dropping all IPS 2014/10/27 16:32:58.452549 [recoverd: 1929]: ctdb_control error: 'node is disconnected' 2014/10/27 16:32:58.452589 [ 1701]: tcp/tcp_connect.c:211Failed to bind socket Cannot assign requested address(99) 2014/10/27 16:32:58.452623 [recoverd: 1929]: ctdb_control error: 'node is disconnected' 2014/10/27 16:32:58.452641 [recoverd: 1929]: Async operation failed with ret=-1 res=-1 opcode=80 2014/10/27 16:32:58.452652 [recoverd: 1929]: Async wait failed - fail_count=1 2014/10/27 16:32:58.452663 [recoverd: 1929]: server/ctdb_recoverd.c:345 Failed to read node capabilities. 2014/10/27 16:32:58.452674 [recoverd: 1929]: server/ctdb_recoverd.c:3678 Unable to update node capabilities. 2014/10/27 16:33:13.454963 [recoverd: 1929]: ctdb_control error: 'node is disconnected' 2014/10/27 16:33:13.454973 [ 1701]: tcp/tcp_connect.c:211Failed to bind socket Cannot assign requested address(99) 2014/10/27 16:33:13.455031 [recoverd: 1929]: ctdb_control error: 'node is disconnected' 2014/10/27 16:33:13.455048 [recoverd: 1929]: Async operation failed with ret=-1 res=-1 opcode=80 2014/10/27 16:33:13.455060 [recoverd: 1929]: Async wait failed - fail_count=1 2014/10/27 16:33:13.455070 [recoverd: 1929]: server/ctdb_recoverd.c:345 Failed to read node capabilities. 2014/10/27 16:33:13.455081 [recoverd: 1929]: server/ctdb_recoverd.c:3678 Unable to update node capabilities. 2014/10/27 16:33:13.456016 [ 1701]: Freeze priority 1 2014/10/27 16:33:13.456174 [ 1701]: Freeze priority 2 2014/10/27 16:33:13.456329 [ 1701]: Freeze priority 3 2014/10/27 16:33:16.461003 [recoverd: 1929]: Taking out recovery lock from recovery daemon 2014/10/27 16:33:16.461070 [recoverd: 1929]: Take the recovery lock 2014/10/27 16:34:10.057443 [recoverd: 1929]: ctdb_recovery_lock: Unable to open /mnt/lock/lockfile - (Transport endpoint is not connected) 2014/10/27 16:34:10.057482 [recoverd: 1929]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds 2014/10/27 16:34:10.057582 [ 1701]: Banning this node for 30 seconds 2014/10/27 16:34:40.058417 [ 1701]: Banning timedout 2014/10/27 16:34:40.102905 [recoverd: 1929]: Taking out recovery lock from recovery daemon 2014/10/27 16:34:40.102957 [recoverd: 1929]: Take the recovery lock 2014/10/27 16:34:40.103175 [recoverd: 1929]: ctdb_recovery_lock: Unable to open /mnt/lock/lockfile - (Transport endpoint is not connected) 2014/10/27 16:34:40.103198 [recoverd: 1929]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds 2014/10/27 16:34:40.103228 [ 1701]: Banning this node for 30 seconds 2014/10/27 16:35:10.103415 [ 1701]: Banning timedout 2014/10/27 16:35:10.148526 [recoverd: 1929]: Taking out recovery lock from recovery daemon 2014/10/27 16:35:10.148596 [recoverd: 1929]: Take the recovery lock 2014/10/27 16:35:10.148995 [recoverd: 1929]: ctdb_recovery_lock: Unable to open /mnt/lock/lockfile - (Transport endpoint is not connected) 2014/10/27 16:35:10.149021 [recoverd: 1929]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds 2014/10/27 16:35:10.149069 [ 1701]: Banning this node for 30 seconds 2014/10/27 16:35:13.457079 [ 1701]: server/ctdb_recover.c:562 Been in recovery mode for too long. Dropping all IPS 2014/10/27 16:35:40.149181 [ 1701]: Banning timedout 2014/10/27 16:35:40.198703 [recoverd: 1929]: Taking out recovery lock from recovery daemon 2014/10/27 16:35:40.198768 [recoverd: 1929]: Take the recovery lock 2014/10/27 16:35:40.199153 [recoverd: 1929]: ctdb_recovery_lock: Unable to open /mnt/lock/lockfile - (Transport endpoint is not connected) 2014/10/27 16:35:40.199168 [recoverd: 1929]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds 2014/10/27 16:35:40.199199 [ 1701]: Banning this node for 30 seconds 2014/10/27 16:35:55.380365 [ 1701]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/27 16:35:55.391833 [ 1701]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/28 11:42:50.796318 [ 1555]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/28 11:42:50.796425 [ 1555]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/28 11:42:50.796440 [ 1555]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/28 13:43:34.009786 [18275]: Starting CTDBD (Version 2.5.3) as PID: 18275 2014/10/28 13:43:34.878010 [18275]: Vacuuming is disabled for persistent database registry.tdb 2014/10/28 13:43:34.895656 [18275]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/28 13:43:34.914100 [18275]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/28 13:43:34.931442 [18275]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/28 13:43:34.948996 [18275]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/28 13:43:34.966424 [18275]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/28 13:43:34.983027 [18275]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/28 13:43:34.983078 [18275]: Freeze priority 1 2014/10/28 13:43:34.994663 [18275]: Freeze priority 2 2014/10/28 13:43:34.995095 [18275]: Freeze priority 3 2014/10/28 13:43:35.084334 [18275]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/28 13:43:35.088408 [18275]: 00.ctdb: Set RecoverTimeout to 60 2014/10/28 13:43:35.091910 [18275]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/28 13:43:35.219371 [18275]: Freeze priority 1 2014/10/28 13:43:35.219451 [18275]: Freeze priority 2 2014/10/28 13:43:35.219495 [18275]: Freeze priority 3 2014/10/28 13:43:35.742105 [18275]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/28 13:43:35.808236 [18275]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/28 13:43:39.226456 [recoverd:18517]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/28 13:43:39.226592 [18275]: Freeze priority 1 2014/10/28 13:43:39.226664 [18275]: Freeze priority 2 2014/10/28 13:43:39.226723 [18275]: Freeze priority 3 2014/10/28 13:43:43.175153 [18275]: Freeze priority 1 2014/10/28 13:43:43.176249 [18275]: Freeze priority 2 2014/10/28 13:43:43.177111 [18275]: Freeze priority 3 2014/10/28 13:43:43.319778 [18275]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/28 13:43:43.320559 [18275]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/28 13:43:44.405392 [18275]: Thawing priority 1 2014/10/28 13:43:44.405449 [18275]: Release freeze handler for prio 1 2014/10/28 13:43:44.405483 [18275]: Thawing priority 2 2014/10/28 13:43:44.405511 [18275]: Release freeze handler for prio 2 2014/10/28 13:43:44.405534 [18275]: Thawing priority 3 2014/10/28 13:43:44.405559 [18275]: Release freeze handler for prio 3 2014/10/28 13:43:58.747418 [18275]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/28 13:43:59.012218 [18275]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/28 13:43:59.023752 [18275]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/28 13:43:59.045309 [18275]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/28 13:43:59.250982 [18275]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/28 13:43:59.256986 [recoverd:18517]: Trigger takeoverrun 2014/10/28 13:44:01.726411 [18275]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation 2014/10/28 13:44:02.179387 [18275]: 60.nfs: Reconfiguring service "nfs"... 2014/10/28 15:07:35.793781 [18275]: Freeze priority 1 2014/10/28 15:07:35.794785 [18275]: Freeze priority 2 2014/10/28 15:07:35.795716 [18275]: Freeze priority 3 2014/10/28 15:07:38.894209 [18275]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/28 15:07:38.905980 [18275]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/28 15:10:36.264082 [ 1560]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/28 15:10:36.264188 [ 1560]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/28 15:10:36.264204 [ 1560]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/28 15:14:37.877527 [ 8024]: Starting CTDBD (Version 2.5.3) as PID: 8024 2014/10/28 15:14:39.319788 [ 8024]: Vacuuming is disabled for persistent database registry.tdb 2014/10/28 15:14:39.337086 [ 8024]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/28 15:14:39.355193 [ 8024]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/28 15:14:39.372606 [ 8024]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/28 15:14:39.390014 [ 8024]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/28 15:14:39.406603 [ 8024]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/28 15:14:39.422159 [ 8024]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/28 15:14:39.422200 [ 8024]: Freeze priority 1 2014/10/28 15:14:39.437229 [ 8024]: Freeze priority 2 2014/10/28 15:14:39.437799 [ 8024]: Freeze priority 3 2014/10/28 15:14:39.534743 [ 8024]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/28 15:14:39.538506 [ 8024]: 00.ctdb: Set RecoverTimeout to 60 2014/10/28 15:14:39.544025 [ 8024]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/28 15:14:39.664832 [ 8024]: Freeze priority 1 2014/10/28 15:14:39.664919 [ 8024]: Freeze priority 2 2014/10/28 15:14:39.664973 [ 8024]: Freeze priority 3 2014/10/28 15:14:43.170726 [recoverd: 8268]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/28 15:14:43.170817 [ 8024]: Freeze priority 1 2014/10/28 15:14:43.170881 [ 8024]: Freeze priority 2 2014/10/28 15:14:43.170936 [ 8024]: Freeze priority 3 2014/10/28 15:14:46.839971 [ 8024]: Freeze priority 1 2014/10/28 15:14:46.841126 [ 8024]: Freeze priority 2 2014/10/28 15:14:46.841848 [ 8024]: Freeze priority 3 2014/10/28 15:14:50.931074 [ 8024]: Thawing priority 1 2014/10/28 15:14:50.931113 [ 8024]: Release freeze handler for prio 1 2014/10/28 15:14:50.931143 [ 8024]: Thawing priority 2 2014/10/28 15:14:50.931164 [ 8024]: Release freeze handler for prio 2 2014/10/28 15:14:50.931191 [ 8024]: Thawing priority 3 2014/10/28 15:14:50.931210 [ 8024]: Release freeze handler for prio 3 2014/10/28 15:14:55.089867 [ 8024]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/28 15:15:01.293444 [ 8024]: Freeze priority 1 2014/10/28 15:15:01.294714 [ 8024]: Freeze priority 2 2014/10/28 15:15:01.295674 [ 8024]: Freeze priority 3 2014/10/28 15:15:04.469018 [ 8024]: Thawing priority 1 2014/10/28 15:15:04.469065 [ 8024]: Release freeze handler for prio 1 2014/10/28 15:15:04.469096 [ 8024]: Thawing priority 2 2014/10/28 15:15:04.469116 [ 8024]: Release freeze handler for prio 2 2014/10/28 15:15:04.469147 [ 8024]: Thawing priority 3 2014/10/28 15:15:04.469166 [ 8024]: Release freeze handler for prio 3 2014/10/28 15:15:19.493509 [recoverd: 8268]: Trigger takeoverrun 2014/10/28 15:15:19.668661 [ 8024]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/28 15:15:20.159380 [ 8024]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/28 15:15:20.174343 [ 8024]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/28 15:15:20.207130 [ 8024]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/28 15:15:20.380708 [ 8024]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/28 15:15:23.955123 [ 8024]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation 2014/10/28 15:15:24.451979 [ 8024]: 60.nfs: Reconfiguring service "nfs"... 2014/10/28 15:15:35.457606 [ 8024]: Freeze priority 1 2014/10/28 15:15:35.458734 [ 8024]: Freeze priority 2 2014/10/28 15:15:35.459571 [ 8024]: Freeze priority 3 2014/10/28 15:15:39.595930 [ 8024]: Thawing priority 1 2014/10/28 15:15:39.595970 [ 8024]: Release freeze handler for prio 1 2014/10/28 15:15:39.596001 [ 8024]: Thawing priority 2 2014/10/28 15:15:39.596020 [ 8024]: Release freeze handler for prio 2 2014/10/28 15:15:39.596049 [ 8024]: Thawing priority 3 2014/10/28 15:15:39.596067 [ 8024]: Release freeze handler for prio 3 2014/10/28 15:20:23.960958 [ 8024]: Freeze priority 1 2014/10/28 15:20:23.965481 [ 8024]: Freeze priority 1 2014/10/28 15:20:29.341908 [ 8024]: Skip monitoring since databases are frozen ===== Start of debug locks PID=13440 ===== 13064 /usr/sbin/smbd smbXsrv_session_global.tdb.2 85332 85332 13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 85330 85332 W 12416 /usr/sbin/smbd smbXsrv_session_global.tdb.2 115124 115124 13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 85329 13069 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=12416 ----- #0 0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f40493041b4 in smbd_accept_connection () #25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f4049300d01 in main () ----- Stack trace for PID=13064 ----- #0 0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f40493041b4 in smbd_accept_connection () #25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f4049300d01 in main () ----- Stack trace for PID=13069 ----- #0 0x00007f71e0cbc094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xbfd6b0, rw=1, off=85330, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xbfd6b0, rw_type=1, offset=85330, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=12) at lib/tdb/common/lock.c:541 #6 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=24) at lib/tdb/common/lock.c:537 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=48) at lib/tdb/common/lock.c:537 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=97) at lib/tdb/common/lock.c:537 #9 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=195) at lib/tdb/common/lock.c:537 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=390) at lib/tdb/common/lock.c:537 #11 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=781) at lib/tdb/common/lock.c:537 #12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=1562) at lib/tdb/common/lock.c:541 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=81418, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=12500) at lib/tdb/common/lock.c:541 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=25001) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=50168, len=50001) at lib/tdb/common/lock.c:541 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xbfd6b0) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff67503dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff67502278) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=13440 ===== ===== Start of debug locks PID=14593 ===== 13064 /usr/sbin/smbd smbXsrv_session_global.tdb.2 85332 85332 13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 85330 85332 W 12416 /usr/sbin/smbd smbXsrv_session_global.tdb.2 115124 115124 13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 85329 13069 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=12416 ----- 2014/10/28 15:20:44.342724 [ 8024]: Skip monitoring since databases are frozen #0 0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f40493041b4 in smbd_accept_connection () #25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f4049300d01 in main () ----- Stack trace for PID=13064 ----- #0 0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f40493041b4 in smbd_accept_connection () #25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f4049300d01 in main () ----- Stack trace for PID=13069 ----- #0 0x00007f71e0cbc094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xbfd6b0, rw=1, off=85330, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xbfd6b0, rw_type=1, offset=85330, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=12) at lib/tdb/common/lock.c:541 #6 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=24) at lib/tdb/common/lock.c:537 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=48) at lib/tdb/common/lock.c:537 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=97) at lib/tdb/common/lock.c:537 #9 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=195) at lib/tdb/common/lock.c:537 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=390) at lib/tdb/common/lock.c:537 #11 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=781) at lib/tdb/common/lock.c:537 #12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=1562) at lib/tdb/common/lock.c:541 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=81418, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=12500) at lib/tdb/common/lock.c:541 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=25001) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=50168, len=50001) at lib/tdb/common/lock.c:541 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xbfd6b0) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff67503dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff67502278) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=14593 ===== ===== Start of debug locks PID=15012 ===== 13064 /usr/sbin/smbd smbXsrv_session_global.tdb.2 85332 85332 13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 85330 85332 W 12416 /usr/sbin/smbd smbXsrv_session_global.tdb.2 115124 115124 13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 85329 13069 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=12416 ----- #0 0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f40493041b4 in smbd_accept_connection () #25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f4049300d01 in main () ----- Stack trace for PID=13064 ----- #0 0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f40493041b4 in smbd_accept_connection () #25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f4049300d01 in main () ----- Stack trace for PID=13069 ----- #0 0x00007f71e0cbc094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xbfd6b0, rw=1, off=85330, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xbfd6b0, rw_type=1, offset=85330, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=12) at lib/tdb/common/lock.c:541 #6 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=24) at lib/tdb/common/lock.c:537 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=48) at lib/tdb/common/lock.c:537 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=97) at lib/tdb/common/lock.c:537 #9 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=195) at lib/tdb/common/lock.c:537 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=390) at lib/tdb/common/lock.c:537 #11 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=781) at lib/tdb/common/lock.c:537 #12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=1562) at lib/tdb/common/lock.c:541 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=81418, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=12500) at lib/tdb/common/lock.c:541 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=25001) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=50168, len=50001) at lib/tdb/common/lock.c:541 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xbfd6b0) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff67503dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff67502278) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=15012 ===== 2014/10/28 15:20:59.343680 [ 8024]: Skip monitoring since databases are frozen ===== Start of debug locks PID=15456 ===== 13064 /usr/sbin/smbd smbXsrv_session_global.tdb.2 85332 85332 13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 85330 85332 W 12416 /usr/sbin/smbd smbXsrv_session_global.tdb.2 115124 115124 13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 85329 13069 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=12416 ----- #0 0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f40493041b4 in smbd_accept_connection () #25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f4049300d01 in main () ----- Stack trace for PID=13064 ----- #0 0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f40493041b4 in smbd_accept_connection () #25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f4049300d01 in main () ----- Stack trace for PID=13069 ----- #0 0x00007f71e0cbc094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xbfd6b0, rw=1, off=85330, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xbfd6b0, rw_type=1, offset=85330, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=12) at lib/tdb/common/lock.c:541 #6 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=24) at lib/tdb/common/lock.c:537 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=48) at lib/tdb/common/lock.c:537 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=97) at lib/tdb/common/lock.c:537 #9 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=195) at lib/tdb/common/lock.c:537 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=390) at lib/tdb/common/lock.c:537 #11 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=781) at lib/tdb/common/lock.c:537 #12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=1562) at lib/tdb/common/lock.c:541 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=81418, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=12500) at lib/tdb/common/lock.c:541 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=25001) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=50168, len=50001) at lib/tdb/common/lock.c:541 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xbfd6b0) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff67503dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff67502278) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=15456 ===== ===== Start of debug locks PID=15948 ===== 13064 /usr/sbin/smbd smbXsrv_session_global.tdb.2 85332 85332 13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 85330 85332 W 12416 /usr/sbin/smbd smbXsrv_session_global.tdb.2 115124 115124 13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 85329 13069 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=12416 ----- 2014/10/28 15:21:14.344067 [ 8024]: Skip monitoring since databases are frozen #0 0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f40493041b4 in smbd_accept_connection () #25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f4049300d01 in main () ----- Stack trace for PID=13064 ----- #0 0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f40493041b4 in smbd_accept_connection () #25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f4049300d01 in main () ----- Stack trace for PID=13069 ----- #0 0x00007f71e0cbc094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xbfd6b0, rw=1, off=85330, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xbfd6b0, rw_type=1, offset=85330, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=12) at lib/tdb/common/lock.c:541 #6 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=24) at lib/tdb/common/lock.c:537 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=48) at lib/tdb/common/lock.c:537 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=97) at lib/tdb/common/lock.c:537 #9 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=195) at lib/tdb/common/lock.c:537 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=390) at lib/tdb/common/lock.c:537 #11 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=781) at lib/tdb/common/lock.c:537 #12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=1562) at lib/tdb/common/lock.c:541 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=81418, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=12500) at lib/tdb/common/lock.c:541 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=25001) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=50168, len=50001) at lib/tdb/common/lock.c:541 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xbfd6b0) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff67503dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff67502278) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=15948 ===== 2014/10/28 15:21:23.964033 [ 8024]: Freeze priority 1 2014/10/28 15:21:23.964085 [ 8024]: Recovery daemon ping timeout. Count : 0 2014/10/28 15:21:23.966262 [recoverd: 8268]: ctdb_control error: 'ctdb_control timed out' 2014/10/28 15:21:23.966310 [recoverd: 8268]: ctdb_control error: 'ctdb_control timed out' 2014/10/28 15:21:23.966349 [recoverd: 8268]: Async operation failed with ret=-1 res=-1 opcode=33 2014/10/28 15:21:23.966367 [recoverd: 8268]: Failed to freeze node 1 during recovery. Set it as ban culprit for 3 credits 2014/10/28 15:21:23.966388 [recoverd: 8268]: ctdb_control error: 'ctdb_control timed out' 2014/10/28 15:21:23.966401 [recoverd: 8268]: ctdb_control error: 'ctdb_control timed out' 2014/10/28 15:21:23.966417 [recoverd: 8268]: Async operation failed with ret=-1 res=-1 opcode=33 2014/10/28 15:21:23.966429 [recoverd: 8268]: Failed to freeze node 2 during recovery. Set it as ban culprit for 3 credits 2014/10/28 15:21:23.966444 [recoverd: 8268]: Async wait failed - fail_count=2 2014/10/28 15:21:23.966457 [recoverd: 8268]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed. 2014/10/28 15:21:23.966472 [recoverd: 8268]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster 2014/10/28 15:21:23.968027 [ 8024]: Freeze priority 1 ===== Start of debug locks PID=16375 ===== 13064 /usr/sbin/smbd smbXsrv_session_global.tdb.2 85332 85332 13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 85330 85332 W 12416 /usr/sbin/smbd smbXsrv_session_global.tdb.2 115124 115124 13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 85329 13069 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=12416 ----- #0 0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f40493041b4 in smbd_accept_connection () #25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f4049300d01 in main () ----- Stack trace for PID=13064 ----- #0 0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f40493041b4 in smbd_accept_connection () #25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f4049300d01 in main () ----- Stack trace for PID=13069 ----- #0 0x00007f71e0cbc094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xbfd6b0, rw=1, off=85330, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xbfd6b0, rw_type=1, offset=85330, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=12) at lib/tdb/common/lock.c:541 #6 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=24) at lib/tdb/common/lock.c:537 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=48) at lib/tdb/common/lock.c:537 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=97) at lib/tdb/common/lock.c:537 #9 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=195) at lib/tdb/common/lock.c:537 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=390) at lib/tdb/common/lock.c:537 #11 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=781) at lib/tdb/common/lock.c:537 #12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=1562) at lib/tdb/common/lock.c:541 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=81418, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=12500) at lib/tdb/common/lock.c:541 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=25001) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=50168, len=50001) at lib/tdb/common/lock.c:541 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xbfd6b0) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff67503dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff67502278) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=16375 ===== 2014/10/28 15:21:29.344420 [ 8024]: Skip monitoring since databases are frozen ===== Start of debug locks PID=16809 ===== 13064 /usr/sbin/smbd smbXsrv_session_global.tdb.2 85332 85332 13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 85330 85332 W 12416 /usr/sbin/smbd smbXsrv_session_global.tdb.2 115124 115124 13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 85329 13069 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=12416 ----- #0 0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f40493041b4 in smbd_accept_connection () #25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f4049300d01 in main () ----- Stack trace for PID=13064 ----- #0 0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f40493041b4 in smbd_accept_connection () #25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f4049300d01 in main () ----- Stack trace for PID=13069 ----- #0 0x00007f71e0cbc094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xbfd6b0, rw=1, off=85330, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xbfd6b0, rw_type=1, offset=85330, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=12) at lib/tdb/common/lock.c:541 #6 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=24) at lib/tdb/common/lock.c:537 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=48) at lib/tdb/common/lock.c:537 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=97) at lib/tdb/common/lock.c:537 #9 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=195) at lib/tdb/common/lock.c:537 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=390) at lib/tdb/common/lock.c:537 #11 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=781) at lib/tdb/common/lock.c:537 #12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=1562) at lib/tdb/common/lock.c:541 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=81418, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=12500) at lib/tdb/common/lock.c:541 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=25001) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=50168, len=50001) at lib/tdb/common/lock.c:541 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xbfd6b0) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff67503dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff67502278) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=16809 ===== ===== Start of debug locks PID=17179 ===== 13064 /usr/sbin/smbd smbXsrv_session_global.tdb.2 85332 85332 13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 85330 85332 W 12416 /usr/sbin/smbd smbXsrv_session_global.tdb.2 115124 115124 13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 85329 13069 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=12416 ----- 2014/10/28 15:21:44.345209 [ 8024]: Skip monitoring since databases are frozen #0 0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f40493041b4 in smbd_accept_connection () #25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f4049300d01 in main () ----- Stack trace for PID=13064 ----- #0 0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f40493041b4 in smbd_accept_connection () #25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f4049300d01 in main () ----- Stack trace for PID=13069 ----- #0 0x00007f71e0cbc094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xbfd6b0, rw=1, off=85330, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xbfd6b0, rw_type=1, offset=85330, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=12) at lib/tdb/common/lock.c:541 #6 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=24) at lib/tdb/common/lock.c:537 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=48) at lib/tdb/common/lock.c:537 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=97) at lib/tdb/common/lock.c:537 #9 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=195) at lib/tdb/common/lock.c:537 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=390) at lib/tdb/common/lock.c:537 #11 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=781) at lib/tdb/common/lock.c:537 #12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=1562) at lib/tdb/common/lock.c:541 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=81418, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=12500) at lib/tdb/common/lock.c:541 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=25001) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=50168, len=50001) at lib/tdb/common/lock.c:541 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xbfd6b0) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff67503dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff67502278) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=17179 ===== ===== Start of debug locks PID=17435 ===== 13064 /usr/sbin/smbd smbXsrv_session_global.tdb.2 85332 85332 13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 85330 85332 W 12416 /usr/sbin/smbd smbXsrv_session_global.tdb.2 115124 115124 13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 85329 13069 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=12416 ----- #0 0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f40493041b4 in smbd_accept_connection () #25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f4049300d01 in main () ----- Stack trace for PID=13064 ----- #0 0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f40493041b4 in smbd_accept_connection () #25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f4049300d01 in main () ----- Stack trace for PID=13069 ----- #0 0x00007f71e0cbc094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xbfd6b0, rw=1, off=85330, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xbfd6b0, rw_type=1, offset=85330, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=12) at lib/tdb/common/lock.c:541 #6 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=24) at lib/tdb/common/lock.c:537 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=48) at lib/tdb/common/lock.c:537 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=97) at lib/tdb/common/lock.c:537 #9 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=195) at lib/tdb/common/lock.c:537 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=390) at lib/tdb/common/lock.c:537 #11 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=781) at lib/tdb/common/lock.c:537 #12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=1562) at lib/tdb/common/lock.c:541 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=81418, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=12500) at lib/tdb/common/lock.c:541 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=25001) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=50168, len=50001) at lib/tdb/common/lock.c:541 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xbfd6b0) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff67503dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff67502278) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=17435 ===== 2014/10/28 15:21:59.345909 [ 8024]: Skip monitoring since databases are frozen 2014/10/28 15:22:00.150854 [ 8024]: pnn 2 Invalid reqid 293987 in ctdb_reply_control ===== Start of debug locks PID=17694 ===== 13064 /usr/sbin/smbd smbXsrv_session_global.tdb.2 85332 85332 13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 85330 85332 W 12416 /usr/sbin/smbd smbXsrv_session_global.tdb.2 115124 115124 13069 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 85329 13069 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 13069 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=12416 ----- #0 0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f40493041b4 in smbd_accept_connection () #25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f4049300d01 in main () ----- Stack trace for PID=13064 ----- #0 0x00007f4045c68df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f4047533db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f40475373bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f40475385ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f404753b10f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f40475409ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f4043e3cafb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f4043e3cb2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f40475425e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f4043e3c8a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f40488b656f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f40488b6fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f40488a6843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f40488a2c21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f40488a319f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f40488a009c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f40472fc534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f40472fc069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f40472faf46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f4045f423f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f404754c71c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f404754ca04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f404888ebb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f40493041b4 in smbd_accept_connection () #25 0x00007f404754c84c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f404754caa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f4045f41bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f4049300d01 in main () ----- Stack trace for PID=13069 ----- #0 0x00007f71e0cbc094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xbfd6b0, rw=1, off=85330, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xbfd6b0, rw_type=1, offset=85330, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85330, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=12) at lib/tdb/common/lock.c:541 #6 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=24) at lib/tdb/common/lock.c:537 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=48) at lib/tdb/common/lock.c:537 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=97) at lib/tdb/common/lock.c:537 #9 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=195) at lib/tdb/common/lock.c:537 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=390) at lib/tdb/common/lock.c:537 #11 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=85324, len=781) at lib/tdb/common/lock.c:537 #12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=1562) at lib/tdb/common/lock.c:541 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=84543, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=81418, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=12500) at lib/tdb/common/lock.c:541 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=75168, len=25001) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=50168, len=50001) at lib/tdb/common/lock.c:541 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xbfd6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xbfd6b0) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff67503dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff67502278) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=17694 ===== 2014/10/28 15:22:12.191506 [ 8024]: pnn 2 Invalid reqid 293988 in ctdb_reply_control 2014/10/28 15:22:12.191593 [ 8024]: Freeze priority 2 2014/10/28 15:22:12.191600 [recoverd: 8268]: ctdb_control error: 'ctdb_control to disconnected node' 2014/10/28 15:22:12.191658 [recoverd: 8268]: ctdb_control error: 'ctdb_control to disconnected node' 2014/10/28 15:22:12.191687 [recoverd: 8268]: Async operation failed with ret=-1 res=-1 opcode=33 2014/10/28 15:22:12.191709 [recoverd: 8268]: Failed to freeze node 1 during recovery. Set it as ban culprit for 3 credits 2014/10/28 15:22:12.192248 [recoverd: 8268]: Async wait failed - fail_count=1 2014/10/28 15:22:12.192277 [recoverd: 8268]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed. 2014/10/28 15:22:12.192297 [recoverd: 8268]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster 2014/10/28 15:22:12.192847 [recoverd: 8268]: ctdb_control error: 'ctdb_control to disconnected node' 2014/10/28 15:22:12.192880 [recoverd: 8268]: client/ctdb_client.c:4643 ctdb_control for set ban state failed 2014/10/28 15:22:12.192892 [recoverd: 8268]: server/ctdb_recoverd.c:176 Failed to ban node 1 2014/10/28 15:22:12.193011 [ 8024]: Freeze priority 1 2014/10/28 15:22:12.193081 [ 8024]: Freeze priority 2 2014/10/28 15:22:12.193155 [ 8024]: Freeze priority 3 2014/10/28 15:22:15.197554 [recoverd: 8268]: Taking out recovery lock from recovery daemon 2014/10/28 15:22:15.197587 [recoverd: 8268]: Take the recovery lock 2014/10/28 15:22:15.199791 [ 8024]: Freeze priority 1 2014/10/28 15:22:15.199846 [ 8024]: Freeze priority 2 2014/10/28 15:22:15.199886 [ 8024]: Freeze priority 3 2014/10/28 15:22:15.599021 [ 8024]: 60.nfs: /etc/ctdb/functions: line 37: /etc/sysconfig/ctdb: Software caused connection abort 2014/10/28 15:22:16.134484 [ 8024]: 00.ctdb: /etc/ctdb/functions: line 37: /etc/sysconfig/ctdb: Software caused connection abort 2014/10/28 15:22:16.134629 [ 8024]: server/ctdb_recover.c:971 startrecovery event script failed (status 1) 2014/10/28 15:22:16.134712 [ 8024]: pnn 2 Invalid reqid 294481 in ctdb_reply_control 2014/10/28 15:22:16.262335 [ 8024]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/28 15:22:16.273200 [ 8024]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/28 15:29:17.403745 [ 1545]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/28 15:29:17.403897 [ 1545]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/28 15:29:17.403911 [ 1545]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/28 15:33:18.798488 [ 7997]: Starting CTDBD (Version 2.5.3) as PID: 7997 2014/10/28 15:33:19.888102 [ 7997]: Vacuuming is disabled for persistent database registry.tdb 2014/10/28 15:33:19.902848 [ 7997]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/28 15:33:19.917427 [ 7997]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/28 15:33:19.931290 [ 7997]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/28 15:33:19.946036 [ 7997]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/28 15:33:19.960934 [ 7997]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/28 15:33:19.976015 [ 7997]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/28 15:33:19.976155 [ 7997]: Freeze priority 1 2014/10/28 15:33:19.991610 [ 7997]: Freeze priority 2 2014/10/28 15:33:19.991981 [ 7997]: Freeze priority 3 2014/10/28 15:33:20.080995 [ 7997]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/28 15:33:20.084774 [ 7997]: 00.ctdb: Set RecoverTimeout to 60 2014/10/28 15:33:20.088310 [ 7997]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/28 15:33:20.217395 [ 7997]: Freeze priority 1 2014/10/28 15:33:20.217476 [ 7997]: Freeze priority 2 2014/10/28 15:33:20.217530 [ 7997]: Freeze priority 3 2014/10/28 15:33:23.803848 [recoverd: 8280]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/28 15:33:23.803931 [ 7997]: Freeze priority 1 2014/10/28 15:33:23.803991 [ 7997]: Freeze priority 2 2014/10/28 15:33:23.804043 [ 7997]: Freeze priority 3 2014/10/28 15:33:27.057437 [ 7997]: Freeze priority 1 2014/10/28 15:33:27.058229 [ 7997]: Freeze priority 2 2014/10/28 15:33:27.058763 [ 7997]: Freeze priority 3 2014/10/28 15:33:27.200565 [ 7997]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/28 15:33:32.275443 [ 7997]: Handling event took 4 seconds! 2014/10/28 15:33:32.278537 [ 7997]: Thawing priority 1 2014/10/28 15:33:32.278614 [ 7997]: Release freeze handler for prio 1 2014/10/28 15:33:32.278652 [ 7997]: Thawing priority 2 2014/10/28 15:33:32.278673 [ 7997]: Release freeze handler for prio 2 2014/10/28 15:33:32.278700 [ 7997]: Thawing priority 3 2014/10/28 15:33:32.278720 [ 7997]: Release freeze handler for prio 3 2014/10/28 15:33:47.282595 [ 7997]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/28 15:33:47.301208 [recoverd: 8280]: Trigger takeoverrun 2014/10/28 15:33:48.048369 [ 7997]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/28 15:33:48.061804 [ 7997]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/28 15:33:48.090275 [ 7997]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/28 15:33:48.278527 [ 7997]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/28 15:33:50.717806 [ 7997]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation 2014/10/28 15:33:52.156153 [ 7997]: 60.nfs: Reconfiguring service "nfs"... 2014/10/28 15:34:37.729290 [ 7997]: Freeze priority 1 2014/10/28 15:34:37.730337 [ 7997]: Freeze priority 2 2014/10/28 15:34:37.731151 [ 7997]: Freeze priority 3 2014/10/28 15:34:38.982118 [ 7997]: Thawing priority 1 2014/10/28 15:34:38.982171 [ 7997]: Release freeze handler for prio 1 2014/10/28 15:34:38.982213 [ 7997]: Thawing priority 2 2014/10/28 15:34:38.982234 [ 7997]: Release freeze handler for prio 2 2014/10/28 15:34:38.982269 [ 7997]: Thawing priority 3 2014/10/28 15:34:38.982288 [ 7997]: Release freeze handler for prio 3 2014/10/28 15:55:31.160893 [ 7997]: Freeze priority 1 2014/10/28 15:55:31.162055 [ 7997]: Freeze priority 2 2014/10/28 15:55:31.163045 [ 7997]: Freeze priority 3 2014/10/28 15:55:34.186193 [recoverd: 8280]: Taking out recovery lock from recovery daemon 2014/10/28 15:55:34.186227 [recoverd: 8280]: Take the recovery lock 2014/10/28 15:55:34.189109 [ 7997]: Freeze priority 1 2014/10/28 15:55:34.189170 [ 7997]: Freeze priority 2 2014/10/28 15:55:34.189213 [ 7997]: Freeze priority 3 2014/10/28 15:55:36.499864 [ 7997]: Thawing priority 1 2014/10/28 15:55:36.499904 [ 7997]: Release freeze handler for prio 1 2014/10/28 15:55:36.499940 [ 7997]: Thawing priority 2 2014/10/28 15:55:36.499961 [ 7997]: Release freeze handler for prio 2 2014/10/28 15:55:36.499996 [ 7997]: Thawing priority 3 2014/10/28 15:55:36.500015 [ 7997]: Release freeze handler for prio 3 2014/10/28 15:55:36.500991 [ 7997]: server/ctdb_call.c:1005 reqid 1253773 not found 2014/10/28 15:55:36.501025 [ 7997]: server/ctdb_call.c:1005 reqid 1253774 not found 2014/10/28 15:55:36.848367 [ 7997]: 60.nfs: Reconfiguring service "nfs"... 2014/10/28 15:55:37.023172 [recoverd: 8280]: Resetting ban count to 0 for all nodes 2014/10/28 15:55:47.875052 [ 7997]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/28 15:55:47.885594 [ 7997]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/28 15:58:38.248954 [ 1546]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/28 15:58:38.249114 [ 1546]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/28 15:58:38.249134 [ 1546]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/28 16:02:39.394901 [ 8039]: Starting CTDBD (Version 2.5.3) as PID: 8039 2014/10/28 16:02:40.284154 [ 8039]: Vacuuming is disabled for persistent database registry.tdb 2014/10/28 16:02:40.297936 [ 8039]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/28 16:02:40.312620 [ 8039]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/28 16:02:40.326540 [ 8039]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/28 16:02:40.340425 [ 8039]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/28 16:02:40.354386 [ 8039]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/28 16:02:40.368231 [ 8039]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/28 16:02:40.368260 [ 8039]: Freeze priority 1 2014/10/28 16:02:40.379777 [ 8039]: Freeze priority 2 2014/10/28 16:02:40.380139 [ 8039]: Freeze priority 3 2014/10/28 16:02:40.469492 [ 8039]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/28 16:02:40.473611 [ 8039]: 00.ctdb: Set RecoverTimeout to 60 2014/10/28 16:02:40.477296 [ 8039]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/28 16:02:40.597178 [ 8039]: Freeze priority 1 2014/10/28 16:02:40.597253 [ 8039]: Freeze priority 2 2014/10/28 16:02:40.597307 [ 8039]: Freeze priority 3 2014/10/28 16:02:44.102517 [recoverd: 8274]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/28 16:02:44.102611 [ 8039]: Freeze priority 1 2014/10/28 16:02:44.102677 [ 8039]: Freeze priority 2 2014/10/28 16:02:44.102738 [ 8039]: Freeze priority 3 2014/10/28 16:02:47.460062 [ 8039]: Freeze priority 1 2014/10/28 16:02:47.460793 [ 8039]: Freeze priority 2 2014/10/28 16:02:47.461545 [ 8039]: Freeze priority 3 2014/10/28 16:02:47.598360 [ 8039]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/28 16:02:49.141359 [ 8039]: Thawing priority 1 2014/10/28 16:02:49.141406 [ 8039]: Release freeze handler for prio 1 2014/10/28 16:02:49.141436 [ 8039]: Thawing priority 2 2014/10/28 16:02:49.141464 [ 8039]: Release freeze handler for prio 2 2014/10/28 16:02:49.141488 [ 8039]: Thawing priority 3 2014/10/28 16:02:49.141504 [ 8039]: Release freeze handler for prio 3 2014/10/28 16:03:03.627551 [recoverd: 8274]: Trigger takeoverrun 2014/10/28 16:03:04.038618 [ 8039]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/28 16:03:04.379270 [ 8039]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/28 16:03:04.390200 [ 8039]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/28 16:03:04.413027 [ 8039]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/28 16:03:04.601594 [ 8039]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/28 16:03:07.054322 [ 8039]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation 2014/10/28 16:03:07.900699 [ 8039]: 60.nfs: Reconfiguring service "nfs"... 2014/10/28 16:03:30.492139 [ 8039]: Freeze priority 1 2014/10/28 16:03:30.493255 [ 8039]: Freeze priority 2 2014/10/28 16:03:30.494283 [ 8039]: Freeze priority 3 2014/10/28 16:03:32.412399 [ 8039]: Thawing priority 1 2014/10/28 16:03:32.412454 [ 8039]: Release freeze handler for prio 1 2014/10/28 16:03:32.412487 [ 8039]: Thawing priority 2 2014/10/28 16:03:32.412515 [ 8039]: Release freeze handler for prio 2 2014/10/28 16:03:32.412540 [ 8039]: Thawing priority 3 2014/10/28 16:03:32.412555 [ 8039]: Release freeze handler for prio 3 2014/10/28 17:08:24.484388 [ 8039]: Freeze priority 1 2014/10/28 17:08:24.494222 [ 8039]: Freeze priority 1 2014/10/28 17:08:39.119651 [ 8039]: Skip monitoring since databases are frozen 2014/10/28 17:08:54.120233 [ 8039]: Skip monitoring since databases are frozen 2014/10/28 17:09:09.120357 [ 8039]: Skip monitoring since databases are frozen 2014/10/28 17:09:24.120973 [ 8039]: Skip monitoring since databases are frozen 2014/10/28 17:09:24.487003 [ 8039]: Freeze priority 1 2014/10/28 17:09:24.493148 [ 8039]: Recovery daemon ping timeout. Count : 0 2014/10/28 17:09:24.495347 [recoverd: 8274]: ctdb_control error: 'ctdb_control timed out' 2014/10/28 17:09:24.495393 [recoverd: 8274]: ctdb_control error: 'ctdb_control timed out' 2014/10/28 17:09:24.495417 [recoverd: 8274]: Async operation failed with ret=-1 res=-1 opcode=33 2014/10/28 17:09:24.495433 [recoverd: 8274]: Failed to freeze node 1 during recovery. Set it as ban culprit for 3 credits 2014/10/28 17:09:24.495450 [recoverd: 8274]: Async wait failed - fail_count=1 2014/10/28 17:09:24.495465 [recoverd: 8274]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed. 2014/10/28 17:09:24.495480 [recoverd: 8274]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster 2014/10/28 17:09:24.497084 [ 8039]: Freeze priority 1 2014/10/28 17:09:39.121758 [ 8039]: Skip monitoring since databases are frozen 2014/10/28 17:09:54.122453 [ 8039]: Skip monitoring since databases are frozen 2014/10/28 17:09:54.578568 [ 8039]: pnn 2 Invalid reqid 2929975 in ctdb_reply_control 2014/10/28 17:09:54.578688 [ 8039]: Freeze priority 2 2014/10/28 17:09:54.578900 [ 8039]: Freeze priority 2 2014/10/28 17:09:54.579710 [ 8039]: Freeze priority 3 2014/10/28 17:09:54.579978 [ 8039]: Freeze priority 3 2014/10/28 17:09:57.586366 [recoverd: 8274]: Taking out recovery lock from recovery daemon 2014/10/28 17:09:57.586420 [recoverd: 8274]: Take the recovery lock 2014/10/28 17:09:57.590009 [ 8039]: Freeze priority 1 2014/10/28 17:09:57.590081 [ 8039]: Freeze priority 2 2014/10/28 17:09:57.590148 [ 8039]: Freeze priority 3 2014/10/28 17:09:59.683656 [ 8039]: Thawing priority 1 2014/10/28 17:09:59.683701 [ 8039]: Release freeze handler for prio 1 2014/10/28 17:09:59.683729 [ 8039]: Thawing priority 2 2014/10/28 17:09:59.683746 [ 8039]: Release freeze handler for prio 2 2014/10/28 17:09:59.683771 [ 8039]: Thawing priority 3 2014/10/28 17:09:59.683794 [ 8039]: Release freeze handler for prio 3 2014/10/28 17:09:59.895066 [ 8039]: common/ctdb_fork.c:131 waitpid() returned error. errno:10 2014/10/28 17:09:59.895186 [ 8039]: pnn 2 Invalid reqid 2930556 in ctdb_reply_control 2014/10/28 17:09:59.914099 [ 8039]: common/ctdb_fork.c:131 waitpid() returned error. errno:10 2014/10/28 17:09:59.914154 [ 8039]: pnn 2 Invalid reqid 2930554 in ctdb_reply_control 2014/10/28 17:10:00.428347 [ 8039]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/28 17:10:00.438986 [ 8039]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/28 17:17:03.248575 [ 1546]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/28 17:17:03.248671 [ 1546]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/28 17:17:03.248685 [ 1546]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/28 17:21:05.044759 [ 8316]: Starting CTDBD (Version 2.5.3) as PID: 8316 2014/10/28 17:21:05.942401 [ 8316]: Vacuuming is disabled for persistent database registry.tdb 2014/10/28 17:21:05.956168 [ 8316]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/28 17:21:05.970671 [ 8316]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/28 17:21:05.984520 [ 8316]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/28 17:21:05.998347 [ 8316]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/28 17:21:06.012493 [ 8316]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/28 17:21:06.026478 [ 8316]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/28 17:21:06.026510 [ 8316]: Freeze priority 1 2014/10/28 17:21:06.038113 [ 8316]: Freeze priority 2 2014/10/28 17:21:06.038456 [ 8316]: Freeze priority 3 2014/10/28 17:21:06.127527 [ 8316]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/28 17:21:06.131659 [ 8316]: 00.ctdb: Set RecoverTimeout to 60 2014/10/28 17:21:06.135335 [ 8316]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/28 17:21:06.262555 [ 8316]: Freeze priority 1 2014/10/28 17:21:06.262639 [ 8316]: Freeze priority 2 2014/10/28 17:21:06.262693 [ 8316]: Freeze priority 3 2014/10/28 17:21:09.767515 [recoverd: 8583]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/28 17:21:09.767619 [ 8316]: Freeze priority 1 2014/10/28 17:21:09.767690 [ 8316]: Freeze priority 2 2014/10/28 17:21:09.767751 [ 8316]: Freeze priority 3 2014/10/28 17:21:13.719252 [ 8316]: Freeze priority 1 2014/10/28 17:21:13.719969 [ 8316]: Freeze priority 2 2014/10/28 17:21:13.720494 [ 8316]: Freeze priority 3 2014/10/28 17:21:13.863229 [ 8316]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/28 17:21:19.116982 [ 8316]: Handling event took 5 seconds! 2014/10/28 17:21:19.118635 [ 8316]: Thawing priority 1 2014/10/28 17:21:19.118677 [ 8316]: Release freeze handler for prio 1 2014/10/28 17:21:19.118707 [ 8316]: Thawing priority 2 2014/10/28 17:21:19.118726 [ 8316]: Release freeze handler for prio 2 2014/10/28 17:21:19.118752 [ 8316]: Thawing priority 3 2014/10/28 17:21:19.118770 [ 8316]: Release freeze handler for prio 3 2014/10/28 17:21:33.437609 [ 8316]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/28 17:21:33.759486 [ 8316]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/28 17:21:33.770638 [ 8316]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/28 17:21:33.809196 [ 8316]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/28 17:21:33.960678 [ 8316]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/28 17:21:34.140581 [recoverd: 8583]: Trigger takeoverrun 2014/10/28 17:21:36.422132 [ 8316]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation 2014/10/28 17:21:37.260465 [ 8316]: 60.nfs: Reconfiguring service "nfs"... 2014/10/28 17:22:25.861996 [ 8316]: Freeze priority 1 2014/10/28 17:22:25.862907 [ 8316]: Freeze priority 2 2014/10/28 17:22:25.863695 [ 8316]: Freeze priority 3 2014/10/28 17:22:27.310243 [ 8316]: Thawing priority 1 2014/10/28 17:22:27.310288 [ 8316]: Release freeze handler for prio 1 2014/10/28 17:22:27.310327 [ 8316]: Thawing priority 2 2014/10/28 17:22:27.310354 [ 8316]: Release freeze handler for prio 2 2014/10/28 17:22:27.310401 [ 8316]: Thawing priority 3 2014/10/28 17:22:27.310420 [ 8316]: Release freeze handler for prio 3 2014/10/28 17:22:37.814884 [ 8316]: Freeze priority 1 2014/10/28 17:22:37.815985 [ 8316]: Freeze priority 2 2014/10/28 17:22:37.816997 [ 8316]: Freeze priority 3 2014/10/28 17:22:39.114387 [ 8316]: Thawing priority 1 2014/10/28 17:22:39.114435 [ 8316]: Release freeze handler for prio 1 2014/10/28 17:22:39.114478 [ 8316]: Thawing priority 2 2014/10/28 17:22:39.114501 [ 8316]: Release freeze handler for prio 2 2014/10/28 17:22:39.114540 [ 8316]: Thawing priority 3 2014/10/28 17:22:39.114564 [ 8316]: Release freeze handler for prio 3 2014/10/28 17:28:55.784892 [ 8316]: Freeze priority 1 2014/10/28 17:28:56.704515 [ 8316]: Freeze priority 1 2014/10/28 17:29:03.997358 [ 8316]: Skip monitoring since databases are frozen ===== Start of debug locks PID=5989 ===== 5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 173347 5256 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 5255 /usr/sbin/smbd smbXsrv_session_global.tdb.2 173348 173348 5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 173348 173350 W ----- Stack trace for PID=5255 ----- #0 0x00007f7016cf2df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f70185bddb9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f70185c13bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f70185c25ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f70185c510f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f70185ca9ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f7014ec6afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f7014ec6b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f70185cc5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f7014ec68a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f701994056f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f7019940fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f7019930843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f701992cc21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f701992d19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f701992a09c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f7018386534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f7018386069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f7018384f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f7016fcc3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f70185d671c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f70185d6a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f7019918bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f701a38e1b4 in smbd_accept_connection () #25 0x00007f70185d684c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f70185d6aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f701a38ad01 in main () ----- Stack trace for PID=5256 ----- #0 0x00007fef801f6094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x179b6b0, rw=1, off=173348, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x179b6b0, rw_type=1, offset=173348, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=13) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173336, len=25) at lib/tdb/common/lock.c:541 #7 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=49) at lib/tdb/common/lock.c:541 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=98) at lib/tdb/common/lock.c:537 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172825, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=1562) at lib/tdb/common/lock.c:541 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168919, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=162669, len=12500) at lib/tdb/common/lock.c:541 #16 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=25000) at lib/tdb/common/lock.c:541 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=100169, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x179b6b0) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff86864dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff868641e8) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=5989 ===== ===== Start of debug locks PID=6398 ===== 5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 173347 5256 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 5255 /usr/sbin/smbd smbXsrv_session_global.tdb.2 173348 173348 5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 173348 173350 W ----- Stack trace for PID=5255 ----- #0 0x00007f7016cf2df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f70185bddb9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f70185c13bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f70185c25ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f70185c510f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f70185ca9ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f7014ec6afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f7014ec6b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f70185cc5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f7014ec68a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f701994056f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f7019940fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f7019930843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f701992cc21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f701992d19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f701992a09c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f7018386534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f7018386069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f7018384f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f7016fcc3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f70185d671c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f70185d6a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f7019918bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f701a38e1b4 in smbd_accept_connection () #25 0x00007f70185d684c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f70185d6aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f701a38ad01 in main () ----- Stack trace for PID=5256 ----- #0 0x00007fef801f6094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x179b6b0, rw=1, off=173348, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x179b6b0, rw_type=1, offset=173348, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=13) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173336, len=25) at lib/tdb/common/lock.c:541 #7 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=49) at lib/tdb/common/lock.c:541 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=98) at lib/tdb/common/lock.c:537 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172825, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=1562) at lib/tdb/common/lock.c:541 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168919, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=162669, len=12500) at lib/tdb/common/lock.c:541 #16 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=25000) at lib/tdb/common/lock.c:541 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=100169, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x179b6b0) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff86864dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff868641e8) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=6398 ===== 2014/10/28 17:29:18.997981 [ 8316]: Skip monitoring since databases are frozen ===== Start of debug locks PID=6760 ===== 5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 173347 5256 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 5255 /usr/sbin/smbd smbXsrv_session_global.tdb.2 173348 173348 5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 173348 173350 W ----- Stack trace for PID=5255 ----- #0 0x00007f7016cf2df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f70185bddb9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f70185c13bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f70185c25ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f70185c510f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f70185ca9ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f7014ec6afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f7014ec6b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f70185cc5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f7014ec68a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f701994056f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f7019940fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f7019930843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f701992cc21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f701992d19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f701992a09c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f7018386534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f7018386069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f7018384f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f7016fcc3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f70185d671c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f70185d6a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f7019918bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f701a38e1b4 in smbd_accept_connection () #25 0x00007f70185d684c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f70185d6aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f701a38ad01 in main () ----- Stack trace for PID=5256 ----- #0 0x00007fef801f6094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x179b6b0, rw=1, off=173348, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x179b6b0, rw_type=1, offset=173348, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=13) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173336, len=25) at lib/tdb/common/lock.c:541 #7 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=49) at lib/tdb/common/lock.c:541 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=98) at lib/tdb/common/lock.c:537 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172825, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=1562) at lib/tdb/common/lock.c:541 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168919, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=162669, len=12500) at lib/tdb/common/lock.c:541 #16 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=25000) at lib/tdb/common/lock.c:541 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=100169, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x179b6b0) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff86864dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff868641e8) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=6760 ===== 2014/10/28 17:29:33.998876 [ 8316]: Skip monitoring since databases are frozen ===== Start of debug locks PID=7107 ===== 5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 173347 5256 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 5255 /usr/sbin/smbd smbXsrv_session_global.tdb.2 173348 173348 5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 173348 173350 W ----- Stack trace for PID=5255 ----- #0 0x00007f7016cf2df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f70185bddb9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f70185c13bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f70185c25ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f70185c510f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f70185ca9ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f7014ec6afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f7014ec6b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f70185cc5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f7014ec68a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f701994056f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f7019940fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f7019930843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f701992cc21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f701992d19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f701992a09c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f7018386534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f7018386069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f7018384f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f7016fcc3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f70185d671c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f70185d6a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f7019918bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f701a38e1b4 in smbd_accept_connection () #25 0x00007f70185d684c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f70185d6aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f701a38ad01 in main () ----- Stack trace for PID=5256 ----- #0 0x00007fef801f6094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x179b6b0, rw=1, off=173348, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x179b6b0, rw_type=1, offset=173348, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=13) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173336, len=25) at lib/tdb/common/lock.c:541 #7 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=49) at lib/tdb/common/lock.c:541 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=98) at lib/tdb/common/lock.c:537 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172825, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=1562) at lib/tdb/common/lock.c:541 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168919, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=162669, len=12500) at lib/tdb/common/lock.c:541 #16 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=25000) at lib/tdb/common/lock.c:541 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=100169, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x179b6b0) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff86864dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff868641e8) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=7107 ===== ===== Start of debug locks PID=7540 ===== 5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 173347 5256 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 5255 /usr/sbin/smbd smbXsrv_session_global.tdb.2 173348 173348 5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 173348 173350 W ----- Stack trace for PID=5255 ----- #0 0x00007f7016cf2df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f70185bddb9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f70185c13bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f70185c25ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f70185c510f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f70185ca9ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f7014ec6afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f7014ec6b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f70185cc5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f7014ec68a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f701994056f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f7019940fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f7019930843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f701992cc21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f701992d19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f701992a09c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f7018386534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f7018386069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f7018384f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f7016fcc3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f70185d671c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f70185d6a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f7019918bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f701a38e1b4 in smbd_accept_connection () #25 0x00007f70185d684c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f70185d6aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f701a38ad01 in main () ----- Stack trace for PID=5256 ----- #0 0x00007fef801f6094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x179b6b0, rw=1, off=173348, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x179b6b0, rw_type=1, offset=173348, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=13) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173336, len=25) at lib/tdb/common/lock.c:541 #7 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=49) at lib/tdb/common/lock.c:541 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=98) at lib/tdb/common/lock.c:537 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172825, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=1562) at lib/tdb/common/lock.c:541 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168919, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=162669, len=12500) at lib/tdb/common/lock.c:541 #16 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=25000) at lib/tdb/common/lock.c:541 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=100169, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x179b6b0) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff86864dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff868641e8) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=7540 ===== 2014/10/28 17:29:48.999822 [ 8316]: Skip monitoring since databases are frozen 2014/10/28 17:29:55.787913 [ 8316]: Freeze priority 1 ===== Start of debug locks PID=7922 ===== 5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 173347 5256 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 5255 /usr/sbin/smbd smbXsrv_session_global.tdb.2 173348 173348 5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 173348 173350 W ----- Stack trace for PID=5255 ----- #0 0x00007f7016cf2df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f70185bddb9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f70185c13bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f70185c25ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f70185c510f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f70185ca9ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f7014ec6afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f7014ec6b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f70185cc5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f7014ec68a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f701994056f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f7019940fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f7019930843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f701992cc21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f701992d19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f701992a09c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f7018386534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f7018386069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f7018384f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f7016fcc3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f70185d671c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f70185d6a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f7019918bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f701a38e1b4 in smbd_accept_connection () #25 0x00007f70185d684c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f70185d6aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f701a38ad01 in main () ----- Stack trace for PID=5256 ----- #0 0x00007fef801f6094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x179b6b0, rw=1, off=173348, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x179b6b0, rw_type=1, offset=173348, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=13) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173336, len=25) at lib/tdb/common/lock.c:541 #7 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=49) at lib/tdb/common/lock.c:541 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=98) at lib/tdb/common/lock.c:537 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172825, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=1562) at lib/tdb/common/lock.c:541 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168919, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=162669, len=12500) at lib/tdb/common/lock.c:541 #16 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=25000) at lib/tdb/common/lock.c:541 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=100169, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x179b6b0) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff86864dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff868641e8) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=7922 ===== 2014/10/28 17:29:56.703260 [ 8316]: Recovery daemon ping timeout. Count : 0 2014/10/28 17:29:56.705448 [recoverd: 8583]: ctdb_control error: 'ctdb_control timed out' 2014/10/28 17:29:56.705497 [recoverd: 8583]: ctdb_control error: 'ctdb_control timed out' 2014/10/28 17:29:56.705516 [recoverd: 8583]: Async operation failed with ret=-1 res=-1 opcode=33 2014/10/28 17:29:56.705529 [recoverd: 8583]: Failed to freeze node 2 during recovery. Set it as ban culprit for 3 credits 2014/10/28 17:29:56.705544 [recoverd: 8583]: Async wait failed - fail_count=1 2014/10/28 17:29:56.705574 [recoverd: 8583]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed. 2014/10/28 17:29:56.705589 [recoverd: 8583]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster 2014/10/28 17:29:56.707222 [ 8316]: Freeze priority 1 2014/10/28 17:30:03.999970 [ 8316]: Skip monitoring since databases are frozen ===== Start of debug locks PID=8331 ===== 5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 173347 5256 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 5255 /usr/sbin/smbd smbXsrv_session_global.tdb.2 173348 173348 5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 173348 173350 W ----- Stack trace for PID=5255 ----- #0 0x00007f7016cf2df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f70185bddb9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f70185c13bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f70185c25ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f70185c510f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f70185ca9ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f7014ec6afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f7014ec6b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f70185cc5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f7014ec68a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f701994056f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f7019940fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f7019930843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f701992cc21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f701992d19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f701992a09c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f7018386534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f7018386069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f7018384f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f7016fcc3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f70185d671c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f70185d6a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f7019918bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f701a38e1b4 in smbd_accept_connection () #25 0x00007f70185d684c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f70185d6aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f701a38ad01 in main () ----- Stack trace for PID=5256 ----- #0 0x00007fef801f6094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x179b6b0, rw=1, off=173348, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x179b6b0, rw_type=1, offset=173348, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=13) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173336, len=25) at lib/tdb/common/lock.c:541 #7 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=49) at lib/tdb/common/lock.c:541 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=98) at lib/tdb/common/lock.c:537 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172825, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=1562) at lib/tdb/common/lock.c:541 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168919, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=162669, len=12500) at lib/tdb/common/lock.c:541 #16 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=25000) at lib/tdb/common/lock.c:541 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=100169, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x179b6b0) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff86864dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff868641e8) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=8331 ===== ===== Start of debug locks PID=8699 ===== 5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 173347 5256 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 5255 /usr/sbin/smbd smbXsrv_session_global.tdb.2 173348 173348 5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 173348 173350 W ----- Stack trace for PID=5255 ----- #0 0x00007f7016cf2df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f70185bddb9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f70185c13bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f70185c25ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f70185c510f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f70185ca9ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f7014ec6afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f7014ec6b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f70185cc5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f7014ec68a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f701994056f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f7019940fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f7019930843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f701992cc21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f701992d19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f701992a09c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f7018386534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f7018386069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f7018384f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f7016fcc3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f70185d671c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f70185d6a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f7019918bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f701a38e1b4 in smbd_accept_connection () #25 0x00007f70185d684c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f70185d6aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f701a38ad01 in main () ----- Stack trace for PID=5256 ----- #0 0x00007fef801f6094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x179b6b0, rw=1, off=173348, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x179b6b0, rw_type=1, offset=173348, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=13) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173336, len=25) at lib/tdb/common/lock.c:541 #7 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=49) at lib/tdb/common/lock.c:541 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=98) at lib/tdb/common/lock.c:537 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172825, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=1562) at lib/tdb/common/lock.c:541 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168919, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=162669, len=12500) at lib/tdb/common/lock.c:541 #16 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=25000) at lib/tdb/common/lock.c:541 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=100169, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x179b6b0) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff86864dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff868641e8) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=8699 ===== 2014/10/28 17:30:19.000450 [ 8316]: Skip monitoring since databases are frozen ===== Start of debug locks PID=9081 ===== 5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 173347 5256 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 5256 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 5255 /usr/sbin/smbd smbXsrv_session_global.tdb.2 173348 173348 5256 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 173348 173350 W ----- Stack trace for PID=5255 ----- #0 0x00007f7016cf2df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f70185bddb9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f70185c13bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f70185c25ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f70185c510f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f70185ca9ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f7014ec6afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f7014ec6b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f70185cc5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f7014ec68a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f701994056f in smbXsrv_session_global_store () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f7019940fc9 in smbXsrv_session_create () from /usr/lib64/samba/libsmbd_base.so #12 0x00007f7019930843 in smbd_smb2_request_process_sesssetup () from /usr/lib64/samba/libsmbd_base.so #13 0x00007f701992cc21 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #14 0x00007f701992d19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f701992a09c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f7018386534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #17 0x00007f7018386069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #18 0x00007f7018384f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #19 0x00007f7016fcc3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #20 0x00007f70185d671c in run_events_poll () from /lib64/libsmbconf.so.0 #21 0x00007f70185d6a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #22 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0 #23 0x00007f7019918bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #24 0x00007f701a38e1b4 in smbd_accept_connection () #25 0x00007f70185d684c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f70185d6aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f7016fcbbcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f701a38ad01 in main () ----- Stack trace for PID=5256 ----- #0 0x00007fef801f6094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x179b6b0, rw=1, off=173348, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x179b6b0, rw_type=1, offset=173348, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173348, len=13) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173336, len=25) at lib/tdb/common/lock.c:541 #7 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=49) at lib/tdb/common/lock.c:541 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173312, len=98) at lib/tdb/common/lock.c:537 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=173215, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172825, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=1562) at lib/tdb/common/lock.c:541 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=172044, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168919, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=162669, len=12500) at lib/tdb/common/lock.c:541 #16 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=25000) at lib/tdb/common/lock.c:541 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=150169, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=100169, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x179b6b0, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x179b6b0) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff86864dc4 "/var/lib/ctdb/smbXsrv_session_global.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff868641e8) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=9081 ===== 2014/10/28 17:30:30.275173 [ 8316]: pnn 2 Invalid reqid 431614 in ctdb_reply_control 2014/10/28 17:30:30.275591 [ 8316]: Freeze priority 2 2014/10/28 17:30:30.275595 [recoverd: 8583]: ctdb_control error: 'ctdb_control to disconnected node' 2014/10/28 17:30:30.275634 [recoverd: 8583]: ctdb_control error: 'ctdb_control to disconnected node' 2014/10/28 17:30:30.275660 [recoverd: 8583]: Async operation failed with ret=-1 res=-1 opcode=33 2014/10/28 17:30:30.275679 [recoverd: 8583]: Failed to freeze node 1 during recovery. Set it as ban culprit for 3 credits 2014/10/28 17:30:30.276313 [recoverd: 8583]: Async wait failed - fail_count=1 2014/10/28 17:30:30.276346 [recoverd: 8583]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed. 2014/10/28 17:30:30.276359 [recoverd: 8583]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster 2014/10/28 17:30:30.277177 [ 8316]: Freeze priority 1 2014/10/28 17:30:30.277245 [ 8316]: Freeze priority 2 2014/10/28 17:30:30.277297 [ 8316]: Freeze priority 3 2014/10/28 17:30:32.040647 [ 8316]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/28 17:30:32.051958 [ 8316]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/28 17:37:34.604030 [ 1520]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/28 17:37:34.604130 [ 1520]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/28 17:37:34.604147 [ 1520]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/28 17:41:33.996789 [ 7969]: Starting CTDBD (Version 2.5.3) as PID: 7969 2014/10/28 17:41:35.055980 [ 7969]: Vacuuming is disabled for persistent database registry.tdb 2014/10/28 17:41:35.071983 [ 7969]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/28 17:41:35.087387 [ 7969]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/28 17:41:35.101489 [ 7969]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/28 17:41:35.116062 [ 7969]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/28 17:41:35.130037 [ 7969]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/28 17:41:35.144014 [ 7969]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/28 17:41:35.144046 [ 7969]: Freeze priority 1 2014/10/28 17:41:35.277767 [ 7969]: Freeze priority 2 2014/10/28 17:41:35.278214 [ 7969]: Freeze priority 3 2014/10/28 17:41:35.293204 [ 7969]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/28 17:41:35.296994 [ 7969]: 00.ctdb: Set RecoverTimeout to 60 2014/10/28 17:41:35.300297 [ 7969]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/28 17:41:35.421222 [ 7969]: Freeze priority 1 2014/10/28 17:41:35.421297 [ 7969]: Freeze priority 2 2014/10/28 17:41:35.421350 [ 7969]: Freeze priority 3 2014/10/28 17:41:38.696426 [ 7969]: Freeze priority 1 2014/10/28 17:41:38.697151 [ 7969]: Freeze priority 2 2014/10/28 17:41:38.697707 [ 7969]: Freeze priority 3 2014/10/28 17:41:38.923315 [recoverd: 8207]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/28 17:41:38.923415 [ 7969]: Freeze priority 1 2014/10/28 17:41:38.923487 [ 7969]: Freeze priority 2 2014/10/28 17:41:38.923551 [ 7969]: Freeze priority 3 2014/10/28 17:41:43.342809 [ 7969]: Handling event took 4 seconds! 2014/10/28 17:41:43.345296 [ 7969]: Thawing priority 1 2014/10/28 17:41:43.345324 [ 7969]: Release freeze handler for prio 1 2014/10/28 17:41:43.345349 [ 7969]: Thawing priority 2 2014/10/28 17:41:43.345376 [ 7969]: Release freeze handler for prio 2 2014/10/28 17:41:43.345398 [ 7969]: Thawing priority 3 2014/10/28 17:41:43.345413 [ 7969]: Release freeze handler for prio 3 2014/10/28 17:41:46.346931 [recoverd: 8207]: Trigger takeoverrun 2014/10/28 17:41:53.943869 [ 7969]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/28 17:41:53.950835 [ 7969]: Freeze priority 1 2014/10/28 17:41:53.951905 [ 7969]: Freeze priority 2 2014/10/28 17:41:53.952769 [ 7969]: Freeze priority 3 2014/10/28 17:41:56.524977 [ 7969]: Thawing priority 1 2014/10/28 17:41:56.525033 [ 7969]: Release freeze handler for prio 1 2014/10/28 17:41:56.525076 [ 7969]: Thawing priority 2 2014/10/28 17:41:56.525098 [ 7969]: Release freeze handler for prio 2 2014/10/28 17:41:56.525141 [ 7969]: Thawing priority 3 2014/10/28 17:41:56.525161 [ 7969]: Release freeze handler for prio 3 2014/10/28 17:42:11.545493 [recoverd: 8207]: Trigger takeoverrun 2014/10/28 17:42:12.439831 [ 7969]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/28 17:42:12.838121 [ 7969]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/28 17:42:12.851341 [ 7969]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/28 17:42:12.881606 [ 7969]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/28 17:42:13.101211 [ 7969]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/28 17:42:15.606644 [ 7969]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation 2014/10/28 17:42:16.423117 [ 7969]: 60.nfs: Reconfiguring service "nfs"... 2014/10/28 17:43:00.998137 [ 7969]: Freeze priority 1 2014/10/28 17:43:00.999122 [ 7969]: Freeze priority 2 2014/10/28 17:43:00.999922 [ 7969]: Freeze priority 3 2014/10/28 17:43:05.301283 [ 7969]: Thawing priority 1 2014/10/28 17:43:05.301323 [ 7969]: Release freeze handler for prio 1 2014/10/28 17:43:05.301356 [ 7969]: Thawing priority 2 2014/10/28 17:43:05.301388 [ 7969]: Release freeze handler for prio 2 2014/10/28 17:43:05.301417 [ 7969]: Thawing priority 3 2014/10/28 17:43:05.301437 [ 7969]: Release freeze handler for prio 3 2014/10/28 17:43:15.821487 [ 7969]: Freeze priority 1 2014/10/28 17:43:15.822320 [ 7969]: Freeze priority 2 2014/10/28 17:43:15.823048 [ 7969]: Freeze priority 3 2014/10/28 17:43:17.254028 [ 7969]: Thawing priority 1 2014/10/28 17:43:17.254081 [ 7969]: Release freeze handler for prio 1 2014/10/28 17:43:17.254128 [ 7969]: Thawing priority 2 2014/10/28 17:43:17.254152 [ 7969]: Release freeze handler for prio 2 2014/10/28 17:43:17.254203 [ 7969]: Thawing priority 3 2014/10/28 17:43:17.254224 [ 7969]: Release freeze handler for prio 3 2014/10/28 17:46:28.001501 [ 7969]: No event for 19 seconds! 2014/10/29 09:05:41.508845 [ 7969]: Freeze priority 1 2014/10/29 09:05:41.510285 [ 7969]: Freeze priority 2 2014/10/29 09:05:41.511314 [ 7969]: Freeze priority 3 2014/10/29 09:05:44.663963 [recoverd: 8207]: Taking out recovery lock from recovery daemon 2014/10/29 09:05:44.664012 [recoverd: 8207]: Take the recovery lock 2014/10/29 09:05:44.675410 [ 7969]: Freeze priority 1 2014/10/29 09:05:44.675705 [ 7969]: Freeze priority 2 2014/10/29 09:05:44.675938 [ 7969]: Freeze priority 3 2014/10/29 09:05:47.314017 [ 7969]: Thawing priority 1 2014/10/29 09:05:47.314063 [ 7969]: Release freeze handler for prio 1 2014/10/29 09:05:47.314102 [ 7969]: Thawing priority 2 2014/10/29 09:05:47.314128 [ 7969]: Release freeze handler for prio 2 2014/10/29 09:05:47.314171 [ 7969]: Thawing priority 3 2014/10/29 09:05:47.314191 [ 7969]: Release freeze handler for prio 3 2014/10/29 09:05:47.831604 [recoverd: 8207]: Resetting ban count to 0 for all nodes 2014/10/29 09:06:00.838421 [recoverd: 8207]: server/ctdb_recoverd.c:3960 The vnnmap count is different from the number of active lmaster nodes: 2 vs 1 2014/10/29 09:06:00.838476 [recoverd: 8207]: Taking out recovery lock from recovery daemon 2014/10/29 09:06:00.838491 [recoverd: 8207]: Take the recovery lock 2014/10/29 09:06:00.843691 [ 7969]: Freeze priority 1 2014/10/29 09:06:00.844501 [ 7969]: Freeze priority 2 2014/10/29 09:06:00.845107 [ 7969]: Freeze priority 3 2014/10/29 09:06:02.815223 [ 7969]: Thawing priority 1 2014/10/29 09:06:02.815277 [ 7969]: Release freeze handler for prio 1 2014/10/29 09:06:02.815308 [ 7969]: Thawing priority 2 2014/10/29 09:06:02.815340 [ 7969]: Release freeze handler for prio 2 2014/10/29 09:06:02.815373 [ 7969]: Thawing priority 3 2014/10/29 09:06:02.815394 [ 7969]: Release freeze handler for prio 3 2014/10/29 09:06:03.175310 [ 7969]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 09:06:03.336407 [recoverd: 8207]: Resetting ban count to 0 for all nodes 2014/10/29 09:06:13.236166 [ 7969]: 10.interface: Re-adding secondary address 10.10.10.182/24 to dev bond1 2014/10/29 09:06:13.583318 [ 7969]: 50.samba: Redirecting to /bin/systemctl stop smb.service 2014/10/29 09:06:13.627326 [ 7969]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 09:06:13.638506 [ 7969]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 09:15:59.869061 [32406]: Starting CTDBD (Version 2.5.3) as PID: 32406 2014/10/29 09:16:00.238118 [32406]: Vacuuming is disabled for persistent database registry.tdb 2014/10/29 09:16:00.261772 [32406]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/29 09:16:00.275908 [32406]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/29 09:16:00.290135 [32406]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/29 09:16:00.304308 [32406]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/29 09:16:00.318257 [32406]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/29 09:16:00.332054 [32406]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/29 09:16:00.332096 [32406]: Freeze priority 1 2014/10/29 09:16:00.332622 [32406]: Freeze priority 2 2014/10/29 09:16:00.332962 [32406]: Freeze priority 3 2014/10/29 09:16:00.347767 [32406]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/29 09:16:00.351006 [32406]: 00.ctdb: Set RecoverTimeout to 60 2014/10/29 09:16:00.354266 [32406]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/29 09:16:00.434704 [32406]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 09:16:00.475130 [32406]: Freeze priority 1 2014/10/29 09:16:00.475205 [32406]: Freeze priority 2 2014/10/29 09:16:00.475259 [32406]: Freeze priority 3 2014/10/29 09:16:00.678341 [32406]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 09:16:03.980660 [recoverd:32690]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/29 09:16:03.980748 [32406]: Freeze priority 1 2014/10/29 09:16:03.980824 [32406]: Freeze priority 2 2014/10/29 09:16:03.980888 [32406]: Freeze priority 3 2014/10/29 09:16:09.761421 [32406]: Freeze priority 1 2014/10/29 09:16:09.762200 [32406]: Freeze priority 2 2014/10/29 09:16:09.762914 [32406]: Freeze priority 3 2014/10/29 09:16:09.897037 [32406]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/29 09:16:09.897736 [32406]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/29 09:16:11.963652 [32406]: Thawing priority 1 2014/10/29 09:16:11.963721 [32406]: Release freeze handler for prio 1 2014/10/29 09:16:11.963770 [32406]: Thawing priority 2 2014/10/29 09:16:11.963791 [32406]: Release freeze handler for prio 2 2014/10/29 09:16:11.963819 [32406]: Thawing priority 3 2014/10/29 09:16:11.963838 [32406]: Release freeze handler for prio 3 2014/10/29 09:16:25.767088 [32406]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/29 09:16:25.956298 [32406]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 09:16:25.969449 [32406]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 09:16:26.000345 [32406]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/29 09:16:26.178695 [32406]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/29 09:16:26.513799 [recoverd:32690]: Trigger takeoverrun 2014/10/29 09:16:28.445157 [32406]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation 2014/10/29 09:16:29.790563 [32406]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 10:04:18.714985 [32406]: Freeze priority 1 2014/10/29 10:04:18.746248 [32406]: Freeze priority 2 2014/10/29 10:04:18.747418 [32406]: Freeze priority 3 2014/10/29 10:04:20.477745 [32406]: Thawing priority 1 2014/10/29 10:04:20.477775 [32406]: Release freeze handler for prio 1 2014/10/29 10:04:20.477802 [32406]: Thawing priority 2 2014/10/29 10:04:20.477813 [32406]: Release freeze handler for prio 2 2014/10/29 10:04:20.477833 [32406]: Thawing priority 3 2014/10/29 10:04:20.477843 [32406]: Release freeze handler for prio 3 2014/10/29 10:04:20.479079 [32406]: server/ctdb_call.c:1005 reqid 69953 not found 2014/10/29 10:04:20.479116 [32406]: server/ctdb_call.c:1005 reqid 69954 not found 2014/10/29 10:04:20.834705 [32406]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 10:14:39.873806 [32406]: Freeze priority 1 2014/10/29 10:14:39.884064 [32406]: Freeze priority 2 2014/10/29 10:14:39.884976 [32406]: Freeze priority 3 2014/10/29 10:14:43.279029 [32406]: Thawing priority 1 2014/10/29 10:14:43.279070 [32406]: Release freeze handler for prio 1 2014/10/29 10:14:43.279103 [32406]: Thawing priority 2 2014/10/29 10:14:43.279137 [32406]: Release freeze handler for prio 2 2014/10/29 10:14:43.279166 [32406]: Thawing priority 3 2014/10/29 10:14:43.279196 [32406]: Release freeze handler for prio 3 2014/10/29 10:14:53.699000 [32406]: 10.interface: Killing TCP connection 10.10.10.206:49595 10.10.10.184:445 2014/10/29 10:14:53.711739 [32406]: 10.interface: Killed 1 TCP connections to released IP 10.10.10.184 2014/10/29 10:14:53.721091 [32406]: 10.interface: Re-adding secondary address 10.10.10.183/24 to dev bond1 2014/10/29 10:14:54.105661 [32406]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 11:18:08.638677 [32406]: Freeze priority 1 2014/10/29 11:18:08.639781 [32406]: Freeze priority 2 2014/10/29 11:18:08.640775 [32406]: Freeze priority 3 2014/10/29 11:18:11.878323 [recoverd:32690]: Taking out recovery lock from recovery daemon 2014/10/29 11:18:11.878356 [recoverd:32690]: Take the recovery lock 2014/10/29 11:18:11.880882 [32406]: Freeze priority 1 2014/10/29 11:18:11.880939 [32406]: Freeze priority 2 2014/10/29 11:18:11.880979 [32406]: Freeze priority 3 2014/10/29 11:18:15.539399 [32406]: Thawing priority 1 2014/10/29 11:18:15.539443 [32406]: Release freeze handler for prio 1 2014/10/29 11:18:15.539480 [32406]: Thawing priority 2 2014/10/29 11:18:15.539503 [32406]: Release freeze handler for prio 2 2014/10/29 11:18:15.539537 [32406]: Thawing priority 3 2014/10/29 11:18:15.539560 [32406]: Release freeze handler for prio 3 2014/10/29 11:18:15.914249 [32406]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 11:18:16.181043 [recoverd:32690]: Resetting ban count to 0 for all nodes 2014/10/29 11:18:22.591276 [32406]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 11:18:22.601991 [32406]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 11:21:14.096010 [ 1536]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/29 11:21:14.096118 [ 1536]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/29 11:21:14.096133 [ 1536]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/29 11:25:14.394731 [ 8359]: Starting CTDBD (Version 2.5.3) as PID: 8359 2014/10/29 11:25:15.387147 [ 8359]: Vacuuming is disabled for persistent database registry.tdb 2014/10/29 11:25:15.415256 [ 8359]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/29 11:25:15.430873 [ 8359]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/29 11:25:15.445127 [ 8359]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/29 11:25:15.459038 [ 8359]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/29 11:25:15.472939 [ 8359]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/29 11:25:15.486751 [ 8359]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/29 11:25:15.486783 [ 8359]: Freeze priority 1 2014/10/29 11:25:15.496220 [ 8359]: Freeze priority 2 2014/10/29 11:25:15.496627 [ 8359]: Freeze priority 3 2014/10/29 11:25:15.660399 [ 8359]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/29 11:25:15.663879 [ 8359]: 00.ctdb: Set RecoverTimeout to 60 2014/10/29 11:25:15.667414 [ 8359]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/29 11:25:15.788313 [ 8359]: Freeze priority 1 2014/10/29 11:25:15.788388 [ 8359]: Freeze priority 2 2014/10/29 11:25:15.788441 [ 8359]: Freeze priority 3 2014/10/29 11:25:19.293664 [recoverd: 8591]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/29 11:25:19.293746 [ 8359]: Freeze priority 1 2014/10/29 11:25:19.293809 [ 8359]: Freeze priority 2 2014/10/29 11:25:19.293863 [ 8359]: Freeze priority 3 2014/10/29 11:25:22.557346 [ 8359]: Freeze priority 1 2014/10/29 11:25:22.557887 [ 8359]: Freeze priority 2 2014/10/29 11:25:22.558372 [ 8359]: Freeze priority 3 2014/10/29 11:25:25.616951 [ 8359]: Thawing priority 1 2014/10/29 11:25:25.616987 [ 8359]: Release freeze handler for prio 1 2014/10/29 11:25:25.617012 [ 8359]: Thawing priority 2 2014/10/29 11:25:25.617030 [ 8359]: Release freeze handler for prio 2 2014/10/29 11:25:25.617065 [ 8359]: Thawing priority 3 2014/10/29 11:25:25.617082 [ 8359]: Release freeze handler for prio 3 2014/10/29 11:25:29.680386 [ 8359]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/29 11:25:35.963365 [ 8359]: Freeze priority 1 2014/10/29 11:25:35.964501 [ 8359]: Freeze priority 2 2014/10/29 11:25:35.965473 [ 8359]: Freeze priority 3 2014/10/29 11:25:38.577176 [ 8359]: Thawing priority 1 2014/10/29 11:25:38.577245 [ 8359]: Release freeze handler for prio 1 2014/10/29 11:25:38.577281 [ 8359]: Thawing priority 2 2014/10/29 11:25:38.577302 [ 8359]: Release freeze handler for prio 2 2014/10/29 11:25:38.577334 [ 8359]: Thawing priority 3 2014/10/29 11:25:38.577354 [ 8359]: Release freeze handler for prio 3 2014/10/29 11:25:53.601380 [recoverd: 8591]: Trigger takeoverrun 2014/10/29 11:25:53.776285 [ 8359]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/29 11:25:54.058009 [ 8359]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 11:25:54.068818 [ 8359]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 11:25:54.090914 [ 8359]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/29 11:25:54.291962 [ 8359]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/29 11:25:56.742700 [ 8359]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation 2014/10/29 11:25:57.570321 [ 8359]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 11:26:12.624219 [recoverd: 8591]: Taking out recovery lock from recovery daemon 2014/10/29 11:26:12.624286 [recoverd: 8591]: Take the recovery lock 2014/10/29 11:26:12.633945 [ 8359]: Freeze priority 1 2014/10/29 11:26:12.635864 [ 8359]: Freeze priority 2 2014/10/29 11:26:12.636875 [ 8359]: Freeze priority 3 2014/10/29 11:26:16.027676 [ 8359]: Thawing priority 1 2014/10/29 11:26:16.027724 [ 8359]: Release freeze handler for prio 1 2014/10/29 11:26:16.027758 [ 8359]: Thawing priority 2 2014/10/29 11:26:16.027790 [ 8359]: Release freeze handler for prio 2 2014/10/29 11:26:16.027819 [ 8359]: Thawing priority 3 2014/10/29 11:26:16.027851 [ 8359]: Release freeze handler for prio 3 2014/10/29 11:26:16.355892 [recoverd: 8591]: Resetting ban count to 0 for all nodes 2014/10/29 14:06:49.479891 [ 1493]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/29 14:06:49.479994 [ 1493]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/29 14:06:49.480009 [ 1493]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/29 14:10:53.418766 [ 8284]: Starting CTDBD (Version 2.5.3) as PID: 8284 2014/10/29 14:10:54.333021 [ 8284]: Vacuuming is disabled for persistent database registry.tdb 2014/10/29 14:10:54.356435 [ 8284]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/29 14:10:54.370534 [ 8284]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/29 14:10:54.384342 [ 8284]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/29 14:10:54.398206 [ 8284]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/29 14:10:54.412186 [ 8284]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/29 14:10:54.426139 [ 8284]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/29 14:10:54.426171 [ 8284]: Freeze priority 1 2014/10/29 14:10:54.445413 [ 8284]: Freeze priority 2 2014/10/29 14:10:54.445767 [ 8284]: Freeze priority 3 2014/10/29 14:10:54.639168 [ 8284]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/29 14:10:54.643377 [ 8284]: 00.ctdb: Set RecoverTimeout to 60 2014/10/29 14:10:54.647529 [ 8284]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/29 14:10:54.770475 [ 8284]: Freeze priority 1 2014/10/29 14:10:54.770548 [ 8284]: Freeze priority 2 2014/10/29 14:10:54.770602 [ 8284]: Freeze priority 3 2014/10/29 14:10:54.780011 [ 8284]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 14:10:55.133919 [ 8284]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 14:10:58.275532 [recoverd: 8515]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/29 14:10:58.275621 [ 8284]: Freeze priority 1 2014/10/29 14:10:58.275684 [ 8284]: Freeze priority 2 2014/10/29 14:10:58.275744 [ 8284]: Freeze priority 3 2014/10/29 14:11:02.770926 [ 8284]: Freeze priority 1 2014/10/29 14:11:02.780257 [ 8284]: Freeze priority 2 2014/10/29 14:11:02.781256 [ 8284]: Freeze priority 3 2014/10/29 14:11:02.918864 [ 8284]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/29 14:11:02.919638 [ 8284]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/29 14:11:04.581838 [ 8284]: Thawing priority 1 2014/10/29 14:11:04.581886 [ 8284]: Release freeze handler for prio 1 2014/10/29 14:11:04.581924 [ 8284]: Thawing priority 2 2014/10/29 14:11:04.581943 [ 8284]: Release freeze handler for prio 2 2014/10/29 14:11:04.581969 [ 8284]: Thawing priority 3 2014/10/29 14:11:04.581986 [ 8284]: Release freeze handler for prio 3 2014/10/29 14:11:18.806494 [recoverd: 8515]: Trigger takeoverrun 2014/10/29 14:11:18.853554 [ 8284]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/29 14:11:19.219152 [ 8284]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 14:11:19.230098 [ 8284]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 14:11:19.253268 [ 8284]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/29 14:11:19.411442 [ 8284]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/29 14:11:21.843929 [ 8284]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation 2014/10/29 14:11:22.323842 [ 8284]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 14:13:44.788288 [recoverd: 8515]: ctdb_control error: 'node is disconnected' 2014/10/29 14:13:44.788337 [recoverd: 8515]: ctdb_control error: 'node is disconnected' 2014/10/29 14:13:44.788354 [recoverd: 8515]: Async operation failed with ret=-1 res=-1 opcode=80 2014/10/29 14:13:44.788365 [recoverd: 8515]: Async wait failed - fail_count=1 2014/10/29 14:13:44.788377 [recoverd: 8515]: server/ctdb_recoverd.c:345 Failed to read node capabilities. 2014/10/29 14:13:44.788389 [recoverd: 8515]: server/ctdb_recoverd.c:3678 Unable to update node capabilities. 2014/10/29 14:13:44.790037 [ 8284]: Freeze priority 1 2014/10/29 14:13:44.791547 [ 8284]: Freeze priority 2 2014/10/29 14:13:44.792587 [ 8284]: Freeze priority 3 2014/10/29 14:14:09.335310 [ 8284]: Freeze priority 1 2014/10/29 14:14:09.335469 [ 8284]: Freeze priority 2 2014/10/29 14:14:09.335594 [ 8284]: Freeze priority 3 2014/10/29 14:14:12.342634 [recoverd: 8515]: Taking out recovery lock from recovery daemon 2014/10/29 14:14:12.342696 [recoverd: 8515]: Take the recovery lock 2014/10/29 14:14:24.099076 [ 8284]: Event script '00.ctdb monitor ' timed out after 60.0s, count: 0, pid: 19353 2014/10/29 14:14:33.373727 [recoverd: 8515]: ctdb_recovery_lock: Failed to get recovery lock on '/mnt/lock/lockfile' 2014/10/29 14:14:33.373764 [recoverd: 8515]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds 2014/10/29 14:14:33.373827 [ 8284]: Banning this node for 30 seconds 2014/10/29 14:14:33.374961 [ 8284]: Hung-script: ===== Start of hung script debug for PID="19353", event="monitor" ===== 2014/10/29 14:14:33.375006 [ 8284]: Hung-script: pstree -p -a 19353: 2014/10/29 14:14:33.415218 [ 8284]: Hung-script: 2014/10/29 14:14:33.416861 [ 8284]: 10.interface: Killing TCP connection 10.10.10.208:57370 10.10.10.184:445 2014/10/29 14:14:33.417298 [ 8284]: Hung-script: ---- ctdb scriptstatus monitor: ---- 2014/10/29 14:14:33.418463 [ 8284]: Hung-script: 1 scripts were executed last monitor cycle 2014/10/29 14:14:33.418498 [ 8284]: Hung-script: 00.ctdb Status:TIMEDOUT Wed Oct 29 14:13:24 2014 2014/10/29 14:14:33.418522 [ 8284]: Hung-script: OUTPUT: 2014/10/29 14:14:33.418657 [ 8284]: Hung-script: ===== End of hung script debug for PID="19353", event="monitor" ===== 2014/10/29 14:14:33.419037 [ 8284]: ctdb_kill: trying to kill(19353, 9) a process that does not exist 2014/10/29 14:14:33.422912 [ 8284]: 10.interface: Killed 1 TCP connections to released IP 10.10.10.184 2014/10/29 14:15:03.373964 [ 8284]: Banning timedout 2014/10/29 14:15:03.597247 [ 8284]: Freeze priority 1 2014/10/29 14:15:03.597326 [ 8284]: Freeze priority 2 2014/10/29 14:15:03.597370 [ 8284]: Freeze priority 3 2014/10/29 14:15:06.602285 [recoverd: 8515]: Public IP '10.10.10.184' is not assigned and we could serve it 2014/10/29 14:15:06.602419 [recoverd: 8515]: Trigger takeoverrun 2014/10/29 14:15:06.602923 [recoverd: 8515]: Taking out recovery lock from recovery daemon 2014/10/29 14:15:06.602943 [recoverd: 8515]: Take the recovery lock 2014/10/29 14:15:06.607383 [recoverd: 8515]: ctdb_recovery_lock: Failed to get recovery lock on '/mnt/lock/lockfile' 2014/10/29 14:15:06.607417 [recoverd: 8515]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds 2014/10/29 14:15:06.607472 [ 8284]: Banning this node for 30 seconds 2014/10/29 14:15:36.608086 [ 8284]: Banning timedout 2014/10/29 14:15:36.649226 [ 8284]: Freeze priority 1 2014/10/29 14:15:36.649407 [ 8284]: Freeze priority 2 2014/10/29 14:15:36.649571 [ 8284]: Freeze priority 3 2014/10/29 14:15:39.654977 [recoverd: 8515]: Public IP '10.10.10.184' is not assigned and we could serve it 2014/10/29 14:15:39.655117 [recoverd: 8515]: Trigger takeoverrun 2014/10/29 14:15:39.655707 [recoverd: 8515]: Taking out recovery lock from recovery daemon 2014/10/29 14:15:39.655721 [recoverd: 8515]: Take the recovery lock 2014/10/29 14:15:39.661064 [ 8284]: Freeze priority 1 2014/10/29 14:15:39.661129 [ 8284]: Freeze priority 2 2014/10/29 14:15:39.661175 [ 8284]: Freeze priority 3 2014/10/29 14:15:41.613703 [ 8284]: Thawing priority 1 2014/10/29 14:15:41.613742 [ 8284]: Release freeze handler for prio 1 2014/10/29 14:15:41.613778 [ 8284]: Thawing priority 2 2014/10/29 14:15:41.613810 [ 8284]: Release freeze handler for prio 2 2014/10/29 14:15:41.613849 [ 8284]: Thawing priority 3 2014/10/29 14:15:41.613869 [ 8284]: Release freeze handler for prio 3 2014/10/29 14:15:41.615178 [ 8284]: server/ctdb_call.c:1005 reqid 4760 not found 2014/10/29 14:15:41.615222 [ 8284]: server/ctdb_call.c:1005 reqid 4761 not found 2014/10/29 14:15:41.915021 [ 8284]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 14:15:42.073080 [recoverd: 8515]: Resetting ban count to 0 for all nodes 2014/10/29 14:15:52.077427 [recoverd: 8515]: Taking out recovery lock from recovery daemon 2014/10/29 14:15:52.077481 [recoverd: 8515]: Take the recovery lock 2014/10/29 14:15:52.090130 [ 8284]: Freeze priority 1 2014/10/29 14:15:52.091017 [ 8284]: Freeze priority 2 2014/10/29 14:15:52.091672 [ 8284]: Freeze priority 3 2014/10/29 14:15:54.653999 [ 8284]: Thawing priority 1 2014/10/29 14:15:54.654052 [ 8284]: Release freeze handler for prio 1 2014/10/29 14:15:54.654095 [ 8284]: Thawing priority 2 2014/10/29 14:15:54.654120 [ 8284]: Release freeze handler for prio 2 2014/10/29 14:15:54.654154 [ 8284]: Thawing priority 3 2014/10/29 14:15:54.654177 [ 8284]: Release freeze handler for prio 3 2014/10/29 14:15:55.102501 [ 8284]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 14:15:55.265147 [recoverd: 8515]: Resetting ban count to 0 for all nodes 2014/10/29 14:21:51.579877 [ 8284]: Event script '00.ctdb monitor ' timed out after 60.0s, count: 0, pid: 8945 2014/10/29 14:22:06.874627 [ 8284]: Hung-script: ===== Start of hung script debug for PID="8945", event="monitor" ===== 2014/10/29 14:22:06.874685 [ 8284]: Hung-script: pstree -p -a 8945: 2014/10/29 14:22:06.913145 [ 8284]: Hung-script: 2014/10/29 14:22:06.914938 [ 8284]: Hung-script: ---- ctdb scriptstatus monitor: ---- 2014/10/29 14:22:06.923328 [ 8284]: Hung-script: 1 scripts were executed last monitor cycle 2014/10/29 14:22:06.923414 [ 8284]: Hung-script: 00.ctdb Status:TIMEDOUT Wed Oct 29 14:20:51 2014 2014/10/29 14:22:06.923440 [ 8284]: Hung-script: OUTPUT: 2014/10/29 14:22:06.923574 [ 8284]: Hung-script: ===== End of hung script debug for PID="8945", event="monitor" ===== 2014/10/29 14:22:06.924125 [ 8284]: ctdb_kill: trying to kill(8945, 9) a process that does not exist 2014/10/29 14:24:28.029239 [recoverd: 8515]: Taking out recovery lock from recovery daemon 2014/10/29 14:24:28.029289 [recoverd: 8515]: Take the recovery lock 2014/10/29 14:24:28.071502 [ 8284]: Freeze priority 1 2014/10/29 14:24:28.082101 [ 8284]: Freeze priority 2 2014/10/29 14:24:28.082974 [ 8284]: Freeze priority 3 2014/10/29 14:24:30.171332 [ 8284]: Thawing priority 1 2014/10/29 14:24:30.171390 [ 8284]: Release freeze handler for prio 1 2014/10/29 14:24:30.171431 [ 8284]: Thawing priority 2 2014/10/29 14:24:30.171453 [ 8284]: Release freeze handler for prio 2 2014/10/29 14:24:30.171485 [ 8284]: Thawing priority 3 2014/10/29 14:24:30.171505 [ 8284]: Release freeze handler for prio 3 2014/10/29 14:24:30.502276 [recoverd: 8515]: Resetting ban count to 0 for all nodes 2014/10/29 14:24:40.686428 [recoverd: 8515]: server/ctdb_recoverd.c:3933 Remote node:0 has different flags for node 1. It has 0x02 vs our 0x00 2014/10/29 14:24:40.686473 [recoverd: 8515]: Use flags 0x00 from local recmaster node for cluster update of node 1 flags 2014/10/29 14:24:40.687211 [recoverd: 8515]: Taking out recovery lock from recovery daemon 2014/10/29 14:24:40.687241 [recoverd: 8515]: Take the recovery lock 2014/10/29 14:24:40.698006 [ 8284]: Freeze priority 1 2014/10/29 14:24:40.699032 [ 8284]: Freeze priority 2 2014/10/29 14:24:40.699853 [ 8284]: Freeze priority 3 2014/10/29 14:24:43.231676 [ 8284]: Thawing priority 1 2014/10/29 14:24:43.231733 [ 8284]: Release freeze handler for prio 1 2014/10/29 14:24:43.231794 [ 8284]: Thawing priority 2 2014/10/29 14:24:43.231841 [ 8284]: Release freeze handler for prio 2 2014/10/29 14:24:43.231871 [ 8284]: Thawing priority 3 2014/10/29 14:24:43.231889 [ 8284]: Release freeze handler for prio 3 2014/10/29 14:24:43.560511 [recoverd: 8515]: Resetting ban count to 0 for all nodes 2014/10/29 14:25:00.585508 [ 8284]: Monitoring event was cancelled 2014/10/29 14:25:00.585556 [ 8284]: server/eventscript.c:569 Sending SIGTERM to child pid:21463 2014/10/29 14:25:00.654842 [ 8284]: 10.interface: Re-adding secondary address 10.10.10.182/24 to dev bond1 2014/10/29 14:25:01.013086 [ 8284]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 15:32:08.634266 [recoverd: 8515]: server/ctdb_recoverd.c:3960 The vnnmap count is different from the number of active lmaster nodes: 3 vs 2 2014/10/29 15:32:08.634312 [recoverd: 8515]: Taking out recovery lock from recovery daemon 2014/10/29 15:32:08.634326 [recoverd: 8515]: Take the recovery lock 2014/10/29 15:32:08.644807 [ 8284]: Freeze priority 1 2014/10/29 15:32:08.645896 [ 8284]: Freeze priority 2 2014/10/29 15:32:08.646766 [ 8284]: Freeze priority 3 2014/10/29 15:32:10.157111 [ 8284]: Thawing priority 1 2014/10/29 15:32:10.157165 [ 8284]: Release freeze handler for prio 1 2014/10/29 15:32:10.157205 [ 8284]: Thawing priority 2 2014/10/29 15:32:10.157221 [ 8284]: Release freeze handler for prio 2 2014/10/29 15:32:10.157256 [ 8284]: Thawing priority 3 2014/10/29 15:32:10.157271 [ 8284]: Release freeze handler for prio 3 2014/10/29 15:32:10.163833 [set_recmode: 676]: ctdb_recovery_lock: Unable to open /mnt/lock/lockfile - (Stale file handle) 2014/10/29 15:32:10.730682 [recoverd: 8515]: Resetting ban count to 0 for all nodes 2014/10/29 15:33:47.569913 [ 8284]: 50.samba: Redirecting to /bin/systemctl stop smb.service 2014/10/29 15:33:47.612808 [ 8284]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 15:33:47.623810 [ 8284]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 15:47:03.213887 [ 1543]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/29 15:47:03.213984 [ 1543]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/29 15:47:03.213998 [ 1543]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/29 16:04:51.484599 [31052]: Starting CTDBD (Version 2.5.3) as PID: 31052 2014/10/29 16:04:52.482214 [31052]: Vacuuming is disabled for persistent database registry.tdb 2014/10/29 16:04:52.506103 [31052]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/29 16:04:52.520207 [31052]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/29 16:04:52.534119 [31052]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/29 16:04:52.547942 [31052]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/29 16:04:52.561833 [31052]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/29 16:04:52.575691 [31052]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/29 16:04:52.575731 [31052]: Freeze priority 1 2014/10/29 16:04:52.594418 [31052]: Freeze priority 2 2014/10/29 16:04:52.594771 [31052]: Freeze priority 3 2014/10/29 16:04:52.759012 [31052]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/29 16:04:52.762740 [31052]: 00.ctdb: Set RecoverTimeout to 60 2014/10/29 16:04:52.766730 [31052]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/29 16:04:52.887677 [31052]: Freeze priority 1 2014/10/29 16:04:52.887750 [31052]: Freeze priority 2 2014/10/29 16:04:52.887802 [31052]: Freeze priority 3 2014/10/29 16:04:53.188060 [31052]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 16:04:53.299282 [31052]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 16:04:56.894524 [recoverd:31372]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/29 16:04:56.894600 [31052]: Freeze priority 1 2014/10/29 16:04:56.894662 [31052]: Freeze priority 2 2014/10/29 16:04:56.894715 [31052]: Freeze priority 3 2014/10/29 16:05:00.622479 [31052]: Freeze priority 1 2014/10/29 16:05:00.623559 [31052]: Freeze priority 2 2014/10/29 16:05:00.624175 [31052]: Freeze priority 3 2014/10/29 16:05:00.785776 [31052]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/29 16:05:00.786481 [31052]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/29 16:05:02.339456 [31052]: Thawing priority 1 2014/10/29 16:05:02.339514 [31052]: Release freeze handler for prio 1 2014/10/29 16:05:02.339548 [31052]: Thawing priority 2 2014/10/29 16:05:02.339569 [31052]: Release freeze handler for prio 2 2014/10/29 16:05:02.339598 [31052]: Thawing priority 3 2014/10/29 16:05:02.339618 [31052]: Release freeze handler for prio 3 2014/10/29 16:05:16.619296 [31052]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/29 16:05:16.885470 [31052]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 16:05:16.896194 [31052]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 16:05:16.918829 [31052]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/29 16:05:16.924715 [recoverd:31372]: Trigger takeoverrun 2014/10/29 16:05:17.128274 [31052]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/29 16:05:19.551456 [31052]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation 2014/10/29 16:05:20.116198 [31052]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 16:21:43.220828 [31052]: Freeze priority 1 2014/10/29 16:21:43.258151 [31052]: Freeze priority 2 2014/10/29 16:21:43.259643 [31052]: Freeze priority 3 2014/10/29 16:21:44.840755 [31052]: Thawing priority 1 2014/10/29 16:21:44.840806 [31052]: Release freeze handler for prio 1 2014/10/29 16:21:44.840862 [31052]: Thawing priority 2 2014/10/29 16:21:44.840886 [31052]: Release freeze handler for prio 2 2014/10/29 16:21:44.840916 [31052]: Thawing priority 3 2014/10/29 16:21:44.840934 [31052]: Release freeze handler for prio 3 2014/10/29 16:21:45.224989 [31052]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 16:21:55.421110 [31052]: Freeze priority 1 2014/10/29 16:21:55.422390 [31052]: Freeze priority 2 2014/10/29 16:21:55.423374 [31052]: Freeze priority 3 2014/10/29 16:21:56.902704 [recoverd:31372]: server/ctdb_recoverd.c:2343 Reload nodes file from recovery daemon 2014/10/29 16:21:56.929327 [31052]: Thawing priority 1 2014/10/29 16:21:56.929361 [31052]: Release freeze handler for prio 1 2014/10/29 16:21:56.929397 [31052]: Thawing priority 2 2014/10/29 16:21:56.929419 [31052]: Release freeze handler for prio 2 2014/10/29 16:21:56.929449 [31052]: Thawing priority 3 2014/10/29 16:21:56.929469 [31052]: Release freeze handler for prio 3 2014/10/29 16:32:18.159385 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1 2014/10/29 16:32:18.159984 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1 2014/10/29 16:33:53.716041 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1 2014/10/29 16:33:53.725644 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1 2014/10/29 16:35:41.704939 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1 2014/10/29 16:35:41.705245 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1 2014/10/29 16:35:42.653587 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1 2014/10/29 16:35:42.653921 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1 2014/10/29 16:42:44.379423 [31052]: Freeze priority 1 2014/10/29 16:42:44.389703 [31052]: Freeze priority 2 2014/10/29 16:42:44.390459 [31052]: Freeze priority 3 2014/10/29 16:42:47.394200 [recoverd:31372]: Taking out recovery lock from recovery daemon 2014/10/29 16:42:47.394233 [recoverd:31372]: Take the recovery lock 2014/10/29 16:42:47.396711 [31052]: Freeze priority 1 2014/10/29 16:42:47.396770 [31052]: Freeze priority 2 2014/10/29 16:42:47.396811 [31052]: Freeze priority 3 2014/10/29 16:42:50.378398 [31052]: Thawing priority 1 2014/10/29 16:42:50.378444 [31052]: Release freeze handler for prio 1 2014/10/29 16:42:50.378477 [31052]: Thawing priority 2 2014/10/29 16:42:50.378499 [31052]: Release freeze handler for prio 2 2014/10/29 16:42:50.378530 [31052]: Thawing priority 3 2014/10/29 16:42:50.378551 [31052]: Release freeze handler for prio 3 2014/10/29 16:42:50.789394 [31052]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 16:42:50.951608 [recoverd:31372]: Resetting ban count to 0 for all nodes 2014/10/29 16:45:18.758701 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1 2014/10/29 16:45:18.758763 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1 2014/10/29 16:55:04.122418 [recoverd:31372]: Taking out recovery lock from recovery daemon 2014/10/29 16:55:04.122464 [recoverd:31372]: Take the recovery lock 2014/10/29 16:55:04.166436 [31052]: Freeze priority 1 2014/10/29 16:55:04.167362 [31052]: Freeze priority 2 2014/10/29 16:55:04.168040 [31052]: Freeze priority 3 2014/10/29 16:55:05.734658 [31052]: Thawing priority 1 2014/10/29 16:55:05.734703 [31052]: Release freeze handler for prio 1 2014/10/29 16:55:05.734758 [31052]: Thawing priority 2 2014/10/29 16:55:05.734779 [31052]: Release freeze handler for prio 2 2014/10/29 16:55:05.734808 [31052]: Thawing priority 3 2014/10/29 16:55:05.734827 [31052]: Release freeze handler for prio 3 2014/10/29 16:55:06.062160 [recoverd:31372]: Resetting ban count to 0 for all nodes 2014/10/29 16:55:23.112168 [31052]: 10.interface: Killing TCP connection 10.10.10.206:49648 10.10.10.183:445 2014/10/29 16:55:23.135261 [31052]: 10.interface: Killed 1 TCP connections to released IP 10.10.10.183 2014/10/29 16:55:23.483946 [31052]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 16:58:19.321135 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1 2014/10/29 16:58:19.322090 [31052]: server/ctdb_server.c:554 Can not queue packet to DELETED node 1 2014/10/29 17:09:31.669668 [31052]: 10.interface: Re-adding secondary address 10.10.10.182/24 to dev bond1 2014/10/29 17:09:32.020211 [31052]: 50.samba: Redirecting to /bin/systemctl stop smb.service 2014/10/29 17:09:32.063776 [31052]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 17:09:32.074771 [31052]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 17:34:39.850701 [ 9566]: Starting CTDBD (Version 2.5.3) as PID: 9566 2014/10/29 17:34:40.265163 [ 9566]: Ignoring persistent database 'registry.tdb.2' 2014/10/29 17:34:40.265186 [ 9566]: Ignoring persistent database 'passdb.tdb.2' 2014/10/29 17:34:40.265196 [ 9566]: Ignoring persistent database 'secrets.tdb.2' 2014/10/29 17:34:40.265205 [ 9566]: Ignoring persistent database 'share_info.tdb.2' 2014/10/29 17:34:40.265214 [ 9566]: Ignoring persistent database 'ctdb.tdb.2' 2014/10/29 17:34:40.265224 [ 9566]: Ignoring persistent database 'account_policy.tdb.2' 2014/10/29 17:34:40.265233 [ 9566]: Ignoring persistent database 'group_mapping.tdb.2' 2014/10/29 17:34:40.265255 [ 9566]: Freeze priority 1 2014/10/29 17:34:40.265609 [ 9566]: Freeze priority 2 2014/10/29 17:34:40.265939 [ 9566]: Freeze priority 3 2014/10/29 17:34:40.280864 [ 9566]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/29 17:34:40.284109 [ 9566]: 00.ctdb: Set RecoverTimeout to 60 2014/10/29 17:34:40.287224 [ 9566]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/29 17:34:40.409963 [ 9566]: Freeze priority 1 2014/10/29 17:34:40.410056 [ 9566]: Freeze priority 2 2014/10/29 17:34:40.410109 [ 9566]: Freeze priority 3 2014/10/29 17:34:40.658460 [ 9566]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 17:34:40.658487 [ 9566]: Unknown db_id 0x6cf2837d in ctdb_ltdb_update_seqnum 2014/10/29 17:34:43.915647 [recoverd: 9796]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 1) have - force an election 2014/10/29 17:34:43.915766 [ 9566]: Freeze priority 1 2014/10/29 17:34:43.915861 [ 9566]: Freeze priority 2 2014/10/29 17:34:43.915942 [ 9566]: Freeze priority 3 2014/10/29 17:34:49.008816 [ 9566]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/29 17:34:49.027289 [ 9566]: Vacuuming is disabled for persistent database registry.tdb 2014/10/29 17:34:49.047947 [ 9566]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/29 17:34:49.066597 [ 9566]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/29 17:34:49.085280 [ 9566]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/29 17:34:49.104282 [ 9566]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/29 17:34:49.123159 [ 9566]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/29 17:34:49.130130 [ 9566]: Freeze priority 1 2014/10/29 17:34:49.130873 [ 9566]: Freeze priority 2 2014/10/29 17:34:49.131412 [ 9566]: Freeze priority 3 2014/10/29 17:34:49.264825 [ 9566]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/29 17:34:50.635880 [ 9566]: Thawing priority 1 2014/10/29 17:34:50.635920 [ 9566]: Release freeze handler for prio 1 2014/10/29 17:34:50.635950 [ 9566]: Thawing priority 2 2014/10/29 17:34:50.635968 [ 9566]: Release freeze handler for prio 2 2014/10/29 17:34:50.635992 [ 9566]: Thawing priority 3 2014/10/29 17:34:50.636009 [ 9566]: Release freeze handler for prio 3 2014/10/29 17:35:04.951870 [ 9566]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/29 17:35:05.140320 [ 9566]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 17:35:05.153538 [ 9566]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 17:35:05.184185 [ 9566]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/29 17:35:05.388373 [ 9566]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/29 17:35:05.448499 [recoverd: 9796]: Trigger takeoverrun 2014/10/29 17:35:07.656092 [ 9566]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation 2014/10/29 17:35:08.389715 [ 9566]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 18:37:49.090781 [ 9566]: Freeze priority 1 2014/10/29 18:37:49.121643 [ 9566]: Freeze priority 2 2014/10/29 18:37:49.122481 [ 9566]: Freeze priority 3 2014/10/29 18:37:52.128366 [recoverd: 9796]: Taking out recovery lock from recovery daemon 2014/10/29 18:37:52.128408 [recoverd: 9796]: Take the recovery lock 2014/10/29 18:37:52.131497 [ 9566]: Freeze priority 1 2014/10/29 18:37:52.131565 [ 9566]: Freeze priority 2 2014/10/29 18:37:52.131616 [ 9566]: Freeze priority 3 2014/10/29 18:37:54.590220 [ 9566]: Thawing priority 1 2014/10/29 18:37:54.590270 [ 9566]: Release freeze handler for prio 1 2014/10/29 18:37:54.590304 [ 9566]: Thawing priority 2 2014/10/29 18:37:54.590326 [ 9566]: Release freeze handler for prio 2 2014/10/29 18:37:54.590357 [ 9566]: Thawing priority 3 2014/10/29 18:37:54.590376 [ 9566]: Release freeze handler for prio 3 2014/10/29 18:37:55.028359 [ 9566]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 18:37:55.208120 [recoverd: 9796]: Resetting ban count to 0 for all nodes 2014/10/29 19:20:53.298177 [recoverd: 9796]: Taking out recovery lock from recovery daemon 2014/10/29 19:20:53.298245 [recoverd: 9796]: Take the recovery lock 2014/10/29 19:20:53.461598 [ 9566]: Freeze priority 1 2014/10/29 19:20:53.462346 [ 9566]: Freeze priority 2 2014/10/29 19:20:53.462908 [ 9566]: Freeze priority 3 2014/10/29 19:20:55.096956 [ 9566]: Thawing priority 1 2014/10/29 19:20:55.097078 [ 9566]: Release freeze handler for prio 1 2014/10/29 19:20:55.097139 [ 9566]: Thawing priority 2 2014/10/29 19:20:55.097161 [ 9566]: Release freeze handler for prio 2 2014/10/29 19:20:55.097202 [ 9566]: Thawing priority 3 2014/10/29 19:20:55.097216 [ 9566]: Release freeze handler for prio 3 2014/10/29 19:20:55.424596 [recoverd: 9796]: Resetting ban count to 0 for all nodes 2014/10/29 19:21:12.521040 [ 9566]: 10.interface: Re-adding secondary address 10.10.10.182/24 to dev bond1 2014/10/29 19:21:12.937205 [ 9566]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 19:30:42.335813 [recoverd: 9796]: server/ctdb_recoverd.c:3960 The vnnmap count is different from the number of active lmaster nodes: 2 vs 1 2014/10/29 19:30:42.335874 [recoverd: 9796]: Taking out recovery lock from recovery daemon 2014/10/29 19:30:42.335889 [recoverd: 9796]: Take the recovery lock 2014/10/29 19:30:42.341301 [ 9566]: Freeze priority 1 2014/10/29 19:30:42.342120 [ 9566]: Freeze priority 2 2014/10/29 19:30:42.342707 [ 9566]: Freeze priority 3 2014/10/29 19:30:43.702157 [ 9566]: Thawing priority 1 2014/10/29 19:30:43.702199 [ 9566]: Release freeze handler for prio 1 2014/10/29 19:30:43.702229 [ 9566]: Thawing priority 2 2014/10/29 19:30:43.702246 [ 9566]: Release freeze handler for prio 2 2014/10/29 19:30:43.702272 [ 9566]: Thawing priority 3 2014/10/29 19:30:43.702288 [ 9566]: Release freeze handler for prio 3 2014/10/29 19:30:43.956338 [ 9566]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 19:30:44.114648 [recoverd: 9796]: Resetting ban count to 0 for all nodes 2014/10/29 19:30:50.588104 [ 9566]: 50.samba: Redirecting to /bin/systemctl stop smb.service 2014/10/29 19:30:50.631129 [ 9566]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 19:30:50.641948 [ 9566]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 19:36:42.884469 [19587]: Starting CTDBD (Version 2.5.3) as PID: 19587 2014/10/29 19:36:44.482193 [19587]: Vacuuming is disabled for persistent database registry.tdb 2014/10/29 19:36:44.508792 [19587]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/29 19:36:44.523263 [19587]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/29 19:36:44.537928 [19587]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/29 19:36:44.537981 [19587]: Ignoring persistent database 'account_policy.tdb.1' 2014/10/29 19:36:44.537991 [19587]: Ignoring persistent database 'ctdb.tdb.1' 2014/10/29 19:36:44.538000 [19587]: Ignoring persistent database 'group_mapping.tdb.1' 2014/10/29 19:36:44.538009 [19587]: Ignoring persistent database 'secrets.tdb.1' 2014/10/29 19:36:44.538018 [19587]: Ignoring persistent database 'share_info.tdb.1' 2014/10/29 19:36:44.553007 [19587]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/29 19:36:44.567757 [19587]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/29 19:36:44.567805 [19587]: Ignoring persistent database 'passdb.tdb.1' 2014/10/29 19:36:44.567815 [19587]: Ignoring persistent database 'registry.tdb.1' 2014/10/29 19:36:44.582528 [19587]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/29 19:36:44.582598 [19587]: Freeze priority 1 2014/10/29 19:36:44.587185 [19587]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 19:36:44.593743 [19587]: Freeze priority 2 2014/10/29 19:36:44.594143 [19587]: Freeze priority 3 2014/10/29 19:36:44.758454 [19587]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/29 19:36:44.762491 [19587]: 00.ctdb: Set RecoverTimeout to 60 2014/10/29 19:36:44.766251 [19587]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/29 19:36:44.892084 [19587]: Freeze priority 1 2014/10/29 19:36:44.892171 [19587]: Freeze priority 2 2014/10/29 19:36:44.892238 [19587]: Freeze priority 3 2014/10/29 19:36:48.397613 [recoverd:19916]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/29 19:36:48.397708 [19587]: Freeze priority 1 2014/10/29 19:36:48.397768 [19587]: Freeze priority 2 2014/10/29 19:36:48.397831 [19587]: Freeze priority 3 2014/10/29 19:36:52.028422 [19587]: Freeze priority 1 2014/10/29 19:36:52.029314 [19587]: Freeze priority 2 2014/10/29 19:36:52.030042 [19587]: Freeze priority 3 2014/10/29 19:36:52.190865 [19587]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/29 19:36:52.191667 [19587]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/29 19:36:53.532585 [19587]: Thawing priority 1 2014/10/29 19:36:53.532640 [19587]: Release freeze handler for prio 1 2014/10/29 19:36:53.532675 [19587]: Thawing priority 2 2014/10/29 19:36:53.532708 [19587]: Release freeze handler for prio 2 2014/10/29 19:36:53.532736 [19587]: Thawing priority 3 2014/10/29 19:36:53.532766 [19587]: Release freeze handler for prio 3 2014/10/29 19:37:07.852270 [19587]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/29 19:37:08.138667 [19587]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 19:37:08.149968 [19587]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 19:37:08.179908 [19587]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/29 19:37:08.428002 [recoverd:19916]: Trigger takeoverrun 2014/10/29 19:37:08.824127 [19587]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/29 19:37:11.423222 [19587]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation 2014/10/29 19:37:12.375109 [19587]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 19:38:46.055746 [19587]: Freeze priority 1 2014/10/29 19:38:46.057420 [19587]: Freeze priority 2 2014/10/29 19:38:46.058510 [19587]: Freeze priority 3 2014/10/29 19:38:46.364846 [recoverd:19916]: server/ctdb_recoverd.c:2343 Reload nodes file from recovery daemon 2014/10/29 19:38:47.588413 [19587]: Thawing priority 1 2014/10/29 19:38:47.588455 [19587]: Release freeze handler for prio 1 2014/10/29 19:38:47.588493 [19587]: Thawing priority 2 2014/10/29 19:38:47.588516 [19587]: Release freeze handler for prio 2 2014/10/29 19:38:47.588558 [19587]: Thawing priority 3 2014/10/29 19:38:47.588578 [19587]: Release freeze handler for prio 3 2014/10/29 19:38:47.605766 [19587]: Freeze priority 1 2014/10/29 19:38:47.606796 [19587]: Freeze priority 2 2014/10/29 19:38:47.607622 [19587]: Freeze priority 3 2014/10/29 19:38:49.126353 [19587]: Thawing priority 1 2014/10/29 19:38:49.126407 [19587]: Release freeze handler for prio 1 2014/10/29 19:38:49.126439 [19587]: Thawing priority 2 2014/10/29 19:38:49.126471 [19587]: Release freeze handler for prio 2 2014/10/29 19:38:49.126501 [19587]: Thawing priority 3 2014/10/29 19:38:49.126520 [19587]: Release freeze handler for prio 3 2014/10/29 19:38:59.631216 [19587]: Freeze priority 1 2014/10/29 19:38:59.632494 [19587]: Freeze priority 2 2014/10/29 19:38:59.633461 [19587]: Freeze priority 3 2014/10/29 19:39:01.163936 [19587]: Thawing priority 1 2014/10/29 19:39:01.164025 [19587]: Release freeze handler for prio 1 2014/10/29 19:39:01.164086 [19587]: Thawing priority 2 2014/10/29 19:39:01.164108 [19587]: Release freeze handler for prio 2 2014/10/29 19:39:01.164146 [19587]: Thawing priority 3 2014/10/29 19:39:01.164164 [19587]: Release freeze handler for prio 3 2014/10/29 19:51:22.676194 [19587]: Freeze priority 1 2014/10/29 19:51:22.701308 [19587]: Freeze priority 2 2014/10/29 19:51:22.704192 [19587]: Freeze priority 3 2014/10/29 19:51:24.633925 [19587]: Thawing priority 1 2014/10/29 19:51:24.633971 [19587]: Release freeze handler for prio 1 2014/10/29 19:51:24.634018 [19587]: Thawing priority 2 2014/10/29 19:51:24.634036 [19587]: Release freeze handler for prio 2 2014/10/29 19:51:24.634062 [19587]: Thawing priority 3 2014/10/29 19:51:24.634078 [19587]: Release freeze handler for prio 3 2014/10/29 19:57:52.819632 [19587]: Freeze priority 1 2014/10/29 19:57:52.835994 [19587]: Freeze priority 2 2014/10/29 19:57:52.837507 [19587]: Freeze priority 3 2014/10/29 19:57:56.080585 [19587]: Thawing priority 1 2014/10/29 19:57:56.080620 [19587]: Release freeze handler for prio 1 2014/10/29 19:57:56.080657 [19587]: Thawing priority 2 2014/10/29 19:57:56.080676 [19587]: Release freeze handler for prio 2 2014/10/29 19:57:56.080706 [19587]: Thawing priority 3 2014/10/29 19:57:56.080720 [19587]: Release freeze handler for prio 3 2014/10/29 19:58:06.470038 [19587]: Freeze priority 1 2014/10/29 19:58:06.472536 [19587]: Freeze priority 2 2014/10/29 19:58:06.476259 [19587]: Freeze priority 3 2014/10/29 19:58:09.343004 [19587]: Thawing priority 1 2014/10/29 19:58:09.343059 [19587]: Release freeze handler for prio 1 2014/10/29 19:58:09.343088 [19587]: Thawing priority 2 2014/10/29 19:58:09.343106 [19587]: Release freeze handler for prio 2 2014/10/29 19:58:09.343131 [19587]: Thawing priority 3 2014/10/29 19:58:09.343147 [19587]: Release freeze handler for prio 3 2014/10/29 20:07:56.594391 [19587]: Freeze priority 1 2014/10/29 20:07:56.607629 [19587]: Freeze priority 1 2014/10/29 20:07:56.609710 [19587]: Freeze priority 1 2014/10/29 20:07:56.610131 [19587]: Freeze priority 2 2014/10/29 20:07:56.610454 [19587]: Freeze priority 2 2014/10/29 20:07:56.610973 [19587]: Freeze priority 2 2014/10/29 20:07:56.611333 [19587]: Freeze priority 3 2014/10/29 20:07:56.611578 [19587]: Freeze priority 3 2014/10/29 20:07:56.611633 [19587]: Freeze priority 3 2014/10/29 20:07:59.638236 [19587]: Freeze priority 1 2014/10/29 20:07:59.638876 [19587]: Freeze priority 2 2014/10/29 20:07:59.639232 [19587]: Freeze priority 3 2014/10/29 20:08:01.721927 [19587]: Thawing priority 1 2014/10/29 20:08:01.721980 [19587]: Release freeze handler for prio 1 2014/10/29 20:08:01.722028 [19587]: Thawing priority 2 2014/10/29 20:08:01.722050 [19587]: Release freeze handler for prio 2 2014/10/29 20:08:01.722090 [19587]: Thawing priority 3 2014/10/29 20:08:01.722113 [19587]: Release freeze handler for prio 3 2014/10/29 20:08:01.723472 [19587]: server/ctdb_call.c:1005 reqid 50163 not found 2014/10/29 20:08:01.723518 [19587]: server/ctdb_call.c:1005 reqid 50164 not found 2014/10/29 20:13:17.913201 [19587]: Freeze priority 1 2014/10/29 20:13:17.930023 [19587]: Freeze priority 2 2014/10/29 20:13:17.931784 [19587]: Freeze priority 3 2014/10/29 20:13:20.606663 [19587]: Thawing priority 1 2014/10/29 20:13:20.606716 [19587]: Release freeze handler for prio 1 2014/10/29 20:13:20.606752 [19587]: Thawing priority 2 2014/10/29 20:13:20.606772 [19587]: Release freeze handler for prio 2 2014/10/29 20:13:20.606799 [19587]: Thawing priority 3 2014/10/29 20:13:20.606815 [19587]: Release freeze handler for prio 3 2014/10/29 20:18:01.185968 [19587]: Freeze priority 1 2014/10/29 20:18:01.200509 [19587]: Freeze priority 2 2014/10/29 20:18:01.201704 [19587]: Freeze priority 3 2014/10/29 20:18:01.958173 [19587]: Freeze priority 1 2014/10/29 20:18:01.958561 [19587]: Freeze priority 2 2014/10/29 20:18:01.962885 [19587]: Freeze priority 3 2014/10/29 20:18:01.964849 [19587]: Freeze priority 1 2014/10/29 20:18:01.965385 [19587]: Freeze priority 2 2014/10/29 20:18:01.965786 [19587]: Freeze priority 3 2014/10/29 20:18:05.979314 [recoverd:19916]: Taking out recovery lock from recovery daemon 2014/10/29 20:18:05.979384 [recoverd:19916]: Take the recovery lock 2014/10/29 20:18:05.990859 [19587]: Freeze priority 1 2014/10/29 20:18:05.991291 [19587]: Freeze priority 2 2014/10/29 20:18:05.991732 [19587]: Freeze priority 3 2014/10/29 20:18:07.677249 [19587]: Thawing priority 1 2014/10/29 20:18:07.677316 [19587]: Release freeze handler for prio 1 2014/10/29 20:18:07.677356 [19587]: Thawing priority 2 2014/10/29 20:18:07.677378 [19587]: Release freeze handler for prio 2 2014/10/29 20:18:07.677454 [19587]: Thawing priority 3 2014/10/29 20:18:07.677471 [19587]: Release freeze handler for prio 3 2014/10/29 20:18:08.265992 [recoverd:19916]: Resetting ban count to 0 for all nodes 2014/10/29 20:24:35.795558 [recoverd:19916]: Taking out recovery lock from recovery daemon 2014/10/29 20:24:35.803344 [recoverd:19916]: Take the recovery lock 2014/10/29 20:24:35.955913 [19587]: Freeze priority 1 2014/10/29 20:24:35.960140 [19587]: Freeze priority 2 2014/10/29 20:24:35.964264 [19587]: Freeze priority 3 2014/10/29 20:24:38.101830 [19587]: Thawing priority 1 2014/10/29 20:24:38.101882 [19587]: Release freeze handler for prio 1 2014/10/29 20:24:38.101917 [19587]: Thawing priority 2 2014/10/29 20:24:38.101939 [19587]: Release freeze handler for prio 2 2014/10/29 20:24:38.101969 [19587]: Thawing priority 3 2014/10/29 20:24:38.101989 [19587]: Release freeze handler for prio 3 2014/10/29 20:24:38.453923 [recoverd:19916]: Resetting ban count to 0 for all nodes 2014/10/29 20:24:48.663352 [recoverd:19916]: server/ctdb_recoverd.c:3933 Remote node:1 has different flags for node 0. It has 0x02 vs our 0x00 2014/10/29 20:24:48.663405 [recoverd:19916]: Use flags 0x00 from local recmaster node for cluster update of node 0 flags 2014/10/29 20:24:48.665287 [recoverd:19916]: Taking out recovery lock from recovery daemon 2014/10/29 20:24:48.665304 [recoverd:19916]: Take the recovery lock 2014/10/29 20:24:48.717549 [19587]: Freeze priority 1 2014/10/29 20:24:48.721595 [19587]: Freeze priority 2 2014/10/29 20:24:48.723699 [19587]: Freeze priority 3 2014/10/29 20:24:50.768575 [19587]: Thawing priority 1 2014/10/29 20:24:50.768613 [19587]: Release freeze handler for prio 1 2014/10/29 20:24:50.768654 [19587]: Thawing priority 2 2014/10/29 20:24:50.768672 [19587]: Release freeze handler for prio 2 2014/10/29 20:24:50.768702 [19587]: Thawing priority 3 2014/10/29 20:24:50.768718 [19587]: Release freeze handler for prio 3 2014/10/29 20:24:51.119071 [recoverd:19916]: Resetting ban count to 0 for all nodes 2014/10/29 20:28:05.383085 [recovery-lock:28201]: failed read from recovery_lock_fd - Transport endpoint is not connected 2014/10/29 20:28:05.383160 [recoverd:19916]: server/ctdb_recoverd.c:3349 reclock child process returned error 2 2014/10/29 20:28:05.383189 [recoverd:19916]: server/ctdb_recoverd.c:3449 reclock child failed when checking file 2014/10/29 20:28:05.383272 [recoverd:19916]: Failed check_recovery_lock. Force a recovery 2014/10/29 20:28:05.383291 [recoverd:19916]: Taking out recovery lock from recovery daemon 2014/10/29 20:28:05.383301 [recoverd:19916]: Take the recovery lock 2014/10/29 20:28:05.421263 [19587]: Freeze priority 1 2014/10/29 20:28:05.424264 [19587]: Freeze priority 2 2014/10/29 20:28:05.426321 [19587]: Freeze priority 3 2014/10/29 20:28:05.562588 [19587]: pnn 2 Invalid reqid 201907 in ctdb_reply_control 2014/10/29 20:28:05.567449 [19587]: pnn 2 Invalid reqid 201906 in ctdb_reply_control 2014/10/29 20:28:05.608227 [19587]: pnn 2 Invalid reqid 201908 in ctdb_reply_control 2014/10/29 20:28:05.622262 [19587]: pnn 2 Invalid reqid 201905 in ctdb_reply_control 2014/10/29 20:28:06.548117 [19587]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 20:28:06.559023 [19587]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 20:29:33.146117 [ 1500]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/29 20:29:33.146219 [ 1500]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/29 20:29:33.146233 [ 1500]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/29 20:33:33.828865 [ 7703]: Starting CTDBD (Version 2.5.3) as PID: 7703 2014/10/29 20:33:35.385619 [ 7703]: Vacuuming is disabled for persistent database registry.tdb 2014/10/29 20:33:35.409261 [ 7703]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/29 20:33:35.423401 [ 7703]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/29 20:33:35.437382 [ 7703]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/29 20:33:35.437400 [ 7703]: Ignoring persistent database 'account_policy.tdb.1' 2014/10/29 20:33:35.437409 [ 7703]: Ignoring persistent database 'ctdb.tdb.1' 2014/10/29 20:33:35.437418 [ 7703]: Ignoring persistent database 'group_mapping.tdb.1' 2014/10/29 20:33:35.437437 [ 7703]: Ignoring persistent database 'secrets.tdb.1' 2014/10/29 20:33:35.437447 [ 7703]: Ignoring persistent database 'share_info.tdb.1' 2014/10/29 20:33:35.451438 [ 7703]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/29 20:33:35.465479 [ 7703]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/29 20:33:35.465498 [ 7703]: Ignoring persistent database 'passdb.tdb.1' 2014/10/29 20:33:35.465507 [ 7703]: Ignoring persistent database 'registry.tdb.1' 2014/10/29 20:33:35.479491 [ 7703]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/29 20:33:35.479524 [ 7703]: Freeze priority 1 2014/10/29 20:33:35.491053 [ 7703]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 20:33:35.497643 [ 7703]: Freeze priority 2 2014/10/29 20:33:35.498034 [ 7703]: Freeze priority 3 2014/10/29 20:33:35.661354 [ 7703]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 20:33:35.662102 [ 7703]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/29 20:33:35.665682 [ 7703]: 00.ctdb: Set RecoverTimeout to 60 2014/10/29 20:33:35.669034 [ 7703]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/29 20:33:35.732269 [ 7703]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 20:33:35.789485 [ 7703]: Freeze priority 1 2014/10/29 20:33:35.789557 [ 7703]: Freeze priority 2 2014/10/29 20:33:35.789611 [ 7703]: Freeze priority 3 2014/10/29 20:33:39.797430 [recoverd: 7987]: server/ctdb_recoverd.c:3692 Current recmaster node 3 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/29 20:33:39.797509 [ 7703]: Freeze priority 1 2014/10/29 20:33:39.797572 [ 7703]: Freeze priority 2 2014/10/29 20:33:39.797626 [ 7703]: Freeze priority 3 2014/10/29 20:33:44.593367 [ 7703]: Freeze priority 1 2014/10/29 20:33:44.628185 [ 7703]: Freeze priority 2 2014/10/29 20:33:44.629196 [ 7703]: Freeze priority 3 2014/10/29 20:33:44.790635 [ 7703]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/29 20:33:44.791300 [ 7703]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/29 20:33:44.792238 [ 7703]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup 2014/10/29 20:33:46.662165 [ 7703]: Thawing priority 1 2014/10/29 20:33:46.662216 [ 7703]: Release freeze handler for prio 1 2014/10/29 20:33:46.662257 [ 7703]: Thawing priority 2 2014/10/29 20:33:46.662277 [ 7703]: Release freeze handler for prio 2 2014/10/29 20:33:46.662304 [ 7703]: Thawing priority 3 2014/10/29 20:33:46.662322 [ 7703]: Release freeze handler for prio 3 2014/10/29 20:34:00.838744 [recoverd: 7987]: Trigger takeoverrun 2014/10/29 20:34:01.547095 [ 7703]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/29 20:34:01.869872 [ 7703]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 20:34:01.880712 [ 7703]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 20:34:01.927504 [ 7703]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/29 20:34:02.102264 [ 7703]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/29 20:34:04.552433 [ 7703]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation 2014/10/29 20:34:05.654833 [ 7703]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 20:38:07.987015 [ 7703]: Freeze priority 1 2014/10/29 20:38:07.988688 [ 7703]: Freeze priority 1 2014/10/29 20:38:08.133702 [ 7703]: Freeze priority 1 2014/10/29 20:38:09.870904 [ 7703]: Skip monitoring since databases are frozen 2014/10/29 20:38:24.871170 [ 7703]: Skip monitoring since databases are frozen 2014/10/29 20:38:39.871412 [ 7703]: Skip monitoring since databases are frozen 2014/10/29 20:38:54.871914 [ 7703]: Skip monitoring since databases are frozen 2014/10/29 20:39:07.990231 [ 7703]: Freeze priority 1 2014/10/29 20:39:07.991166 [ 7703]: Freeze priority 1 2014/10/29 20:39:08.132278 [ 7703]: Recovery daemon ping timeout. Count : 0 2014/10/29 20:39:08.134454 [recoverd: 7987]: ctdb_control error: 'ctdb_control timed out' 2014/10/29 20:39:08.134515 [recoverd: 7987]: ctdb_control error: 'ctdb_control timed out' 2014/10/29 20:39:08.134539 [recoverd: 7987]: Async operation failed with ret=-1 res=-1 opcode=33 2014/10/29 20:39:08.134555 [recoverd: 7987]: Failed to freeze node 0 during recovery. Set it as ban culprit for 4 credits 2014/10/29 20:39:08.134573 [recoverd: 7987]: Async wait failed - fail_count=1 2014/10/29 20:39:08.134587 [recoverd: 7987]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed. 2014/10/29 20:39:08.134602 [recoverd: 7987]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster 2014/10/29 20:39:08.136121 [ 7703]: Freeze priority 1 2014/10/29 20:39:09.872470 [ 7703]: Skip monitoring since databases are frozen 2014/10/29 20:39:24.873425 [ 7703]: Skip monitoring since databases are frozen 2014/10/29 20:39:39.873551 [ 7703]: Skip monitoring since databases are frozen 2014/10/29 20:39:54.873814 [ 7703]: Skip monitoring since databases are frozen 2014/10/29 20:40:08.134995 [ 7703]: Recovery daemon ping timeout. Count : 0 2014/10/29 20:40:08.136375 [recoverd: 7987]: ctdb_control error: 'ctdb_control timed out' 2014/10/29 20:40:08.136410 [recoverd: 7987]: ctdb_control error: 'ctdb_control timed out' 2014/10/29 20:40:08.136426 [recoverd: 7987]: Async operation failed with ret=-1 res=-1 opcode=33 2014/10/29 20:40:08.136437 [recoverd: 7987]: Failed to freeze node 0 during recovery. Set it as ban culprit for 4 credits 2014/10/29 20:40:08.136450 [recoverd: 7987]: Async wait failed - fail_count=1 2014/10/29 20:40:08.136461 [recoverd: 7987]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed. 2014/10/29 20:40:08.136472 [recoverd: 7987]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster 2014/10/29 20:40:08.136836 [ 7703]: Freeze priority 1 2014/10/29 20:40:08.138207 [ 7703]: Freeze priority 1 2014/10/29 20:40:08.145633 [ 7703]: pnn 2 Invalid reqid 12817 in ctdb_reply_control 2014/10/29 20:40:08.145673 [ 7703]: pnn 2 Invalid reqid 12618 in ctdb_reply_control 2014/10/29 20:40:08.145731 [ 7703]: Freeze priority 2 2014/10/29 20:40:08.145881 [ 7703]: Freeze priority 2 2014/10/29 20:40:08.146649 [ 7703]: Freeze priority 3 2014/10/29 20:40:08.146787 [ 7703]: Freeze priority 3 2014/10/29 20:40:11.166825 [ 7703]: Freeze priority 1 2014/10/29 20:40:11.167189 [ 7703]: Freeze priority 2 2014/10/29 20:40:11.167450 [ 7703]: Freeze priority 3 2014/10/29 20:40:12.931813 [ 7703]: Thawing priority 1 2014/10/29 20:40:12.931857 [ 7703]: Release freeze handler for prio 1 2014/10/29 20:40:12.931891 [ 7703]: Thawing priority 2 2014/10/29 20:40:12.931910 [ 7703]: Release freeze handler for prio 2 2014/10/29 20:40:12.931939 [ 7703]: Thawing priority 3 2014/10/29 20:40:12.931956 [ 7703]: Release freeze handler for prio 3 2014/10/29 20:40:13.383420 [ 7703]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 20:40:38.747328 [ 7703]: Freeze priority 1 2014/10/29 20:40:38.762646 [ 7703]: Freeze priority 2 2014/10/29 20:40:38.769499 [ 7703]: Freeze priority 3 2014/10/29 20:40:41.763907 [ 7703]: Thawing priority 1 2014/10/29 20:40:41.763959 [ 7703]: Release freeze handler for prio 1 2014/10/29 20:40:41.764002 [ 7703]: Thawing priority 2 2014/10/29 20:40:41.764026 [ 7703]: Release freeze handler for prio 2 2014/10/29 20:40:41.764071 [ 7703]: Thawing priority 3 2014/10/29 20:40:41.764090 [ 7703]: Release freeze handler for prio 3 2014/10/29 20:40:41.877602 [ 7703]: 10.interface: Killing TCP connection 10.10.10.205:54678 10.10.10.184:445 2014/10/29 20:40:41.884118 [ 7703]: 10.interface: Killed 1 TCP connections to released IP 10.10.10.184 2014/10/29 20:40:41.893310 [ 7703]: 10.interface: Re-adding secondary address 10.10.10.182/24 to dev bond1 2014/10/29 20:40:42.347878 [ 7703]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 20:43:15.815605 [ 7703]: Freeze priority 1 2014/10/29 20:43:15.830273 [ 7703]: Freeze priority 2 2014/10/29 20:43:15.831558 [ 7703]: Freeze priority 3 2014/10/29 20:43:18.927342 [ 7703]: Thawing priority 1 2014/10/29 20:43:18.927384 [ 7703]: Release freeze handler for prio 1 2014/10/29 20:43:18.927427 [ 7703]: Thawing priority 2 2014/10/29 20:43:18.927449 [ 7703]: Release freeze handler for prio 2 2014/10/29 20:43:18.927478 [ 7703]: Thawing priority 3 2014/10/29 20:43:18.927497 [ 7703]: Release freeze handler for prio 3 2014/10/29 20:43:36.724040 [ 7703]: Monitoring event was cancelled 2014/10/29 20:53:15.123874 [ 7703]: Freeze priority 1 2014/10/29 20:53:27.745637 [ 7703]: Skip monitoring since databases are frozen 2014/10/29 20:53:42.746568 [ 7703]: Skip monitoring since databases are frozen 2014/10/29 20:53:57.747386 [ 7703]: Skip monitoring since databases are frozen 2014/10/29 20:54:12.748302 [ 7703]: Skip monitoring since databases are frozen 2014/10/29 20:54:15.142595 [ 7703]: Freeze priority 1 2014/10/29 20:54:27.748974 [ 7703]: Skip monitoring since databases are frozen 2014/10/29 20:54:42.750051 [ 7703]: Skip monitoring since databases are frozen 2014/10/29 20:54:57.750707 [ 7703]: Skip monitoring since databases are frozen 2014/10/29 20:55:12.751741 [ 7703]: Skip monitoring since databases are frozen 2014/10/29 20:55:15.323543 [ 7703]: Freeze priority 1 2014/10/29 20:55:15.323863 [ 7703]: Freeze priority 2 2014/10/29 20:55:15.336130 [ 7703]: Freeze priority 3 2014/10/29 20:55:17.469643 [ 7703]: Thawing priority 1 2014/10/29 20:55:17.469685 [ 7703]: Release freeze handler for prio 1 2014/10/29 20:55:17.469717 [ 7703]: Thawing priority 2 2014/10/29 20:55:17.469738 [ 7703]: Release freeze handler for prio 2 2014/10/29 20:55:17.469770 [ 7703]: Thawing priority 3 2014/10/29 20:55:17.469799 [ 7703]: Release freeze handler for prio 3 2014/10/29 20:55:17.955612 [ 7703]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 20:55:45.609546 [ 7703]: Freeze priority 1 2014/10/29 20:55:45.619347 [ 7703]: Freeze priority 2 2014/10/29 20:55:45.622235 [ 7703]: Freeze priority 3 2014/10/29 20:55:47.896212 [ 7703]: Thawing priority 1 2014/10/29 20:55:47.896270 [ 7703]: Release freeze handler for prio 1 2014/10/29 20:55:47.896301 [ 7703]: Thawing priority 2 2014/10/29 20:55:47.896319 [ 7703]: Release freeze handler for prio 2 2014/10/29 20:55:47.896358 [ 7703]: Thawing priority 3 2014/10/29 20:55:47.896375 [ 7703]: Release freeze handler for prio 3 2014/10/29 20:55:47.980705 [ 7703]: 10.interface: Killing TCP connection 10.10.10.206:49659 10.10.10.183:445 2014/10/29 20:55:47.980836 [ 7703]: 10.interface: Killing TCP connection 10.10.10.205:54682 10.10.10.183:445 2014/10/29 20:55:47.980937 [ 7703]: 10.interface: Killing TCP connection 10.10.10.208:49559 10.10.10.183:445 2014/10/29 20:55:47.997639 [ 7703]: 10.interface: Killed 3 TCP connections to released IP 10.10.10.183 2014/10/29 20:55:48.412545 [ 7703]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 20:58:35.926501 [ 7703]: Freeze priority 1 2014/10/29 20:58:35.962450 [ 7703]: Freeze priority 2 2014/10/29 20:58:35.964598 [ 7703]: Freeze priority 3 2014/10/29 20:58:38.293222 [ 7703]: Thawing priority 1 2014/10/29 20:58:38.293261 [ 7703]: Release freeze handler for prio 1 2014/10/29 20:58:38.293291 [ 7703]: Thawing priority 2 2014/10/29 20:58:38.293320 [ 7703]: Release freeze handler for prio 2 2014/10/29 20:58:38.293350 [ 7703]: Thawing priority 3 2014/10/29 20:58:38.293366 [ 7703]: Release freeze handler for prio 3 2014/10/29 20:58:52.711848 [ 7703]: Monitoring event was cancelled 2014/10/29 20:58:52.711911 [ 7703]: server/eventscript.c:569 Sending SIGTERM to child pid:22264 2014/10/29 21:03:19.688685 [ 7703]: Freeze priority 1 2014/10/29 21:03:19.701810 [ 7703]: Freeze priority 1 2014/10/29 21:03:19.705917 [ 7703]: Freeze priority 2 2014/10/29 21:03:19.709275 [ 7703]: Freeze priority 2 2014/10/29 21:03:19.710564 [ 7703]: Freeze priority 3 2014/10/29 21:03:19.715646 [ 7703]: Freeze priority 3 2014/10/29 21:03:22.728977 [recoverd: 7987]: Taking out recovery lock from recovery daemon 2014/10/29 21:03:22.729042 [recoverd: 7987]: Take the recovery lock 2014/10/29 21:03:22.742293 [ 7703]: Freeze priority 1 2014/10/29 21:03:22.742744 [ 7703]: Freeze priority 2 2014/10/29 21:03:22.743127 [ 7703]: Freeze priority 3 2014/10/29 21:03:24.968853 [ 7703]: Thawing priority 1 2014/10/29 21:03:24.968895 [ 7703]: Release freeze handler for prio 1 2014/10/29 21:03:24.968925 [ 7703]: Thawing priority 2 2014/10/29 21:03:24.968941 [ 7703]: Release freeze handler for prio 2 2014/10/29 21:03:24.968975 [ 7703]: Thawing priority 3 2014/10/29 21:03:24.968990 [ 7703]: Release freeze handler for prio 3 2014/10/29 21:03:25.536232 [recoverd: 7987]: Resetting ban count to 0 for all nodes 2014/10/29 21:09:52.683379 [ 7703]: Freeze priority 1 2014/10/29 21:09:52.758389 [ 7703]: Freeze priority 2 2014/10/29 21:09:52.762013 [ 7703]: Freeze priority 3 2014/10/29 21:09:55.000036 [ 7703]: Thawing priority 1 2014/10/29 21:09:55.000091 [ 7703]: Release freeze handler for prio 1 2014/10/29 21:09:55.000145 [ 7703]: Thawing priority 2 2014/10/29 21:09:55.000173 [ 7703]: Release freeze handler for prio 2 2014/10/29 21:09:55.000219 [ 7703]: Thawing priority 3 2014/10/29 21:09:55.000239 [ 7703]: Release freeze handler for prio 3 2014/10/29 21:13:20.539806 [ 7703]: 50.samba: ERROR: samba tcp port 445 is not responding 2014/10/29 21:13:20.551343 [ 7703]: 50.samba: Redirecting to /bin/systemctl restart smb.service 2014/10/29 21:13:26.475886 [ 7703]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 21:13:26.489390 [ 7703]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 21:14:55.894453 [ 1545]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/29 21:14:55.894586 [ 1545]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/29 21:14:55.894607 [ 1545]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/29 21:18:57.121914 [ 7733]: Starting CTDBD (Version 2.5.3) as PID: 7733 2014/10/29 21:18:58.522610 [ 7733]: Vacuuming is disabled for persistent database registry.tdb 2014/10/29 21:18:58.547699 [ 7733]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/29 21:18:58.562058 [ 7733]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/29 21:18:58.576048 [ 7733]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/29 21:18:58.576066 [ 7733]: Ignoring persistent database 'account_policy.tdb.1' 2014/10/29 21:18:58.576076 [ 7733]: Ignoring persistent database 'ctdb.tdb.1' 2014/10/29 21:18:58.576085 [ 7733]: Ignoring persistent database 'group_mapping.tdb.1' 2014/10/29 21:18:58.576094 [ 7733]: Ignoring persistent database 'secrets.tdb.1' 2014/10/29 21:18:58.576103 [ 7733]: Ignoring persistent database 'share_info.tdb.1' 2014/10/29 21:18:58.590021 [ 7733]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/29 21:18:58.604179 [ 7733]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/29 21:18:58.604198 [ 7733]: Ignoring persistent database 'passdb.tdb.1' 2014/10/29 21:18:58.604208 [ 7733]: Ignoring persistent database 'registry.tdb.1' 2014/10/29 21:18:58.618380 [ 7733]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/29 21:18:58.618433 [ 7733]: Freeze priority 1 2014/10/29 21:18:58.632379 [ 7733]: Freeze priority 2 2014/10/29 21:18:58.632742 [ 7733]: Freeze priority 3 2014/10/29 21:18:58.796205 [ 7733]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 21:18:58.796293 [ 7733]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 21:18:58.796315 [ 7733]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 21:18:58.797398 [ 7733]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/29 21:18:58.801127 [ 7733]: 00.ctdb: Set RecoverTimeout to 60 2014/10/29 21:18:58.804912 [ 7733]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/29 21:18:58.931054 [ 7733]: Freeze priority 1 2014/10/29 21:18:58.931132 [ 7733]: Freeze priority 2 2014/10/29 21:18:58.931186 [ 7733]: Freeze priority 3 2014/10/29 21:18:59.074947 [ 7733]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 21:18:59.074996 [ 7733]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 21:18:59.075019 [ 7733]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 21:18:59.075056 [ 7733]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 21:18:59.075205 [recoverd: 8019]: server/ctdb_recoverd.c:1058 Unable to find db_id 0xaf029e9d on local node 2014/10/29 21:18:59.115692 [ 7733]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 21:18:59.115805 [recoverd: 8019]: server/ctdb_recoverd.c:1058 Unable to find db_id 0xaf029e9d on local node 2014/10/29 21:18:59.115832 [recoverd: 8019]: server/ctdb_recoverd.c:1058 Unable to find db_id 0x6afb8c09 on local node 2014/10/29 21:18:59.115845 [recoverd: 8019]: server/ctdb_recoverd.c:1058 Unable to find db_id 0x4e66c2b2 on local node 2014/10/29 21:18:59.115856 [recoverd: 8019]: server/ctdb_recoverd.c:1058 Unable to find db_id 0x6afb8c09 on local node 2014/10/29 21:18:59.115869 [recoverd: 8019]: server/ctdb_recoverd.c:1058 Unable to find db_id 0xaf029e9d on local node 2014/10/29 21:19:03.438871 [recoverd: 8019]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/29 21:19:03.438927 [ 7733]: Freeze priority 1 2014/10/29 21:19:03.438986 [ 7733]: Freeze priority 2 2014/10/29 21:19:03.439038 [ 7733]: Freeze priority 3 2014/10/29 21:19:07.568045 [ 7733]: Freeze priority 1 2014/10/29 21:19:07.584929 [ 7733]: Freeze priority 2 2014/10/29 21:19:07.586052 [ 7733]: Freeze priority 3 2014/10/29 21:19:07.927762 [ 7733]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/29 21:19:07.928596 [ 7733]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/29 21:19:07.929713 [ 7733]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup 2014/10/29 21:19:13.202457 [ 7733]: Handling event took 4 seconds! 2014/10/29 21:19:13.214740 [ 7733]: Thawing priority 1 2014/10/29 21:19:13.214790 [ 7733]: Release freeze handler for prio 1 2014/10/29 21:19:13.214825 [ 7733]: Thawing priority 2 2014/10/29 21:19:13.214844 [ 7733]: Release freeze handler for prio 2 2014/10/29 21:19:13.214871 [ 7733]: Thawing priority 3 2014/10/29 21:19:13.214890 [ 7733]: Release freeze handler for prio 3 2014/10/29 21:19:28.229571 [recoverd: 8019]: Trigger takeoverrun 2014/10/29 21:19:28.654289 [ 7733]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/29 21:19:28.937295 [ 7733]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 21:19:28.951032 [ 7733]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 21:19:29.077939 [ 7733]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/29 21:19:29.282157 [ 7733]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/29 21:19:32.658785 [ 7733]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation 2014/10/29 21:19:33.785605 [ 7733]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 21:23:30.683205 [ 7733]: Freeze priority 1 2014/10/29 21:23:30.691674 [ 7733]: Freeze priority 2 2014/10/29 21:23:30.692923 [ 7733]: Freeze priority 3 2014/10/29 21:23:32.390311 [ 7733]: Thawing priority 1 2014/10/29 21:23:32.390360 [ 7733]: Release freeze handler for prio 1 2014/10/29 21:23:32.390391 [ 7733]: Thawing priority 2 2014/10/29 21:23:32.390420 [ 7733]: Release freeze handler for prio 2 2014/10/29 21:23:32.390455 [ 7733]: Thawing priority 3 2014/10/29 21:23:32.390474 [ 7733]: Release freeze handler for prio 3 2014/10/29 21:28:42.503439 [ 7733]: Freeze priority 1 2014/10/29 21:28:42.550319 [ 7733]: Freeze priority 2 2014/10/29 21:28:42.551750 [ 7733]: Freeze priority 3 2014/10/29 21:28:48.214657 [ 7733]: Thawing priority 1 2014/10/29 21:28:48.214712 [ 7733]: Release freeze handler for prio 1 2014/10/29 21:28:48.214755 [ 7733]: Thawing priority 2 2014/10/29 21:28:48.214774 [ 7733]: Release freeze handler for prio 2 2014/10/29 21:28:48.214800 [ 7733]: Thawing priority 3 2014/10/29 21:28:48.214827 [ 7733]: Release freeze handler for prio 3 2014/10/29 21:38:44.119638 [recoverd: 8019]: ctdb_control error: 'node is disconnected' 2014/10/29 21:38:44.184309 [recoverd: 8019]: client/ctdb_client.c:1535 ctdb_control for getnodes failed ret:-1 res:-1 2014/10/29 21:38:44.184348 [recoverd: 8019]: server/ctdb_recoverd.c:3739 Unable to get nodemap from recovery master 0 2014/10/29 21:38:44.188220 [ 7733]: Freeze priority 1 2014/10/29 21:38:44.306435 [ 7733]: Freeze priority 2 2014/10/29 21:38:44.318573 [ 7733]: Freeze priority 3 2014/10/29 21:38:48.102867 [ 7733]: Freeze priority 1 2014/10/29 21:38:48.103309 [ 7733]: Freeze priority 2 2014/10/29 21:38:48.103699 [ 7733]: Freeze priority 3 2014/10/29 21:38:50.545125 [ 7733]: Thawing priority 1 2014/10/29 21:38:50.545182 [ 7733]: Release freeze handler for prio 1 2014/10/29 21:38:50.545229 [ 7733]: Thawing priority 2 2014/10/29 21:38:50.545250 [ 7733]: Release freeze handler for prio 2 2014/10/29 21:38:50.545293 [ 7733]: Thawing priority 3 2014/10/29 21:38:50.545315 [ 7733]: Release freeze handler for prio 3 2014/10/29 21:44:21.726587 [ 7733]: Freeze priority 1 2014/10/29 21:44:21.833622 [ 7733]: Freeze priority 2 2014/10/29 21:44:21.835851 [ 7733]: Freeze priority 3 2014/10/29 21:44:27.149173 [ 7733]: Thawing priority 1 2014/10/29 21:44:27.149206 [ 7733]: Release freeze handler for prio 1 2014/10/29 21:44:27.149249 [ 7733]: Thawing priority 2 2014/10/29 21:44:27.149272 [ 7733]: Release freeze handler for prio 2 2014/10/29 21:44:27.149312 [ 7733]: Thawing priority 3 2014/10/29 21:44:27.149332 [ 7733]: Release freeze handler for prio 3 2014/10/29 21:48:49.509367 [ 7733]: Freeze priority 1 2014/10/29 21:48:49.546322 [ 7733]: Freeze priority 1 2014/10/29 21:48:49.613279 [ 7733]: Freeze priority 2 2014/10/29 21:48:49.616277 [ 7733]: Freeze priority 2 2014/10/29 21:48:49.618482 [ 7733]: Freeze priority 3 2014/10/29 21:48:49.621113 [ 7733]: Freeze priority 3 2014/10/29 21:48:53.135398 [recoverd: 8019]: Taking out recovery lock from recovery daemon 2014/10/29 21:48:53.135459 [recoverd: 8019]: Take the recovery lock 2014/10/29 21:48:53.162481 [ 7733]: Freeze priority 1 2014/10/29 21:48:53.162905 [ 7733]: Freeze priority 2 2014/10/29 21:48:53.163261 [ 7733]: Freeze priority 3 2014/10/29 21:48:55.633723 [ 7733]: Thawing priority 1 2014/10/29 21:48:55.633763 [ 7733]: Release freeze handler for prio 1 2014/10/29 21:48:55.633793 [ 7733]: Thawing priority 2 2014/10/29 21:48:55.633823 [ 7733]: Release freeze handler for prio 2 2014/10/29 21:48:55.633851 [ 7733]: Thawing priority 3 2014/10/29 21:48:55.633867 [ 7733]: Release freeze handler for prio 3 2014/10/29 21:48:56.125072 [recoverd: 8019]: Resetting ban count to 0 for all nodes 2014/10/29 21:55:25.664194 [recoverd: 8019]: Taking out recovery lock from recovery daemon 2014/10/29 21:55:25.680005 [recoverd: 8019]: Take the recovery lock 2014/10/29 21:55:25.894437 [ 7733]: Freeze priority 1 2014/10/29 21:55:25.971850 [ 7733]: Freeze priority 2 2014/10/29 21:55:25.974183 [ 7733]: Freeze priority 3 2014/10/29 21:55:28.587627 [ 7733]: Thawing priority 1 2014/10/29 21:55:28.587672 [ 7733]: Release freeze handler for prio 1 2014/10/29 21:55:28.587708 [ 7733]: Thawing priority 2 2014/10/29 21:55:28.587727 [ 7733]: Release freeze handler for prio 2 2014/10/29 21:55:28.587754 [ 7733]: Thawing priority 3 2014/10/29 21:55:28.587770 [ 7733]: Release freeze handler for prio 3 2014/10/29 21:55:29.050719 [recoverd: 8019]: Resetting ban count to 0 for all nodes 2014/10/29 21:58:57.499627 [ 7733]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 21:58:57.562483 [ 7733]: common/ctdb_fork.c:131 waitpid() returned error. errno:10 2014/10/29 21:58:57.562514 [ 7733]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 22:00:22.787574 [ 1544]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/29 22:00:22.787679 [ 1544]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/29 22:00:22.787693 [ 1544]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/29 22:04:24.642995 [ 7695]: Starting CTDBD (Version 2.5.3) as PID: 7695 2014/10/29 22:04:26.406387 [ 7695]: Vacuuming is disabled for persistent database registry.tdb 2014/10/29 22:04:26.429877 [ 7695]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/29 22:04:26.444024 [ 7695]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/29 22:04:26.457952 [ 7695]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/29 22:04:26.457970 [ 7695]: Ignoring persistent database 'account_policy.tdb.1' 2014/10/29 22:04:26.457978 [ 7695]: Ignoring persistent database 'ctdb.tdb.1' 2014/10/29 22:04:26.457987 [ 7695]: Ignoring persistent database 'group_mapping.tdb.1' 2014/10/29 22:04:26.457995 [ 7695]: Ignoring persistent database 'secrets.tdb.1' 2014/10/29 22:04:26.458003 [ 7695]: Ignoring persistent database 'share_info.tdb.1' 2014/10/29 22:04:26.471922 [ 7695]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/29 22:04:26.485876 [ 7695]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/29 22:04:26.485895 [ 7695]: Ignoring persistent database 'passdb.tdb.1' 2014/10/29 22:04:26.485903 [ 7695]: Ignoring persistent database 'registry.tdb.1' 2014/10/29 22:04:26.499876 [ 7695]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/29 22:04:26.499909 [ 7695]: Freeze priority 1 2014/10/29 22:04:26.518342 [ 7695]: Freeze priority 2 2014/10/29 22:04:26.518693 [ 7695]: Freeze priority 3 2014/10/29 22:04:26.682010 [ 7695]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 22:04:26.682978 [ 7695]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/29 22:04:26.686767 [ 7695]: 00.ctdb: Set RecoverTimeout to 60 2014/10/29 22:04:26.690108 [ 7695]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/29 22:04:26.809850 [ 7695]: Freeze priority 1 2014/10/29 22:04:26.809925 [ 7695]: Freeze priority 2 2014/10/29 22:04:26.809981 [ 7695]: Freeze priority 3 2014/10/29 22:04:30.817245 [recoverd: 8069]: server/ctdb_recoverd.c:3692 Current recmaster node 3 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/29 22:04:30.872895 [ 7695]: Freeze priority 1 2014/10/29 22:04:30.873007 [ 7695]: Freeze priority 2 2014/10/29 22:04:30.873082 [ 7695]: Freeze priority 3 2014/10/29 22:04:34.989486 [ 7695]: Freeze priority 1 2014/10/29 22:04:35.013016 [ 7695]: Freeze priority 2 2014/10/29 22:04:35.015139 [ 7695]: Freeze priority 3 2014/10/29 22:04:35.183761 [ 7695]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/29 22:04:35.184917 [ 7695]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/29 22:04:35.186958 [ 7695]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup 2014/10/29 22:04:39.731992 [ 7695]: Handling event took 4 seconds! 2014/10/29 22:04:39.737870 [ 7695]: Thawing priority 1 2014/10/29 22:04:39.737924 [ 7695]: Release freeze handler for prio 1 2014/10/29 22:04:39.737971 [ 7695]: Thawing priority 2 2014/10/29 22:04:39.738001 [ 7695]: Release freeze handler for prio 2 2014/10/29 22:04:39.738044 [ 7695]: Thawing priority 3 2014/10/29 22:04:39.738062 [ 7695]: Release freeze handler for prio 3 2014/10/29 22:04:54.759553 [recoverd: 8069]: Trigger takeoverrun 2014/10/29 22:04:54.891173 [ 7695]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/29 22:04:55.332796 [ 7695]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 22:04:55.346536 [ 7695]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 22:04:55.381160 [ 7695]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/29 22:04:55.560516 [ 7695]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/29 22:04:57.993373 [ 7695]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation 2014/10/29 22:04:58.607435 [ 7695]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 22:09:02.089179 [ 7695]: Freeze priority 1 2014/10/29 22:09:02.095662 [ 7695]: Freeze priority 1 2014/10/29 22:09:02.105818 [ 7695]: Freeze priority 1 2014/10/29 22:09:04.215481 [ 7695]: Skip monitoring since databases are frozen 2014/10/29 22:09:19.216271 [ 7695]: Skip monitoring since databases are frozen 2014/10/29 22:09:34.216980 [ 7695]: Skip monitoring since databases are frozen 2014/10/29 22:09:49.217970 [ 7695]: Skip monitoring since databases are frozen 2014/10/29 22:10:02.092533 [ 7695]: Freeze priority 1 2014/10/29 22:10:02.095650 [ 7695]: Recovery daemon ping timeout. Count : 0 2014/10/29 22:10:02.095760 [recoverd: 8069]: ctdb_control error: 'ctdb_control timed out' 2014/10/29 22:10:02.095794 [recoverd: 8069]: ctdb_control error: 'ctdb_control timed out' 2014/10/29 22:10:02.095811 [recoverd: 8069]: Async operation failed with ret=-1 res=-1 opcode=33 2014/10/29 22:10:02.095823 [recoverd: 8069]: Failed to freeze node 0 during recovery. Set it as ban culprit for 4 credits 2014/10/29 22:10:02.095837 [recoverd: 8069]: Async wait failed - fail_count=1 2014/10/29 22:10:02.095860 [recoverd: 8069]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed. 2014/10/29 22:10:02.095875 [recoverd: 8069]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster 2014/10/29 22:10:02.097114 [ 7695]: Freeze priority 1 2014/10/29 22:10:02.107117 [ 7695]: Freeze priority 1 2014/10/29 22:10:04.218509 [ 7695]: Skip monitoring since databases are frozen 2014/10/29 22:10:19.218773 [ 7695]: Skip monitoring since databases are frozen 2014/10/29 22:10:34.219457 [ 7695]: Skip monitoring since databases are frozen 2014/10/29 22:10:49.220503 [ 7695]: Skip monitoring since databases are frozen 2014/10/29 22:11:02.096430 [ 7695]: Recovery daemon ping timeout. Count : 0 2014/10/29 22:11:02.097581 [recoverd: 8069]: ctdb_control error: 'ctdb_control timed out' 2014/10/29 22:11:02.097618 [recoverd: 8069]: ctdb_control error: 'ctdb_control timed out' 2014/10/29 22:11:02.097634 [recoverd: 8069]: Async operation failed with ret=-1 res=-1 opcode=33 2014/10/29 22:11:02.097645 [recoverd: 8069]: Failed to freeze node 0 during recovery. Set it as ban culprit for 4 credits 2014/10/29 22:11:02.097657 [recoverd: 8069]: Async wait failed - fail_count=1 2014/10/29 22:11:02.097668 [recoverd: 8069]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed. 2014/10/29 22:11:02.097678 [recoverd: 8069]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster 2014/10/29 22:11:02.171284 [ 7695]: pnn 2 Invalid reqid 7683 in ctdb_reply_control 2014/10/29 22:11:02.171321 [ 7695]: pnn 2 Invalid reqid 7474 in ctdb_reply_control 2014/10/29 22:11:02.174281 [ 7695]: Freeze priority 1 2014/10/29 22:11:02.174889 [ 7695]: Freeze priority 2 2014/10/29 22:11:02.176002 [ 7695]: Freeze priority 3 2014/10/29 22:11:02.282396 [ 7695]: Freeze priority 1 2014/10/29 22:11:02.282894 [ 7695]: Freeze priority 2 2014/10/29 22:11:02.283393 [ 7695]: Freeze priority 3 2014/10/29 22:11:05.300510 [ 7695]: Freeze priority 1 2014/10/29 22:11:05.300694 [ 7695]: Freeze priority 2 2014/10/29 22:11:05.300903 [ 7695]: Freeze priority 3 2014/10/29 22:11:08.089106 [ 7695]: Thawing priority 1 2014/10/29 22:11:08.089157 [ 7695]: Release freeze handler for prio 1 2014/10/29 22:11:08.089204 [ 7695]: Thawing priority 2 2014/10/29 22:11:08.089238 [ 7695]: Release freeze handler for prio 2 2014/10/29 22:11:08.089278 [ 7695]: Thawing priority 3 2014/10/29 22:11:08.089299 [ 7695]: Release freeze handler for prio 3 2014/10/29 22:11:08.091596 [ 7695]: server/ctdb_call.c:1005 reqid 7956 not found 2014/10/29 22:11:08.091645 [ 7695]: server/ctdb_call.c:1005 reqid 7957 not found 2014/10/29 22:11:08.091668 [ 7695]: server/ctdb_call.c:1005 reqid 7958 not found 2014/10/29 22:11:08.091689 [ 7695]: server/ctdb_call.c:1005 reqid 7959 not found 2014/10/29 22:11:08.501152 [ 7695]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 22:11:32.728170 [ 7695]: Freeze priority 1 2014/10/29 22:11:32.735922 [ 7695]: Freeze priority 2 2014/10/29 22:11:32.737233 [ 7695]: Freeze priority 3 2014/10/29 22:11:37.307586 [ 7695]: Thawing priority 1 2014/10/29 22:11:37.307639 [ 7695]: Release freeze handler for prio 1 2014/10/29 22:11:37.307670 [ 7695]: Thawing priority 2 2014/10/29 22:11:37.307687 [ 7695]: Release freeze handler for prio 2 2014/10/29 22:11:37.307716 [ 7695]: Thawing priority 3 2014/10/29 22:11:37.307732 [ 7695]: Release freeze handler for prio 3 2014/10/29 22:11:37.964911 [ 7695]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 22:14:07.433182 [vacuum-locking.tdb: 4972]: Error storing record copies on node 3: ret[0] res[-1] 2014/10/29 22:14:11.442373 [ 7695]: Freeze priority 1 2014/10/29 22:14:11.495362 [ 7695]: Freeze priority 2 2014/10/29 22:14:11.497606 [ 7695]: Freeze priority 3 2014/10/29 22:14:15.511627 [ 7695]: Thawing priority 1 2014/10/29 22:14:15.511691 [ 7695]: Release freeze handler for prio 1 2014/10/29 22:14:15.511729 [ 7695]: Thawing priority 2 2014/10/29 22:14:15.511747 [ 7695]: Release freeze handler for prio 2 2014/10/29 22:14:15.511774 [ 7695]: Thawing priority 3 2014/10/29 22:14:15.511791 [ 7695]: Release freeze handler for prio 3 2014/10/29 22:24:11.727635 [ 7695]: Freeze priority 1 2014/10/29 22:24:11.835898 [ 7695]: Freeze priority 2 2014/10/29 22:24:11.837055 [ 7695]: Freeze priority 3 2014/10/29 22:24:14.602534 [ 7695]: Thawing priority 1 2014/10/29 22:24:14.602598 [ 7695]: Release freeze handler for prio 1 2014/10/29 22:24:14.602642 [ 7695]: Thawing priority 2 2014/10/29 22:24:14.602660 [ 7695]: Release freeze handler for prio 2 2014/10/29 22:24:14.602686 [ 7695]: Thawing priority 3 2014/10/29 22:24:14.602701 [ 7695]: Release freeze handler for prio 3 2014/10/29 22:29:28.932187 [ 7695]: Freeze priority 1 2014/10/29 22:29:29.052498 [ 7695]: Freeze priority 2 2014/10/29 22:29:29.054779 [ 7695]: Freeze priority 3 2014/10/29 22:29:34.468102 [ 7695]: Thawing priority 1 2014/10/29 22:29:34.468149 [ 7695]: Release freeze handler for prio 1 2014/10/29 22:29:34.468183 [ 7695]: Thawing priority 2 2014/10/29 22:29:34.468200 [ 7695]: Release freeze handler for prio 2 2014/10/29 22:29:34.468227 [ 7695]: Thawing priority 3 2014/10/29 22:29:34.468243 [ 7695]: Release freeze handler for prio 3 2014/10/29 22:34:20.295762 [ 7695]: Freeze priority 1 2014/10/29 22:34:20.597091 [ 7695]: Freeze priority 1 2014/10/29 22:34:20.597650 [ 7695]: Freeze priority 2 2014/10/29 22:34:20.600329 [ 7695]: Freeze priority 3 2014/10/29 22:34:23.612575 [recoverd: 8069]: Taking out recovery lock from recovery daemon 2014/10/29 22:34:23.612621 [recoverd: 8069]: Take the recovery lock 2014/10/29 22:34:23.653004 [ 7695]: Freeze priority 1 2014/10/29 22:34:23.653841 [ 7695]: Freeze priority 2 2014/10/29 22:34:23.655403 [ 7695]: Freeze priority 3 2014/10/29 22:34:32.997051 [ 7695]: Handling event took 6 seconds! 2014/10/29 22:34:33.002155 [ 7695]: Thawing priority 1 2014/10/29 22:34:33.002183 [ 7695]: Release freeze handler for prio 1 2014/10/29 22:34:33.002213 [ 7695]: Thawing priority 2 2014/10/29 22:34:33.002235 [ 7695]: Release freeze handler for prio 2 2014/10/29 22:34:33.002265 [ 7695]: Thawing priority 3 2014/10/29 22:34:33.002282 [ 7695]: Release freeze handler for prio 3 2014/10/29 22:34:33.004932 [ 7695]: pnn 2 Invalid reqid 64206 in ctdb_become_dmaster from node 0 2014/10/29 22:34:33.005017 [ 7695]: server/ctdb_call.c:1005 reqid 64207 not found 2014/10/29 22:34:33.597318 [recoverd: 8069]: Resetting ban count to 0 for all nodes 2014/10/29 22:41:01.148617 [recoverd: 8069]: Taking out recovery lock from recovery daemon 2014/10/29 22:41:01.168964 [recoverd: 8069]: Take the recovery lock 2014/10/29 22:41:01.386753 [ 7695]: Freeze priority 1 2014/10/29 22:41:01.435934 [ 7695]: Freeze priority 2 2014/10/29 22:41:01.440804 [ 7695]: Freeze priority 3 2014/10/29 22:41:05.960833 [ 7695]: Thawing priority 1 2014/10/29 22:41:05.960897 [ 7695]: Release freeze handler for prio 1 2014/10/29 22:41:05.960936 [ 7695]: Thawing priority 2 2014/10/29 22:41:05.960955 [ 7695]: Release freeze handler for prio 2 2014/10/29 22:41:05.960983 [ 7695]: Thawing priority 3 2014/10/29 22:41:05.961000 [ 7695]: Release freeze handler for prio 3 2014/10/29 22:41:06.336975 [recoverd: 8069]: Resetting ban count to 0 for all nodes 2014/10/29 22:41:23.735790 [ 7695]: Monitoring event was cancelled 2014/10/29 22:41:23.735866 [ 7695]: server/eventscript.c:569 Sending SIGTERM to child pid:22549 2014/10/29 22:44:30.967991 [ 7695]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 22:44:31.028272 [ 7695]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 22:45:56.442406 [ 1547]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/29 22:45:56.442515 [ 1547]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/29 22:45:56.522882 [ 1547]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/29 22:49:57.568477 [ 7729]: Starting CTDBD (Version 2.5.3) as PID: 7729 2014/10/29 22:49:59.050435 [ 7729]: Vacuuming is disabled for persistent database registry.tdb 2014/10/29 22:49:59.073952 [ 7729]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/29 22:49:59.088288 [ 7729]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/29 22:49:59.102299 [ 7729]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/29 22:49:59.102318 [ 7729]: Ignoring persistent database 'account_policy.tdb.1' 2014/10/29 22:49:59.102327 [ 7729]: Ignoring persistent database 'ctdb.tdb.1' 2014/10/29 22:49:59.102336 [ 7729]: Ignoring persistent database 'group_mapping.tdb.1' 2014/10/29 22:49:59.102344 [ 7729]: Ignoring persistent database 'secrets.tdb.1' 2014/10/29 22:49:59.102353 [ 7729]: Ignoring persistent database 'share_info.tdb.1' 2014/10/29 22:49:59.116256 [ 7729]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/29 22:49:59.130246 [ 7729]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/29 22:49:59.130264 [ 7729]: Ignoring persistent database 'passdb.tdb.1' 2014/10/29 22:49:59.130273 [ 7729]: Ignoring persistent database 'registry.tdb.1' 2014/10/29 22:49:59.144231 [ 7729]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/29 22:49:59.144261 [ 7729]: Freeze priority 1 2014/10/29 22:49:59.162305 [ 7729]: Freeze priority 2 2014/10/29 22:49:59.162649 [ 7729]: Freeze priority 3 2014/10/29 22:49:59.326011 [ 7729]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 22:49:59.327161 [ 7729]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/29 22:49:59.330731 [ 7729]: 00.ctdb: Set RecoverTimeout to 60 2014/10/29 22:49:59.334264 [ 7729]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/29 22:49:59.454141 [ 7729]: Freeze priority 1 2014/10/29 22:49:59.454220 [ 7729]: Freeze priority 2 2014/10/29 22:49:59.454274 [ 7729]: Freeze priority 3 2014/10/29 22:49:59.875144 [ 7729]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 22:49:59.889926 [ 7729]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 22:50:03.461069 [recoverd: 8044]: server/ctdb_recoverd.c:3692 Current recmaster node 3 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/29 22:50:03.461151 [ 7729]: Freeze priority 1 2014/10/29 22:50:03.461224 [ 7729]: Freeze priority 2 2014/10/29 22:50:03.461285 [ 7729]: Freeze priority 3 2014/10/29 22:50:07.979011 [ 7729]: Freeze priority 1 2014/10/29 22:50:08.005448 [ 7729]: Freeze priority 2 2014/10/29 22:50:08.011733 [ 7729]: Freeze priority 3 2014/10/29 22:50:08.190335 [ 7729]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/29 22:50:08.195204 [ 7729]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/29 22:50:08.204382 [ 7729]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup 2014/10/29 22:50:10.797286 [ 7729]: Thawing priority 1 2014/10/29 22:50:10.797325 [ 7729]: Release freeze handler for prio 1 2014/10/29 22:50:10.797358 [ 7729]: Thawing priority 2 2014/10/29 22:50:10.797379 [ 7729]: Release freeze handler for prio 2 2014/10/29 22:50:10.797405 [ 7729]: Thawing priority 3 2014/10/29 22:50:10.797424 [ 7729]: Release freeze handler for prio 3 2014/10/29 22:50:24.816503 [recoverd: 8044]: Trigger takeoverrun 2014/10/29 22:50:25.402247 [ 7729]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/29 22:50:25.924084 [ 7729]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 22:50:25.924127 [ 7729]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 22:50:25.924143 [ 7729]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/29 22:50:25.936175 [ 7729]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/29 22:50:28.408966 [ 7729]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation 2014/10/29 22:50:29.641802 [ 7729]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 22:54:35.142201 [ 7729]: Freeze priority 1 2014/10/29 22:54:35.152126 [ 7729]: Freeze priority 1 2014/10/29 22:54:35.169292 [ 7729]: Freeze priority 1 2014/10/29 22:54:35.175011 [ 7729]: Freeze priority 2 2014/10/29 22:54:35.175283 [ 7729]: Freeze priority 2 2014/10/29 22:54:35.175457 [ 7729]: Freeze priority 2 2014/10/29 22:54:35.176254 [ 7729]: Freeze priority 3 2014/10/29 22:54:35.178201 [ 7729]: Freeze priority 3 2014/10/29 22:54:35.178352 [ 7729]: Freeze priority 3 2014/10/29 22:54:38.207528 [ 7729]: Freeze priority 1 2014/10/29 22:54:38.207817 [ 7729]: Freeze priority 2 2014/10/29 22:54:38.208081 [ 7729]: Freeze priority 3 2014/10/29 22:54:41.337852 [ 7729]: Thawing priority 1 2014/10/29 22:54:41.337907 [ 7729]: Release freeze handler for prio 1 2014/10/29 22:54:41.337939 [ 7729]: Thawing priority 2 2014/10/29 22:54:41.337960 [ 7729]: Release freeze handler for prio 2 2014/10/29 22:54:41.337991 [ 7729]: Thawing priority 3 2014/10/29 22:54:41.338011 [ 7729]: Release freeze handler for prio 3 2014/10/29 22:59:38.444753 [ 7729]: Freeze priority 1 2014/10/29 22:59:38.455755 [ 7729]: Freeze priority 2 2014/10/29 22:59:38.460534 [ 7729]: Freeze priority 3 2014/10/29 22:59:41.557976 [ 7729]: Thawing priority 1 2014/10/29 22:59:41.558028 [ 7729]: Release freeze handler for prio 1 2014/10/29 22:59:41.558064 [ 7729]: Thawing priority 2 2014/10/29 22:59:41.558084 [ 7729]: Release freeze handler for prio 2 2014/10/29 22:59:41.558113 [ 7729]: Thawing priority 3 2014/10/29 22:59:41.558131 [ 7729]: Release freeze handler for prio 3 2014/10/29 23:09:35.333728 [ 7729]: Freeze priority 1 2014/10/29 23:09:35.336905 [ 7729]: Freeze priority 1 2014/10/29 23:09:35.343959 [ 7729]: Freeze priority 2 2014/10/29 23:09:35.346200 [ 7729]: Freeze priority 2 2014/10/29 23:09:35.347036 [ 7729]: Freeze priority 3 2014/10/29 23:09:35.350977 [ 7729]: Freeze priority 3 2014/10/29 23:09:38.372768 [ 7729]: Freeze priority 1 2014/10/29 23:09:38.373171 [ 7729]: Freeze priority 2 2014/10/29 23:09:38.373455 [ 7729]: Freeze priority 3 2014/10/29 23:09:41.167856 [ 7729]: Thawing priority 1 2014/10/29 23:09:41.167907 [ 7729]: Release freeze handler for prio 1 2014/10/29 23:09:41.167944 [ 7729]: Thawing priority 2 2014/10/29 23:09:41.167964 [ 7729]: Release freeze handler for prio 2 2014/10/29 23:09:41.167992 [ 7729]: Thawing priority 3 2014/10/29 23:09:41.168011 [ 7729]: Release freeze handler for prio 3 2014/10/29 23:14:50.614114 [recoverd: 8044]: Taking out recovery lock from recovery daemon 2014/10/29 23:14:50.614200 [recoverd: 8044]: Take the recovery lock 2014/10/29 23:14:50.786906 [ 7729]: Freeze priority 1 2014/10/29 23:14:50.803207 [ 7729]: Freeze priority 2 2014/10/29 23:14:50.807766 [ 7729]: Freeze priority 3 2014/10/29 23:14:53.623758 [ 7729]: Thawing priority 1 2014/10/29 23:14:53.623821 [ 7729]: Release freeze handler for prio 1 2014/10/29 23:14:53.623869 [ 7729]: Thawing priority 2 2014/10/29 23:14:53.623889 [ 7729]: Release freeze handler for prio 2 2014/10/29 23:14:53.623915 [ 7729]: Thawing priority 3 2014/10/29 23:14:53.623930 [ 7729]: Release freeze handler for prio 3 2014/10/29 23:14:53.631706 [recoverd: 8044]: ctdb_control error: 'managed to lock reclock file from inside daemon' 2014/10/29 23:14:53.631746 [recoverd: 8044]: ctdb_control error: 'managed to lock reclock file from inside daemon' 2014/10/29 23:14:53.631765 [recoverd: 8044]: Async operation failed with ret=-1 res=-1 opcode=16 2014/10/29 23:14:53.634033 [set_recmode:20412]: ERROR: recovery lock file /mnt/lock/lockfile not locked when recovering! 2014/10/29 23:14:53.634232 [recoverd: 8044]: ctdb_control error: 'managed to lock reclock file from inside daemon' 2014/10/29 23:14:53.634266 [recoverd: 8044]: ctdb_control error: 'managed to lock reclock file from inside daemon' 2014/10/29 23:14:53.634290 [recoverd: 8044]: Async operation failed with ret=-1 res=-1 opcode=16 2014/10/29 23:14:53.634334 [recoverd: 8044]: Async wait failed - fail_count=2 2014/10/29 23:14:53.634360 [recoverd: 8044]: server/ctdb_recoverd.c:412 Unable to set recovery mode. Recovery failed. 2014/10/29 23:14:53.634380 [recoverd: 8044]: server/ctdb_recoverd.c:1996 Unable to set recovery mode to normal on cluster 2014/10/29 23:14:56.659434 [recoverd: 8044]: Taking out recovery lock from recovery daemon 2014/10/29 23:14:56.659501 [recoverd: 8044]: Take the recovery lock 2014/10/29 23:14:56.708790 [ 7729]: Freeze priority 1 2014/10/29 23:14:56.710013 [ 7729]: Freeze priority 2 2014/10/29 23:14:56.710958 [ 7729]: Freeze priority 3 2014/10/29 23:14:59.321285 [ 7729]: Thawing priority 1 2014/10/29 23:14:59.321341 [ 7729]: Release freeze handler for prio 1 2014/10/29 23:14:59.321377 [ 7729]: Thawing priority 2 2014/10/29 23:14:59.321395 [ 7729]: Release freeze handler for prio 2 2014/10/29 23:14:59.321425 [ 7729]: Thawing priority 3 2014/10/29 23:14:59.321441 [ 7729]: Release freeze handler for prio 3 2014/10/29 23:14:59.735550 [recoverd: 8044]: Resetting ban count to 0 for all nodes 2014/10/29 23:19:41.107313 [recoverd: 8044]: server/ctdb_recoverd.c:3960 The vnnmap count is different from the number of active lmaster nodes: 4 vs 3 2014/10/29 23:19:41.121047 [recoverd: 8044]: Taking out recovery lock from recovery daemon 2014/10/29 23:19:41.121072 [recoverd: 8044]: Take the recovery lock 2014/10/29 23:19:41.176354 [ 7729]: Freeze priority 1 2014/10/29 23:19:41.236485 [ 7729]: Freeze priority 2 2014/10/29 23:19:41.239826 [ 7729]: Freeze priority 3 2014/10/29 23:19:44.020922 [ 7729]: Thawing priority 1 2014/10/29 23:19:44.020972 [ 7729]: Release freeze handler for prio 1 2014/10/29 23:19:44.021023 [ 7729]: Thawing priority 2 2014/10/29 23:19:44.021043 [ 7729]: Release freeze handler for prio 2 2014/10/29 23:19:44.021072 [ 7729]: Thawing priority 3 2014/10/29 23:19:44.021089 [ 7729]: Release freeze handler for prio 3 2014/10/29 23:19:44.025304 [ 7729]: pnn 2 Invalid reqid 57283 in ctdb_become_dmaster from node 3 2014/10/29 23:19:44.025987 [ 7729]: server/ctdb_call.c:1005 reqid 57284 not found 2014/10/29 23:19:44.566011 [recoverd: 8044]: Resetting ban count to 0 for all nodes 2014/10/29 23:26:22.113291 [recoverd: 8044]: Taking out recovery lock from recovery daemon 2014/10/29 23:26:22.130858 [recoverd: 8044]: Take the recovery lock 2014/10/29 23:26:22.243073 [ 7729]: Freeze priority 1 2014/10/29 23:26:22.261883 [ 7729]: Freeze priority 2 2014/10/29 23:26:22.266598 [ 7729]: Freeze priority 3 2014/10/29 23:26:25.799245 [ 7729]: Thawing priority 1 2014/10/29 23:26:25.799287 [ 7729]: Release freeze handler for prio 1 2014/10/29 23:26:25.799322 [ 7729]: Thawing priority 2 2014/10/29 23:26:25.799343 [ 7729]: Release freeze handler for prio 2 2014/10/29 23:26:25.799372 [ 7729]: Thawing priority 3 2014/10/29 23:26:25.799390 [ 7729]: Release freeze handler for prio 3 2014/10/29 23:26:26.153355 [recoverd: 8044]: Resetting ban count to 0 for all nodes 2014/10/29 23:29:48.437778 [recovery-lock: 2433]: failed read from recovery_lock_fd - Transport endpoint is not connected 2014/10/29 23:29:48.801422 [recoverd: 8044]: server/ctdb_recoverd.c:3349 reclock child process returned error 2 2014/10/29 23:29:48.801458 [recoverd: 8044]: server/ctdb_recoverd.c:3449 reclock child failed when checking file 2014/10/29 23:29:48.801548 [recoverd: 8044]: Failed check_recovery_lock. Force a recovery 2014/10/29 23:29:48.801565 [recoverd: 8044]: Taking out recovery lock from recovery daemon 2014/10/29 23:29:48.801575 [recoverd: 8044]: Take the recovery lock 2014/10/29 23:29:48.903663 [ 7729]: Freeze priority 1 2014/10/29 23:29:48.958435 [ 7729]: Freeze priority 2 2014/10/29 23:29:48.963127 [ 7729]: Freeze priority 3 2014/10/29 23:29:49.379307 [ 7729]: pnn 2 Invalid reqid 86537 in ctdb_reply_control 2014/10/29 23:29:49.393106 [ 7729]: pnn 2 Invalid reqid 86539 in ctdb_reply_control 2014/10/29 23:29:49.423906 [ 7729]: pnn 2 Invalid reqid 86536 in ctdb_reply_control 2014/10/29 23:29:49.550269 [ 7729]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 23:29:49.560911 [ 7729]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 23:31:20.462344 [ 1523]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/29 23:31:20.462448 [ 1523]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/29 23:31:20.462466 [ 1523]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/29 23:35:20.437306 [ 7517]: Starting CTDBD (Version 2.5.3) as PID: 7517 2014/10/29 23:35:21.852604 [ 7517]: Vacuuming is disabled for persistent database registry.tdb 2014/10/29 23:35:21.876129 [ 7517]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/29 23:35:21.890263 [ 7517]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/29 23:35:21.904314 [ 7517]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/29 23:35:21.904332 [ 7517]: Ignoring persistent database 'account_policy.tdb.1' 2014/10/29 23:35:21.904341 [ 7517]: Ignoring persistent database 'ctdb.tdb.1' 2014/10/29 23:35:21.904350 [ 7517]: Ignoring persistent database 'group_mapping.tdb.1' 2014/10/29 23:35:21.904359 [ 7517]: Ignoring persistent database 'secrets.tdb.1' 2014/10/29 23:35:21.904367 [ 7517]: Ignoring persistent database 'share_info.tdb.1' 2014/10/29 23:35:21.918360 [ 7517]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/29 23:35:21.932297 [ 7517]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/29 23:35:21.932316 [ 7517]: Ignoring persistent database 'passdb.tdb.1' 2014/10/29 23:35:21.932325 [ 7517]: Ignoring persistent database 'registry.tdb.1' 2014/10/29 23:35:21.946216 [ 7517]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/29 23:35:21.946246 [ 7517]: Freeze priority 1 2014/10/29 23:35:21.964554 [ 7517]: Freeze priority 2 2014/10/29 23:35:21.964913 [ 7517]: Freeze priority 3 2014/10/29 23:35:22.129126 [ 7517]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/29 23:35:22.133292 [ 7517]: 00.ctdb: Set RecoverTimeout to 60 2014/10/29 23:35:22.136881 [ 7517]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/29 23:35:22.259327 [ 7517]: Freeze priority 1 2014/10/29 23:35:22.259402 [ 7517]: Freeze priority 2 2014/10/29 23:35:22.259456 [ 7517]: Freeze priority 3 2014/10/29 23:35:22.599896 [ 7517]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/29 23:35:26.267285 [recoverd: 7799]: server/ctdb_recoverd.c:3692 Current recmaster node 3 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/29 23:35:26.267390 [ 7517]: Freeze priority 1 2014/10/29 23:35:26.267468 [ 7517]: Freeze priority 2 2014/10/29 23:35:26.267542 [ 7517]: Freeze priority 3 2014/10/29 23:35:30.631487 [ 7517]: Freeze priority 1 2014/10/29 23:35:30.645045 [ 7517]: Freeze priority 2 2014/10/29 23:35:30.651808 [ 7517]: Freeze priority 3 2014/10/29 23:35:30.837055 [ 7517]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/29 23:35:30.838974 [ 7517]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/29 23:35:30.845611 [ 7517]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup 2014/10/29 23:35:34.653197 [ 7517]: Thawing priority 1 2014/10/29 23:35:34.653241 [ 7517]: Release freeze handler for prio 1 2014/10/29 23:35:34.653275 [ 7517]: Thawing priority 2 2014/10/29 23:35:34.653296 [ 7517]: Release freeze handler for prio 2 2014/10/29 23:35:34.653323 [ 7517]: Thawing priority 3 2014/10/29 23:35:34.653343 [ 7517]: Release freeze handler for prio 3 2014/10/29 23:35:49.665525 [ 7517]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/29 23:35:49.672285 [recoverd: 7799]: Trigger takeoverrun 2014/10/29 23:35:50.030247 [ 7517]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/29 23:35:50.043913 [ 7517]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/29 23:35:50.081005 [ 7517]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/29 23:35:50.393738 [ 7517]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/29 23:35:52.857040 [ 7517]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation 2014/10/29 23:35:53.689664 [ 7517]: 60.nfs: Reconfiguring service "nfs"... 2014/10/29 23:39:52.002593 [ 7517]: Freeze priority 1 2014/10/29 23:39:52.002843 [ 7517]: Freeze priority 1 2014/10/29 23:39:52.016368 [ 7517]: Freeze priority 2 2014/10/29 23:39:52.016508 [ 7517]: Freeze priority 2 2014/10/29 23:39:52.017521 [ 7517]: Freeze priority 3 2014/10/29 23:39:52.017659 [ 7517]: Freeze priority 3 2014/10/29 23:39:55.036875 [ 7517]: Freeze priority 1 2014/10/29 23:39:55.037190 [ 7517]: Freeze priority 2 2014/10/29 23:39:55.037453 [ 7517]: Freeze priority 3 2014/10/29 23:39:57.560875 [ 7517]: Thawing priority 1 2014/10/29 23:39:57.560925 [ 7517]: Release freeze handler for prio 1 2014/10/29 23:39:57.560960 [ 7517]: Thawing priority 2 2014/10/29 23:39:57.560982 [ 7517]: Release freeze handler for prio 2 2014/10/29 23:39:57.561012 [ 7517]: Thawing priority 3 2014/10/29 23:39:57.561031 [ 7517]: Release freeze handler for prio 3 2014/10/29 23:44:55.998836 [recoverd: 7799]: Taking out recovery lock from recovery daemon 2014/10/29 23:44:55.998906 [recoverd: 7799]: Take the recovery lock 2014/10/29 23:44:56.081334 [ 7517]: Freeze priority 1 2014/10/29 23:44:56.084543 [ 7517]: Freeze priority 2 2014/10/29 23:44:56.089685 [ 7517]: Freeze priority 3 2014/10/29 23:44:58.921741 [ 7517]: Thawing priority 1 2014/10/29 23:44:58.921805 [ 7517]: Release freeze handler for prio 1 2014/10/29 23:44:58.921857 [ 7517]: Thawing priority 2 2014/10/29 23:44:58.921881 [ 7517]: Release freeze handler for prio 2 2014/10/29 23:44:58.921913 [ 7517]: Thawing priority 3 2014/10/29 23:44:58.921932 [ 7517]: Release freeze handler for prio 3 2014/10/29 23:44:58.928547 [recoverd: 7799]: ctdb_control error: 'managed to lock reclock file from inside daemon' 2014/10/29 23:44:58.928628 [recoverd: 7799]: ctdb_control error: 'managed to lock reclock file from inside daemon' 2014/10/29 23:44:58.928647 [recoverd: 7799]: Async operation failed with ret=-1 res=-1 opcode=16 2014/10/29 23:44:58.929613 [set_recmode: 4994]: ERROR: recovery lock file /mnt/lock/lockfile not locked when recovering! 2014/10/29 23:44:58.929825 [recoverd: 7799]: ctdb_control error: 'managed to lock reclock file from inside daemon' 2014/10/29 23:44:58.929859 [recoverd: 7799]: ctdb_control error: 'managed to lock reclock file from inside daemon' 2014/10/29 23:44:58.929875 [recoverd: 7799]: Async operation failed with ret=-1 res=-1 opcode=16 2014/10/29 23:44:58.944487 [recoverd: 7799]: ctdb_control error: 'managed to lock reclock file from inside daemon' 2014/10/29 23:44:58.944522 [recoverd: 7799]: ctdb_control error: 'managed to lock reclock file from inside daemon' 2014/10/29 23:44:58.944540 [recoverd: 7799]: Async operation failed with ret=-1 res=-1 opcode=16 2014/10/29 23:44:58.944552 [recoverd: 7799]: Async wait failed - fail_count=3 2014/10/29 23:44:58.944565 [recoverd: 7799]: server/ctdb_recoverd.c:412 Unable to set recovery mode. Recovery failed. 2014/10/29 23:44:58.944577 [recoverd: 7799]: server/ctdb_recoverd.c:1996 Unable to set recovery mode to normal on cluster 2014/10/29 23:45:02.023026 [ 7517]: Freeze priority 1 2014/10/29 23:45:02.024833 [ 7517]: Freeze priority 2 2014/10/29 23:45:02.027860 [ 7517]: Freeze priority 3 2014/10/29 23:45:06.757661 [ 7517]: Thawing priority 1 2014/10/29 23:45:06.757709 [ 7517]: Release freeze handler for prio 1 2014/10/29 23:45:06.757743 [ 7517]: Thawing priority 2 2014/10/29 23:45:06.757764 [ 7517]: Release freeze handler for prio 2 2014/10/29 23:45:06.757793 [ 7517]: Thawing priority 3 2014/10/29 23:45:06.757812 [ 7517]: Release freeze handler for prio 3 2014/10/29 23:54:58.562006 [ 7517]: Freeze priority 1 2014/10/29 23:54:58.566363 [ 7517]: Freeze priority 2 2014/10/29 23:54:58.570732 [ 7517]: Freeze priority 3 2014/10/29 23:55:01.591803 [ 7517]: Freeze priority 1 2014/10/29 23:55:01.592138 [ 7517]: Freeze priority 2 2014/10/29 23:55:01.592520 [ 7517]: Freeze priority 3 2014/10/29 23:55:05.236799 [ 7517]: Thawing priority 1 2014/10/29 23:55:05.236891 [ 7517]: Release freeze handler for prio 1 2014/10/29 23:55:05.236939 [ 7517]: Thawing priority 2 2014/10/29 23:55:05.236959 [ 7517]: Release freeze handler for prio 2 2014/10/29 23:55:05.236990 [ 7517]: Thawing priority 3 2014/10/29 23:55:05.237008 [ 7517]: Release freeze handler for prio 3 2014/10/30 00:00:19.275139 [ 7517]: Freeze priority 1 2014/10/30 00:00:19.381570 [ 7517]: Freeze priority 2 2014/10/30 00:00:19.382780 [ 7517]: Freeze priority 3 2014/10/30 00:00:24.319174 [ 7517]: Thawing priority 1 2014/10/30 00:00:24.319222 [ 7517]: Release freeze handler for prio 1 2014/10/30 00:00:24.319256 [ 7517]: Thawing priority 2 2014/10/30 00:00:24.319277 [ 7517]: Release freeze handler for prio 2 2014/10/30 00:00:24.319309 [ 7517]: Thawing priority 3 2014/10/30 00:00:24.319328 [ 7517]: Release freeze handler for prio 3 2014/10/30 00:05:05.687461 [ 7517]: Freeze priority 1 2014/10/30 00:05:05.693033 [ 7517]: Freeze priority 1 2014/10/30 00:05:05.695439 [ 7517]: Freeze priority 1 2014/10/30 00:05:05.702088 [ 7517]: Freeze priority 2 2014/10/30 00:05:05.704678 [ 7517]: Freeze priority 2 2014/10/30 00:05:05.706283 [ 7517]: Freeze priority 2 2014/10/30 00:05:05.706377 [ 7517]: Freeze priority 3 2014/10/30 00:05:05.706836 [ 7517]: Freeze priority 3 2014/10/30 00:05:05.710962 [ 7517]: Freeze priority 3 2014/10/30 00:05:09.222426 [recoverd: 7799]: Taking out recovery lock from recovery daemon 2014/10/30 00:05:09.222482 [recoverd: 7799]: Take the recovery lock 2014/10/30 00:05:09.236333 [ 7517]: Freeze priority 1 2014/10/30 00:05:09.236689 [ 7517]: Freeze priority 2 2014/10/30 00:05:09.237025 [ 7517]: Freeze priority 3 2014/10/30 00:05:11.688669 [ 7517]: Thawing priority 1 2014/10/30 00:05:11.688709 [ 7517]: Release freeze handler for prio 1 2014/10/30 00:05:11.688739 [ 7517]: Thawing priority 2 2014/10/30 00:05:11.688766 [ 7517]: Release freeze handler for prio 2 2014/10/30 00:05:11.688791 [ 7517]: Thawing priority 3 2014/10/30 00:05:11.688806 [ 7517]: Release freeze handler for prio 3 2014/10/30 00:05:12.278504 [recoverd: 7799]: Resetting ban count to 0 for all nodes 2014/10/30 00:11:46.429803 [ 7517]: Freeze priority 1 2014/10/30 00:11:46.493960 [ 7517]: Freeze priority 2 2014/10/30 00:11:46.499003 [ 7517]: Freeze priority 3 2014/10/30 00:11:51.797756 [ 7517]: Thawing priority 1 2014/10/30 00:11:51.797867 [ 7517]: Release freeze handler for prio 1 2014/10/30 00:11:51.797926 [ 7517]: Thawing priority 2 2014/10/30 00:11:51.797947 [ 7517]: Release freeze handler for prio 2 2014/10/30 00:11:51.797980 [ 7517]: Thawing priority 3 2014/10/30 00:11:51.797995 [ 7517]: Release freeze handler for prio 3 2014/10/30 00:15:14.434917 [ 7517]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 00:15:14.555376 [ 7517]: common/ctdb_fork.c:131 waitpid() returned error. errno:10 2014/10/30 00:15:14.555410 [ 7517]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 00:16:40.983759 [ 1538]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/30 00:16:40.983858 [ 1538]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 00:16:40.983872 [ 1538]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 00:20:40.944984 [ 7818]: Starting CTDBD (Version 2.5.3) as PID: 7818 2014/10/30 00:20:42.884811 [ 7818]: Vacuuming is disabled for persistent database registry.tdb 2014/10/30 00:20:42.908316 [ 7818]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/30 00:20:42.922385 [ 7818]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/30 00:20:42.936300 [ 7818]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/30 00:20:42.936318 [ 7818]: Ignoring persistent database 'account_policy.tdb.1' 2014/10/30 00:20:42.936327 [ 7818]: Ignoring persistent database 'ctdb.tdb.1' 2014/10/30 00:20:42.936336 [ 7818]: Ignoring persistent database 'group_mapping.tdb.1' 2014/10/30 00:20:42.936345 [ 7818]: Ignoring persistent database 'secrets.tdb.1' 2014/10/30 00:20:42.936353 [ 7818]: Ignoring persistent database 'share_info.tdb.1' 2014/10/30 00:20:42.950277 [ 7818]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/30 00:20:42.964190 [ 7818]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/30 00:20:42.964209 [ 7818]: Ignoring persistent database 'passdb.tdb.1' 2014/10/30 00:20:42.964218 [ 7818]: Ignoring persistent database 'registry.tdb.1' 2014/10/30 00:20:42.978215 [ 7818]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/30 00:20:42.978247 [ 7818]: Freeze priority 1 2014/10/30 00:20:42.985056 [ 7818]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 00:20:42.988498 [ 7818]: Freeze priority 2 2014/10/30 00:20:42.988845 [ 7818]: Freeze priority 3 2014/10/30 00:20:43.152836 [ 7818]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/30 00:20:43.156710 [ 7818]: 00.ctdb: Set RecoverTimeout to 60 2014/10/30 00:20:43.160453 [ 7818]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/30 00:20:43.282466 [ 7818]: Freeze priority 1 2014/10/30 00:20:43.282540 [ 7818]: Freeze priority 2 2014/10/30 00:20:43.282593 [ 7818]: Freeze priority 3 2014/10/30 00:20:43.302134 [ 7818]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 00:20:43.484755 [ 7818]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 00:20:47.290378 [recoverd: 8097]: server/ctdb_recoverd.c:3692 Current recmaster node 3 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/30 00:20:47.290494 [ 7818]: Freeze priority 1 2014/10/30 00:20:47.290571 [ 7818]: Freeze priority 2 2014/10/30 00:20:47.290624 [ 7818]: Freeze priority 3 2014/10/30 00:20:51.691775 [ 7818]: Freeze priority 1 2014/10/30 00:20:51.693474 [ 7818]: Freeze priority 2 2014/10/30 00:20:51.695242 [ 7818]: Freeze priority 3 2014/10/30 00:20:51.867529 [ 7818]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/30 00:20:51.869836 [ 7818]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/30 00:20:51.872102 [ 7818]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup 2014/10/30 00:20:54.036238 [ 7818]: Thawing priority 1 2014/10/30 00:20:54.036287 [ 7818]: Release freeze handler for prio 1 2014/10/30 00:20:54.036321 [ 7818]: Thawing priority 2 2014/10/30 00:20:54.036341 [ 7818]: Release freeze handler for prio 2 2014/10/30 00:20:54.036367 [ 7818]: Thawing priority 3 2014/10/30 00:20:54.036384 [ 7818]: Release freeze handler for prio 3 2014/10/30 00:21:08.329645 [recoverd: 8097]: Trigger takeoverrun 2014/10/30 00:21:08.406254 [ 7818]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/30 00:21:08.720224 [ 7818]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 00:21:08.731642 [ 7818]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 00:21:08.756017 [ 7818]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/30 00:21:08.934080 [ 7818]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/30 00:21:11.408393 [ 7818]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation 2014/10/30 00:21:12.003982 [ 7818]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 00:25:17.387048 [ 7818]: Freeze priority 1 2014/10/30 00:25:17.398957 [ 7818]: Freeze priority 2 2014/10/30 00:25:17.400094 [ 7818]: Freeze priority 3 2014/10/30 00:25:20.486223 [ 7818]: Freeze priority 1 2014/10/30 00:25:20.486637 [ 7818]: Freeze priority 2 2014/10/30 00:25:20.486984 [ 7818]: Freeze priority 3 2014/10/30 00:25:23.306957 [ 7818]: Thawing priority 1 2014/10/30 00:25:23.307015 [ 7818]: Release freeze handler for prio 1 2014/10/30 00:25:23.307053 [ 7818]: Thawing priority 2 2014/10/30 00:25:23.307074 [ 7818]: Release freeze handler for prio 2 2014/10/30 00:25:23.307098 [ 7818]: Thawing priority 3 2014/10/30 00:25:23.307109 [ 7818]: Release freeze handler for prio 3 2014/10/30 00:25:23.309681 [ 7818]: pnn 2 Invalid reqid 8045 in ctdb_become_dmaster from node 0 2014/10/30 00:25:23.309742 [ 7818]: server/ctdb_call.c:1005 reqid 8046 not found 2014/10/30 00:30:23.316826 [ 7818]: Freeze priority 1 2014/10/30 00:30:23.410573 [ 7818]: Freeze priority 2 2014/10/30 00:30:23.411530 [ 7818]: Freeze priority 3 2014/10/30 00:30:26.712131 [ 7818]: Thawing priority 1 2014/10/30 00:30:26.712181 [ 7818]: Release freeze handler for prio 1 2014/10/30 00:30:26.712213 [ 7818]: Thawing priority 2 2014/10/30 00:30:26.712229 [ 7818]: Release freeze handler for prio 2 2014/10/30 00:30:26.712255 [ 7818]: Thawing priority 3 2014/10/30 00:30:26.712270 [ 7818]: Release freeze handler for prio 3 2014/10/30 00:40:26.491813 [ 7818]: Freeze priority 1 2014/10/30 00:40:26.530234 [ 7818]: Freeze priority 1 2014/10/30 00:40:26.530297 [ 7818]: Freeze priority 1 2014/10/30 00:40:33.912559 [ 7818]: Skip monitoring since databases are frozen ===== Start of debug locks PID=5236 ===== 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454 4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W 4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23687 /usr/sbin/smbd locking.tdb.2 27456 27456 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W ----- Stack trace for PID=23687 ----- #0 0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0 #28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #31 0x00007f95c57ad1b4 in smbd_accept_connection () #32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0 #33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #35 0x00007f95c57a9d01 in main () ----- Stack trace for PID=4899 ----- #0 0x00007f319a430094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541 #7 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541 #10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537 #13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541 #14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541 #18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=5236 ===== ===== Start of debug locks PID=5748 ===== 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454 4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W 4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23687 /usr/sbin/smbd locking.tdb.2 27456 27456 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W ----- Stack trace for PID=23687 ----- #0 0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0 #28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #31 0x00007f95c57ad1b4 in smbd_accept_connection () #32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0 #33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #35 0x00007f95c57a9d01 in main () ----- Stack trace for PID=4899 ----- #0 0x00007f319a430094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541 #7 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541 #10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537 #13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541 #14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541 #18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=5748 ===== 2014/10/30 00:40:48.912906 [ 7818]: Skip monitoring since databases are frozen ===== Start of debug locks PID=6443 ===== 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454 4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W 4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23687 /usr/sbin/smbd locking.tdb.2 27456 27456 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W ----- Stack trace for PID=23687 ----- #0 0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0 #28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #31 0x00007f95c57ad1b4 in smbd_accept_connection () #32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0 #33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #35 0x00007f95c57a9d01 in main () ----- Stack trace for PID=4899 ----- #0 0x00007f319a430094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541 #7 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541 #10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537 #13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541 #14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541 #18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=6443 ===== 2014/10/30 00:41:03.913859 [ 7818]: Skip monitoring since databases are frozen ===== Start of debug locks PID=6839 ===== 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454 4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W 4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23687 /usr/sbin/smbd locking.tdb.2 27456 27456 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W ----- Stack trace for PID=23687 ----- #0 0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0 #28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #31 0x00007f95c57ad1b4 in smbd_accept_connection () #32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0 #33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #35 0x00007f95c57a9d01 in main () ----- Stack trace for PID=4899 ----- #0 0x00007f319a430094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541 #7 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541 #10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537 #13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541 #14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541 #18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=6839 ===== ===== Start of debug locks PID=7275 ===== 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454 4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W 4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23687 /usr/sbin/smbd locking.tdb.2 27456 27456 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W ----- Stack trace for PID=23687 ----- #0 0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0 #28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #31 0x00007f95c57ad1b4 in smbd_accept_connection () #32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0 #33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #35 0x00007f95c57a9d01 in main () ----- Stack trace for PID=4899 ----- #0 0x00007f319a430094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541 #7 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541 #10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537 #13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541 #14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541 #18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=7275 ===== 2014/10/30 00:41:18.914467 [ 7818]: Skip monitoring since databases are frozen 2014/10/30 00:41:26.495095 [ 7818]: Freeze priority 1 2014/10/30 00:41:26.530205 [ 7818]: Recovery daemon ping timeout. Count : 0 2014/10/30 00:41:26.530332 [recoverd: 8097]: ctdb_control error: 'ctdb_control timed out' 2014/10/30 00:41:26.530379 [recoverd: 8097]: ctdb_control error: 'ctdb_control timed out' 2014/10/30 00:41:26.530403 [recoverd: 8097]: Async operation failed with ret=-1 res=-1 opcode=33 2014/10/30 00:41:26.530421 [recoverd: 8097]: Failed to freeze node 2 during recovery. Set it as ban culprit for 4 credits 2014/10/30 00:41:26.530440 [recoverd: 8097]: Async wait failed - fail_count=1 2014/10/30 00:41:26.530455 [recoverd: 8097]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed. 2014/10/30 00:41:26.530472 [recoverd: 8097]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster 2014/10/30 00:41:26.532117 [ 7818]: Freeze priority 1 2014/10/30 00:41:26.532157 [ 7818]: Freeze priority 1 ===== Start of debug locks PID=7670 ===== 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454 4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W 4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23687 /usr/sbin/smbd locking.tdb.2 27456 27456 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W ----- Stack trace for PID=23687 ----- #0 0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0 #28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #31 0x00007f95c57ad1b4 in smbd_accept_connection () #32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0 #33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #35 0x00007f95c57a9d01 in main () ----- Stack trace for PID=4899 ----- #0 0x00007f319a430094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541 #7 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541 #10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537 #13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541 #14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541 #18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=7670 ===== 2014/10/30 00:41:33.914908 [ 7818]: Skip monitoring since databases are frozen ===== Start of debug locks PID=8027 ===== 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454 4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W 4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23687 /usr/sbin/smbd locking.tdb.2 27456 27456 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W ----- Stack trace for PID=23687 ----- #0 0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0 #28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #31 0x00007f95c57ad1b4 in smbd_accept_connection () #32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0 #33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #35 0x00007f95c57a9d01 in main () ----- Stack trace for PID=4899 ----- #0 0x00007f319a430094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541 #7 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541 #10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537 #13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541 #14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541 #18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=8027 ===== ===== Start of debug locks PID=8466 ===== 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454 4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W 4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23687 /usr/sbin/smbd locking.tdb.2 27456 27456 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W ----- Stack trace for PID=23687 ----- #0 0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0 #28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #31 0x00007f95c57ad1b4 in smbd_accept_connection () #32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0 #33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #35 0x00007f95c57a9d01 in main () ----- Stack trace for PID=4899 ----- #0 0x00007f319a430094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541 #7 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541 #10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537 #13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541 #14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541 #18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=8466 ===== 2014/10/30 00:41:48.915071 [ 7818]: Skip monitoring since databases are frozen ===== Start of debug locks PID=8990 ===== 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454 4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W 4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23687 /usr/sbin/smbd locking.tdb.2 27456 27456 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W ----- Stack trace for PID=23687 ----- #0 0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0 #28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #31 0x00007f95c57ad1b4 in smbd_accept_connection () #32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0 #33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #35 0x00007f95c57a9d01 in main () ----- Stack trace for PID=4899 ----- #0 0x00007f319a430094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541 #7 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541 #10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537 #13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541 #14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541 #18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=8990 ===== 2014/10/30 00:42:03.915852 [ 7818]: Skip monitoring since databases are frozen ===== Start of debug locks PID=9397 ===== 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454 4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W 4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23687 /usr/sbin/smbd locking.tdb.2 27456 27456 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W ----- Stack trace for PID=23687 ----- #0 0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0 #28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #31 0x00007f95c57ad1b4 in smbd_accept_connection () #32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0 #33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #35 0x00007f95c57a9d01 in main () ----- Stack trace for PID=4899 ----- #0 0x00007f319a430094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541 #7 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541 #10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537 #13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541 #14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541 #18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=9397 ===== ===== Start of debug locks PID=9836 ===== 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454 4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W 4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 10172 /usr/sbin/smbd printer_list.tdb.2 2984 2984 W 23687 /usr/sbin/smbd locking.tdb.2 27456 27456 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W ----- Stack trace for PID=23687 ----- #0 0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f95c39dcdb9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f95c39e03bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f95c39e15ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f95c39e410f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f95c39e99ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f95c02e5afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f95c02e5b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f95c39eb5e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f95c02e58a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f95c4d9c077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f95c25f9e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f95c25f9c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f95c25f67db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f95c4d192f9 in open_file_ntcreate () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f95c4d1be02 in create_file_unixpath () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f95c4d1ce5a in create_file_default () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f95c4de641b in vfswrap_create_file () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f95c4d23415 in smb_vfs_call_create_file () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f95c4d51ca6 in smbd_smb2_request_process_create () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f95c4d4bc5d in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f95c4d4c19f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #22 0x00007f95c4d4909c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #23 0x00007f95c37a5534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f95c37a5069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #25 0x00007f95c37a3f46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #26 0x00007f95c23eb3f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #27 0x00007f95c39f571c in run_events_poll () from /lib64/libsmbconf.so.0 #28 0x00007f95c39f5a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #29 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #30 0x00007f95c4d37bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #31 0x00007f95c57ad1b4 in smbd_accept_connection () #32 0x00007f95c39f584c in run_events_poll () from /lib64/libsmbconf.so.0 #33 0x00007f95c39f5aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #34 0x00007f95c23eabcd in _tevent_loop_once () from /lib64/libtevent.so.0 #35 0x00007f95c57a9d01 in main () ----- Stack trace for PID=4899 ----- #0 0x00007f319a430094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0xb85370, rw=1, off=27455, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0xb85370, rw_type=1, offset=27455, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27455, len=7) at lib/tdb/common/lock.c:537 #5 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27449, len=13) at lib/tdb/common/lock.c:541 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27437, len=25) at lib/tdb/common/lock.c:541 #7 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=49) at lib/tdb/common/lock.c:541 #8 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27413, len=98) at lib/tdb/common/lock.c:537 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27315, len=196) at lib/tdb/common/lock.c:541 #10 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=27120, len=391) at lib/tdb/common/lock.c:541 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=26730, len=1563) at lib/tdb/common/lock.c:537 #13 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=3125) at lib/tdb/common/lock.c:541 #14 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=6250) at lib/tdb/common/lock.c:537 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=25168, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=50000) at lib/tdb/common/lock.c:541 #18 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=100001) at lib/tdb/common/lock.c:537 #19 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=200002) at lib/tdb/common/lock.c:537 #20 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:537 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0xb85370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0xb85370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fffaa232d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:129 ===== End of debug locks PID=9836 ===== 2014/10/30 00:42:18.916751 [ 7818]: Skip monitoring since databases are frozen 2014/10/30 00:42:26.497770 [ 7818]: Banning this node for 30 seconds 2014/10/30 00:42:26.497824 [ 7818]: Freeze priority 1 2014/10/30 00:42:26.497838 [ 7818]: Freeze priority 2 2014/10/30 00:42:26.498006 [ 7818]: Freeze priority 3 2014/10/30 00:42:26.531568 [ 7818]: Recovery daemon ping timeout. Count : 0 2014/10/30 00:42:26.532695 [recoverd: 8097]: ctdb_control error: 'ctdb_control timed out' 2014/10/30 00:42:26.532731 [recoverd: 8097]: ctdb_control error: 'ctdb_control timed out' 2014/10/30 00:42:26.532746 [recoverd: 8097]: Async operation failed with ret=-1 res=-1 opcode=33 2014/10/30 00:42:26.532757 [recoverd: 8097]: Failed to freeze node 2 during recovery. Set it as ban culprit for 4 credits 2014/10/30 00:42:26.532769 [recoverd: 8097]: Async wait failed - fail_count=1 2014/10/30 00:42:26.532779 [recoverd: 8097]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed. 2014/10/30 00:42:26.532789 [recoverd: 8097]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster 2014/10/30 00:42:26.533422 [ 7818]: Banning this node for 30 seconds 2014/10/30 00:42:26.533457 [ 7818]: Freeze priority 1 2014/10/30 00:42:26.533760 [ 7818]: Banning this node for 30 seconds 2014/10/30 00:42:26.533791 [ 7818]: Freeze priority 1 ===== Start of debug locks PID=10316 ===== 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 168 27454 4899 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4679 /usr/sbin/smbd smbXsrv_open_global.tdb.2 200940 200940 W 4899 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 10172 /usr/sbin/smbd printer_list.tdb.2 2984 2984 W 10277 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF 10277 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF 23687 /usr/sbin/smbd locking.tdb.2 27456 27456 4899 /usr/bin/ctdb_lock_helper locking.tdb.2 27455 27457 W 10276 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF 10276 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF 10276 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF ----- Stack trace for PID=23687 ----- 2014/10/30 00:42:26.625068 [ 7818]: 10.interface: Killing TCP connection 10.10.10.210:53439 10.10.10.183:445 2014/10/30 00:42:26.625244 [ 7818]: 10.interface: Killing TCP connection 10.10.10.205:54702 10.10.10.183:445 2014/10/30 00:42:26.625343 [ 7818]: 10.interface: Killing TCP connection 10.10.10.208:54232 10.10.10.183:445 2014/10/30 00:42:26.625439 [ 7818]: 10.interface: Killing TCP connection 10.10.10.206:49680 10.10.10.183:445 2014/10/30 00:42:26.626103 [ 7818]: 10.interface: Killing TCP connection 10.10.10.210:53439 10.10.10.183:445 2014/10/30 00:42:26.626217 [ 7818]: 10.interface: Killing TCP connection 10.10.10.205:54702 10.10.10.183:445 2014/10/30 00:42:26.626325 [ 7818]: 10.interface: Killing TCP connection 10.10.10.208:54232 10.10.10.183:445 2014/10/30 00:42:26.626409 [ 7818]: 10.interface: Killing TCP connection 10.10.10.206:49680 10.10.10.183:445 2014/10/30 00:42:26.626836 [ 7818]: 10.interface: Killing TCP connection 10.10.10.210:53439 10.10.10.183:445 2014/10/30 00:42:26.626966 [ 7818]: 10.interface: Killing TCP connection 10.10.10.205:54702 10.10.10.183:445 2014/10/30 00:42:26.627070 [ 7818]: 10.interface: Killing TCP connection 10.10.10.208:54232 10.10.10.183:445 2014/10/30 00:42:26.627158 [ 7818]: 10.interface: Killing TCP connection 10.10.10.206:49680 10.10.10.183:445 2014/10/30 00:42:26.655282 [ 7818]: 10.interface: Killed 4 TCP connections to released IP 10.10.10.183 2014/10/30 00:42:26.655536 [ 7818]: 10.interface: Killed 4 TCP connections to released IP 10.10.10.183 2014/10/30 00:42:26.655699 [ 7818]: 10.interface: Killed 4 TCP connections to released IP 10.10.10.183 2014/10/30 00:42:26.669693 [ 7818]: 10.interface: RTNETLINK answers: Cannot assign requested address 2014/10/30 00:42:26.669840 [ 7818]: 10.interface: Failed to del 10.10.10.183 on dev bond1 2014/10/30 00:42:26.672502 [ 7818]: 10.interface: RTNETLINK answers: Cannot assign requested address 2014/10/30 00:42:26.672650 [ 7818]: 10.interface: Failed to del 10.10.10.183 on dev bond1 2014/10/30 00:42:26.789022 [ 7818]: Freeze priority 1 2014/10/30 00:42:26.789126 [ 7818]: Freeze priority 1 2014/10/30 00:42:26.837155 [ 7818]: pnn 2 Invalid reqid 40862 in ctdb_reply_control 2014/10/30 00:42:26.837188 [ 7818]: pnn 2 Invalid reqid 40656 in ctdb_reply_control 2014/10/30 00:42:26.837545 [ 7818]: Freeze priority 2 2014/10/30 00:42:26.837596 [ 7818]: Freeze priority 2 2014/10/30 00:42:26.838430 [ 7818]: Freeze priority 3 2014/10/30 00:42:26.838522 [ 7818]: Freeze priority 3 #0 0x00007f95c2111df0 in __poll_nocancel () from /lib64/libc.so.6 ----- Stack trace for PID=4899 ----- #0 0x00007f319a405890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f319a405744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fffaa230b18) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=10316 ===== 2014/10/30 00:42:33.547346 [ 7818]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 00:42:36.389861 [ 7818]: DB Attach to database ctdb.tdb refused since node is inactive (flags=0x8) 2014/10/30 00:42:56.534749 [ 7818]: Banning timedout 2014/10/30 00:42:56.886614 [ 7818]: Freeze priority 1 2014/10/30 00:42:56.891571 [ 7818]: Freeze priority 2 2014/10/30 00:42:56.896050 [ 7818]: Freeze priority 3 2014/10/30 00:42:59.646260 [ 7818]: Thawing priority 1 2014/10/30 00:42:59.646336 [ 7818]: Release freeze handler for prio 1 2014/10/30 00:42:59.646377 [ 7818]: Thawing priority 2 2014/10/30 00:42:59.646392 [ 7818]: Release freeze handler for prio 2 2014/10/30 00:42:59.646418 [ 7818]: Thawing priority 3 2014/10/30 00:42:59.646438 [ 7818]: Release freeze handler for prio 3 2014/10/30 00:42:59.647993 [ 7818]: server/ctdb_call.c:1005 reqid 41465 not found 2014/10/30 00:42:59.648069 [ 7818]: server/ctdb_call.c:1005 reqid 41466 not found 2014/10/30 00:43:00.261087 [ 7818]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 00:45:41.909761 [ 7818]: Freeze priority 1 2014/10/30 00:45:42.054977 [ 7818]: Freeze priority 2 2014/10/30 00:45:42.082373 [ 7818]: Freeze priority 3 2014/10/30 00:45:45.405501 [ 7818]: Thawing priority 1 2014/10/30 00:45:45.405545 [ 7818]: Release freeze handler for prio 1 2014/10/30 00:45:45.405577 [ 7818]: Thawing priority 2 2014/10/30 00:45:45.405593 [ 7818]: Release freeze handler for prio 2 2014/10/30 00:45:45.405619 [ 7818]: Thawing priority 3 2014/10/30 00:45:45.405633 [ 7818]: Release freeze handler for prio 3 2014/10/30 00:50:30.261912 [ 7818]: Freeze priority 1 2014/10/30 00:50:30.403324 [ 7818]: Freeze priority 2 2014/10/30 00:50:30.415014 [ 7818]: Freeze priority 3 2014/10/30 00:50:33.225966 [ 7818]: Thawing priority 1 2014/10/30 00:50:33.226003 [ 7818]: Release freeze handler for prio 1 2014/10/30 00:50:33.226031 [ 7818]: Thawing priority 2 2014/10/30 00:50:33.226049 [ 7818]: Release freeze handler for prio 2 2014/10/30 00:50:33.226078 [ 7818]: Thawing priority 3 2014/10/30 00:50:33.226094 [ 7818]: Release freeze handler for prio 3 2014/10/30 00:57:16.326220 [ 7818]: Freeze priority 1 2014/10/30 00:57:16.468323 [ 7818]: Freeze priority 2 2014/10/30 00:57:16.484816 [ 7818]: Freeze priority 3 2014/10/30 00:57:19.140647 [ 7818]: Thawing priority 1 2014/10/30 00:57:19.140700 [ 7818]: Release freeze handler for prio 1 2014/10/30 00:57:19.140732 [ 7818]: Thawing priority 2 2014/10/30 00:57:19.140749 [ 7818]: Release freeze handler for prio 2 2014/10/30 00:57:19.140776 [ 7818]: Thawing priority 3 2014/10/30 00:57:19.140792 [ 7818]: Release freeze handler for prio 3 2014/10/30 01:00:40.696663 [ 7818]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 01:00:40.708314 [ 7818]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 01:02:06.927678 [ 1542]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/30 01:02:06.927777 [ 1542]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 01:02:06.927791 [ 1542]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 01:06:06.551288 [ 7671]: Starting CTDBD (Version 2.5.3) as PID: 7671 2014/10/30 01:06:08.031432 [ 7671]: Vacuuming is disabled for persistent database registry.tdb 2014/10/30 01:06:08.055104 [ 7671]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/30 01:06:08.069248 [ 7671]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/30 01:06:08.083141 [ 7671]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/30 01:06:08.083159 [ 7671]: Ignoring persistent database 'account_policy.tdb.1' 2014/10/30 01:06:08.083168 [ 7671]: Ignoring persistent database 'ctdb.tdb.1' 2014/10/30 01:06:08.083177 [ 7671]: Ignoring persistent database 'group_mapping.tdb.1' 2014/10/30 01:06:08.083186 [ 7671]: Ignoring persistent database 'secrets.tdb.1' 2014/10/30 01:06:08.083194 [ 7671]: Ignoring persistent database 'share_info.tdb.1' 2014/10/30 01:06:08.097072 [ 7671]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/30 01:06:08.110955 [ 7671]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/30 01:06:08.110973 [ 7671]: Ignoring persistent database 'passdb.tdb.1' 2014/10/30 01:06:08.110983 [ 7671]: Ignoring persistent database 'registry.tdb.1' 2014/10/30 01:06:08.124888 [ 7671]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/30 01:06:08.124923 [ 7671]: Freeze priority 1 2014/10/30 01:06:08.143125 [ 7671]: Freeze priority 2 2014/10/30 01:06:08.143540 [ 7671]: Freeze priority 3 2014/10/30 01:06:08.307477 [ 7671]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/30 01:06:08.311274 [ 7671]: 00.ctdb: Set RecoverTimeout to 60 2014/10/30 01:06:08.314694 [ 7671]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/30 01:06:08.433478 [ 7671]: Freeze priority 1 2014/10/30 01:06:08.433551 [ 7671]: Freeze priority 2 2014/10/30 01:06:08.433604 [ 7671]: Freeze priority 3 2014/10/30 01:06:08.878277 [ 7671]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 01:06:13.615452 [ 7671]: Freeze priority 1 2014/10/30 01:06:13.632204 [ 7671]: Freeze priority 2 2014/10/30 01:06:13.636861 [ 7671]: Freeze priority 3 2014/10/30 01:06:13.835966 [ 7671]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/30 01:06:13.836480 [ 7671]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/30 01:06:13.837485 [ 7671]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup 2014/10/30 01:06:16.379645 [ 7671]: Thawing priority 1 2014/10/30 01:06:16.379682 [ 7671]: Release freeze handler for prio 1 2014/10/30 01:06:16.379712 [ 7671]: Thawing priority 2 2014/10/30 01:06:16.379729 [ 7671]: Release freeze handler for prio 2 2014/10/30 01:06:16.379762 [ 7671]: Thawing priority 3 2014/10/30 01:06:16.379777 [ 7671]: Release freeze handler for prio 3 2014/10/30 01:06:30.403218 [recoverd: 7950]: Trigger takeoverrun 2014/10/30 01:06:30.836087 [ 7671]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/30 01:06:31.135355 [ 7671]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 01:06:31.146472 [ 7671]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 01:06:31.168316 [ 7671]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/30 01:06:31.347091 [ 7671]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/30 01:06:33.765203 [ 7671]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation 2014/10/30 01:06:34.265995 [ 7671]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 01:10:42.744205 [ 7671]: Freeze priority 1 2014/10/30 01:10:42.754744 [ 7671]: Freeze priority 1 2014/10/30 01:10:42.765990 [ 7671]: Freeze priority 2 2014/10/30 01:10:42.767464 [ 7671]: Freeze priority 2 2014/10/30 01:10:42.768599 [ 7671]: Freeze priority 3 2014/10/30 01:10:42.768956 [ 7671]: Freeze priority 3 2014/10/30 01:10:45.789669 [ 7671]: Freeze priority 1 2014/10/30 01:10:45.790081 [ 7671]: Freeze priority 2 2014/10/30 01:10:45.790776 [ 7671]: Freeze priority 3 2014/10/30 01:10:48.754812 [ 7671]: Thawing priority 1 2014/10/30 01:10:48.754854 [ 7671]: Release freeze handler for prio 1 2014/10/30 01:10:48.754910 [ 7671]: Thawing priority 2 2014/10/30 01:10:48.754954 [ 7671]: Release freeze handler for prio 2 2014/10/30 01:10:48.754984 [ 7671]: Thawing priority 3 2014/10/30 01:10:48.755016 [ 7671]: Release freeze handler for prio 3 2014/10/30 01:15:47.008563 [ 7671]: Freeze priority 1 2014/10/30 01:15:47.072168 [ 7671]: Freeze priority 2 2014/10/30 01:15:47.075867 [ 7671]: Freeze priority 3 2014/10/30 01:15:50.440067 [ 7671]: Thawing priority 1 2014/10/30 01:15:50.440153 [ 7671]: Release freeze handler for prio 1 2014/10/30 01:15:50.440198 [ 7671]: Thawing priority 2 2014/10/30 01:15:50.440218 [ 7671]: Release freeze handler for prio 2 2014/10/30 01:15:50.440251 [ 7671]: Thawing priority 3 2014/10/30 01:15:50.440269 [ 7671]: Release freeze handler for prio 3 2014/10/30 01:16:00.943534 [ 7671]: Monitoring event was cancelled 2014/10/30 01:16:01.114035 [ 7671]: Freeze priority 1 2014/10/30 01:16:01.116321 [ 7671]: Freeze priority 2 2014/10/30 01:16:01.117798 [ 7671]: Freeze priority 3 2014/10/30 01:16:04.153080 [ 7671]: Thawing priority 1 2014/10/30 01:16:04.153123 [ 7671]: Release freeze handler for prio 1 2014/10/30 01:16:04.153158 [ 7671]: Thawing priority 2 2014/10/30 01:16:04.153178 [ 7671]: Release freeze handler for prio 2 2014/10/30 01:16:04.153210 [ 7671]: Thawing priority 3 2014/10/30 01:16:04.153229 [ 7671]: Release freeze handler for prio 3 2014/10/30 01:25:53.930571 [ 7671]: Freeze priority 1 2014/10/30 01:25:53.946193 [ 7671]: Freeze priority 1 2014/10/30 01:25:53.979062 [ 7671]: Freeze priority 1 2014/10/30 01:25:53.984722 [ 7671]: Freeze priority 2 2014/10/30 01:25:53.985562 [ 7671]: Freeze priority 2 2014/10/30 01:25:53.986101 [ 7671]: Freeze priority 2 2014/10/30 01:25:53.986540 [ 7671]: Freeze priority 3 2014/10/30 01:25:53.987071 [ 7671]: Freeze priority 3 2014/10/30 01:25:53.987316 [ 7671]: Freeze priority 3 2014/10/30 01:25:57.506025 [ 7671]: Freeze priority 1 2014/10/30 01:25:57.506350 [ 7671]: Freeze priority 2 2014/10/30 01:25:57.506666 [ 7671]: Freeze priority 3 2014/10/30 01:26:01.992757 [ 7671]: Thawing priority 1 2014/10/30 01:26:01.992809 [ 7671]: Release freeze handler for prio 1 2014/10/30 01:26:01.992840 [ 7671]: Thawing priority 2 2014/10/30 01:26:01.992859 [ 7671]: Release freeze handler for prio 2 2014/10/30 01:26:01.992894 [ 7671]: Thawing priority 3 2014/10/30 01:26:01.992916 [ 7671]: Release freeze handler for prio 3 2014/10/30 01:31:15.018448 [ 7671]: Freeze priority 1 2014/10/30 01:31:15.243361 [ 7671]: Freeze priority 2 2014/10/30 01:31:15.244585 [ 7671]: Freeze priority 3 2014/10/30 01:31:18.685801 [ 7671]: Thawing priority 1 2014/10/30 01:31:18.685856 [ 7671]: Release freeze handler for prio 1 2014/10/30 01:31:18.685899 [ 7671]: Thawing priority 2 2014/10/30 01:31:18.685920 [ 7671]: Release freeze handler for prio 2 2014/10/30 01:31:18.685953 [ 7671]: Thawing priority 3 2014/10/30 01:31:18.685970 [ 7671]: Release freeze handler for prio 3 2014/10/30 01:31:18.687402 [ 7671]: server/ctdb_call.c:1005 reqid 46805 not found 2014/10/30 01:31:18.687445 [ 7671]: server/ctdb_call.c:1005 reqid 46806 not found 2014/10/30 01:36:01.071753 [ 7671]: Freeze priority 1 2014/10/30 01:36:01.087906 [ 7671]: Freeze priority 1 2014/10/30 01:36:01.090507 [ 7671]: Freeze priority 1 2014/10/30 01:36:01.149678 [ 7671]: Freeze priority 2 2014/10/30 01:36:01.151637 [ 7671]: Freeze priority 2 2014/10/30 01:36:01.152381 [ 7671]: Freeze priority 2 2014/10/30 01:36:01.152909 [ 7671]: Freeze priority 3 2014/10/30 01:36:01.153502 [ 7671]: Freeze priority 3 2014/10/30 01:36:01.155226 [ 7671]: Freeze priority 3 2014/10/30 01:36:04.167545 [recoverd: 7950]: Taking out recovery lock from recovery daemon 2014/10/30 01:36:04.167579 [recoverd: 7950]: Take the recovery lock 2014/10/30 01:36:04.196520 [ 7671]: Freeze priority 1 2014/10/30 01:36:04.196867 [ 7671]: Freeze priority 2 2014/10/30 01:36:04.197152 [ 7671]: Freeze priority 3 2014/10/30 01:36:06.982332 [ 7671]: Thawing priority 1 2014/10/30 01:36:06.982379 [ 7671]: Release freeze handler for prio 1 2014/10/30 01:36:06.982431 [ 7671]: Thawing priority 2 2014/10/30 01:36:06.982465 [ 7671]: Release freeze handler for prio 2 2014/10/30 01:36:06.982513 [ 7671]: Thawing priority 3 2014/10/30 01:36:06.982536 [ 7671]: Release freeze handler for prio 3 2014/10/30 01:36:06.984715 [ 7671]: server/ctdb_call.c:1005 reqid 70174 not found 2014/10/30 01:36:06.984772 [ 7671]: server/ctdb_call.c:1005 reqid 70175 not found 2014/10/30 01:36:06.984801 [ 7671]: server/ctdb_call.c:1005 reqid 70176 not found 2014/10/30 01:36:06.984823 [ 7671]: server/ctdb_call.c:1005 reqid 70177 not found 2014/10/30 01:36:08.030817 [recoverd: 7950]: Resetting ban count to 0 for all nodes 2014/10/30 01:42:43.600216 [recoverd: 7950]: Taking out recovery lock from recovery daemon 2014/10/30 01:42:43.610535 [recoverd: 7950]: Take the recovery lock 2014/10/30 01:42:44.654301 [ 7671]: High RECLOCK latency 1.043671s for operation recd reclock 2014/10/30 01:42:44.697237 [ 7671]: Freeze priority 1 2014/10/30 01:42:44.724353 [ 7671]: Freeze priority 2 2014/10/30 01:42:44.725658 [ 7671]: Freeze priority 3 2014/10/30 01:42:48.589487 [ 7671]: Thawing priority 1 2014/10/30 01:42:48.589527 [ 7671]: Release freeze handler for prio 1 2014/10/30 01:42:48.589564 [ 7671]: Thawing priority 2 2014/10/30 01:42:48.589599 [ 7671]: Release freeze handler for prio 2 2014/10/30 01:42:48.589632 [ 7671]: Thawing priority 3 2014/10/30 01:42:48.589651 [ 7671]: Release freeze handler for prio 3 2014/10/30 01:42:48.919489 [recoverd: 7950]: Resetting ban count to 0 for all nodes 2014/10/30 01:46:10.513121 [ 7671]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 01:46:10.539069 [ 7671]: common/ctdb_fork.c:131 waitpid() returned error. errno:10 2014/10/30 01:46:10.539096 [ 7671]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 01:47:40.448907 [ 1531]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/30 01:47:40.449009 [ 1531]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 01:47:40.449024 [ 1531]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 01:51:40.820220 [ 7638]: Starting CTDBD (Version 2.5.3) as PID: 7638 2014/10/30 01:51:42.285542 [ 7638]: Vacuuming is disabled for persistent database registry.tdb 2014/10/30 01:51:42.309438 [ 7638]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/30 01:51:42.323793 [ 7638]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/30 01:51:42.337895 [ 7638]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/30 01:51:42.337920 [ 7638]: Ignoring persistent database 'account_policy.tdb.1' 2014/10/30 01:51:42.337935 [ 7638]: Ignoring persistent database 'ctdb.tdb.1' 2014/10/30 01:51:42.337956 [ 7638]: Ignoring persistent database 'group_mapping.tdb.1' 2014/10/30 01:51:42.337966 [ 7638]: Ignoring persistent database 'secrets.tdb.1' 2014/10/30 01:51:42.337975 [ 7638]: Ignoring persistent database 'share_info.tdb.1' 2014/10/30 01:51:42.352162 [ 7638]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/30 01:51:42.366408 [ 7638]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/30 01:51:42.366427 [ 7638]: Ignoring persistent database 'passdb.tdb.1' 2014/10/30 01:51:42.366451 [ 7638]: Ignoring persistent database 'registry.tdb.1' 2014/10/30 01:51:42.381460 [ 7638]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/30 01:51:42.381497 [ 7638]: Freeze priority 1 2014/10/30 01:51:42.405497 [ 7638]: Freeze priority 2 2014/10/30 01:51:42.405854 [ 7638]: Freeze priority 3 2014/10/30 01:51:42.569517 [ 7638]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 01:51:42.570246 [ 7638]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/30 01:51:42.574766 [ 7638]: 00.ctdb: Set RecoverTimeout to 60 2014/10/30 01:51:42.578884 [ 7638]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/30 01:51:42.701508 [ 7638]: Freeze priority 1 2014/10/30 01:51:42.701586 [ 7638]: Freeze priority 2 2014/10/30 01:51:42.701639 [ 7638]: Freeze priority 3 2014/10/30 01:51:43.094042 [ 7638]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 01:51:43.166105 [ 7638]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 01:51:46.505061 [ 7638]: Freeze priority 1 2014/10/30 01:51:46.515471 [ 7638]: Freeze priority 2 2014/10/30 01:51:46.520928 [ 7638]: Freeze priority 3 2014/10/30 01:51:46.707693 [recoverd: 7923]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/30 01:51:46.707781 [ 7638]: Freeze priority 1 2014/10/30 01:51:46.707867 [ 7638]: Freeze priority 2 2014/10/30 01:51:46.707947 [ 7638]: Freeze priority 3 2014/10/30 01:51:50.069689 [ 7638]: Thawing priority 1 2014/10/30 01:51:50.069731 [ 7638]: Release freeze handler for prio 1 2014/10/30 01:51:50.069763 [ 7638]: Thawing priority 2 2014/10/30 01:51:50.069784 [ 7638]: Release freeze handler for prio 2 2014/10/30 01:51:50.069810 [ 7638]: Thawing priority 3 2014/10/30 01:51:50.069828 [ 7638]: Release freeze handler for prio 3 2014/10/30 01:51:52.094002 [ 7638]: Freeze priority 1 2014/10/30 01:51:52.095453 [ 7638]: Freeze priority 2 2014/10/30 01:51:52.096809 [ 7638]: Freeze priority 3 2014/10/30 01:51:52.263251 [ 7638]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/30 01:51:52.263850 [ 7638]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/30 01:51:52.265529 [ 7638]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup 2014/10/30 01:51:54.975981 [ 7638]: Thawing priority 1 2014/10/30 01:51:54.976054 [ 7638]: Release freeze handler for prio 1 2014/10/30 01:51:54.976117 [ 7638]: Thawing priority 2 2014/10/30 01:51:54.976142 [ 7638]: Release freeze handler for prio 2 2014/10/30 01:51:54.976186 [ 7638]: Thawing priority 3 2014/10/30 01:51:54.976205 [ 7638]: Release freeze handler for prio 3 2014/10/30 01:52:09.896451 [ 7638]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/30 01:52:10.002851 [recoverd: 7923]: Trigger takeoverrun 2014/10/30 01:52:10.720329 [ 7638]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 01:52:10.732338 [ 7638]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 01:52:10.760227 [ 7638]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/30 01:52:10.959626 [ 7638]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/30 01:52:13.398476 [ 7638]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation 2014/10/30 01:56:15.831445 [ 7638]: Freeze priority 1 2014/10/30 01:56:17.896360 [ 7638]: Skip monitoring since databases are frozen ===== Start of debug locks PID=24063 ===== 23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W 23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=23499 ----- #0 0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f97e1e49744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=24063 ===== 2014/10/30 01:56:32.896794 [ 7638]: Skip monitoring since databases are frozen ===== Start of debug locks PID=24422 ===== 23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W 23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=23499 ----- #0 0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f97e1e49744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=24422 ===== ===== Start of debug locks PID=24884 ===== 23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W 23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=23499 ----- #0 0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f97e1e49744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=24884 ===== 2014/10/30 01:56:47.897908 [ 7638]: Skip monitoring since databases are frozen ===== Start of debug locks PID=25373 ===== 23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W 23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=23499 ----- #0 0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f97e1e49744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=25373 ===== 2014/10/30 01:57:02.899012 [ 7638]: Skip monitoring since databases are frozen ===== Start of debug locks PID=25704 ===== 23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W 23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=23499 ----- #0 0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f97e1e49744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=25704 ===== 2014/10/30 01:57:15.849073 [ 7638]: Freeze priority 1 ===== Start of debug locks PID=26108 ===== 23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W 23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=23499 ----- #0 0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f97e1e49744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=26108 ===== 2014/10/30 01:57:17.899503 [ 7638]: Skip monitoring since databases are frozen ===== Start of debug locks PID=26586 ===== 23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W 23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=23499 ----- #0 0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f97e1e49744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=26586 ===== 2014/10/30 01:57:32.900138 [ 7638]: Skip monitoring since databases are frozen ===== Start of debug locks PID=26928 ===== 23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W 23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=23499 ----- #0 0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f97e1e49744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=26928 ===== ===== Start of debug locks PID=27349 ===== 23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W 23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=23499 ----- #0 0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f97e1e49744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=27349 ===== 2014/10/30 01:57:47.900344 [ 7638]: Skip monitoring since databases are frozen ===== Start of debug locks PID=27950 ===== 23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W 23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=23499 ----- #0 0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f97e1e49744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=27950 ===== 2014/10/30 01:58:02.900696 [ 7638]: Skip monitoring since databases are frozen ===== Start of debug locks PID=28857 ===== 23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W 23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=23499 ----- #0 0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f97e1e49744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=28857 ===== ===== Start of debug locks PID=29261 ===== 23499 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 23527 /usr/bin/ctdb_lock_helper locking.tdb.2 305944 305944 W 23499 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 23499 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=23499 ----- #0 0x00007f97e1e49890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f97e1e49744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fffe2e7ba18) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=29261 ===== 2014/10/30 01:58:16.504479 [ 7638]: Freeze priority 1 2014/10/30 01:58:16.505094 [ 7638]: Freeze priority 2 2014/10/30 01:58:16.506219 [ 7638]: Freeze priority 3 2014/10/30 01:58:19.525097 [ 7638]: Freeze priority 1 2014/10/30 01:58:19.525487 [ 7638]: Freeze priority 2 2014/10/30 01:58:19.525796 [ 7638]: Freeze priority 3 2014/10/30 01:58:22.934716 [ 7638]: Thawing priority 1 2014/10/30 01:58:22.934760 [ 7638]: Release freeze handler for prio 1 2014/10/30 01:58:22.934804 [ 7638]: Thawing priority 2 2014/10/30 01:58:22.934825 [ 7638]: Release freeze handler for prio 2 2014/10/30 01:58:22.934858 [ 7638]: Thawing priority 3 2014/10/30 01:58:22.934888 [ 7638]: Release freeze handler for prio 3 2014/10/30 01:58:23.290622 [ 7638]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 01:58:46.592550 [ 7638]: Freeze priority 1 2014/10/30 01:58:46.594777 [ 7638]: Freeze priority 2 2014/10/30 01:58:46.597023 [ 7638]: Freeze priority 3 2014/10/30 01:58:49.988048 [ 7638]: Thawing priority 1 2014/10/30 01:58:49.988102 [ 7638]: Release freeze handler for prio 1 2014/10/30 01:58:49.988141 [ 7638]: Thawing priority 2 2014/10/30 01:58:49.988165 [ 7638]: Release freeze handler for prio 2 2014/10/30 01:58:49.988216 [ 7638]: Thawing priority 3 2014/10/30 01:58:49.988232 [ 7638]: Release freeze handler for prio 3 2014/10/30 01:58:50.114793 [ 7638]: 10.interface: Killing TCP connection 10.10.10.205:54715 10.10.10.183:445 2014/10/30 01:58:50.114966 [ 7638]: 10.interface: Killing TCP connection 10.10.10.206:49692 10.10.10.183:445 2014/10/30 01:58:50.115095 [ 7638]: 10.interface: Killing TCP connection 10.10.10.208:55833 10.10.10.183:445 2014/10/30 01:58:50.138472 [ 7638]: 10.interface: Waiting for 2 connections to be killed for IP 10.10.10.183 2014/10/30 01:58:51.142827 [ 7638]: 10.interface: Killed 3 TCP connections to released IP 10.10.10.183 2014/10/30 01:58:51.151174 [ 7638]: 10.interface: Re-adding secondary address 10.10.10.182/24 to dev bond1 2014/10/30 01:58:51.545742 [ 7638]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 02:01:19.115676 [ 7638]: Freeze priority 1 2014/10/30 02:01:19.121665 [ 7638]: Freeze priority 2 2014/10/30 02:01:19.127993 [ 7638]: Freeze priority 3 2014/10/30 02:01:22.536733 [ 7638]: Thawing priority 1 2014/10/30 02:01:22.536825 [ 7638]: Release freeze handler for prio 1 2014/10/30 02:01:22.536866 [ 7638]: Thawing priority 2 2014/10/30 02:01:22.536899 [ 7638]: Release freeze handler for prio 2 2014/10/30 02:01:22.536933 [ 7638]: Thawing priority 3 2014/10/30 02:01:22.536953 [ 7638]: Release freeze handler for prio 3 2014/10/30 02:01:33.811722 [ 7638]: Freeze priority 1 2014/10/30 02:01:33.816157 [ 7638]: Freeze priority 2 2014/10/30 02:01:33.820637 [ 7638]: Freeze priority 3 2014/10/30 02:01:36.742206 [ 7638]: Thawing priority 1 2014/10/30 02:01:36.742264 [ 7638]: Release freeze handler for prio 1 2014/10/30 02:01:36.742305 [ 7638]: Thawing priority 2 2014/10/30 02:01:36.742329 [ 7638]: Release freeze handler for prio 2 2014/10/30 02:01:36.742361 [ 7638]: Thawing priority 3 2014/10/30 02:01:36.742382 [ 7638]: Release freeze handler for prio 3 2014/10/30 02:11:22.053583 [ 7638]: Freeze priority 1 2014/10/30 02:11:22.059544 [ 7638]: Freeze priority 2 2014/10/30 02:11:22.060649 [ 7638]: Freeze priority 3 2014/10/30 02:11:24.939058 [ 7638]: Thawing priority 1 2014/10/30 02:11:24.939132 [ 7638]: Release freeze handler for prio 1 2014/10/30 02:11:24.939169 [ 7638]: Thawing priority 2 2014/10/30 02:11:24.939187 [ 7638]: Release freeze handler for prio 2 2014/10/30 02:11:24.939225 [ 7638]: Thawing priority 3 2014/10/30 02:11:24.939241 [ 7638]: Release freeze handler for prio 3 2014/10/30 02:16:41.029396 [ 7638]: Freeze priority 1 2014/10/30 02:16:41.036342 [ 7638]: Freeze priority 2 2014/10/30 02:16:41.038555 [ 7638]: Freeze priority 3 2014/10/30 02:16:44.497775 [ 7638]: Thawing priority 1 2014/10/30 02:16:44.497810 [ 7638]: Release freeze handler for prio 1 2014/10/30 02:16:44.497852 [ 7638]: Thawing priority 2 2014/10/30 02:16:44.497870 [ 7638]: Release freeze handler for prio 2 2014/10/30 02:16:44.497923 [ 7638]: Thawing priority 3 2014/10/30 02:16:44.497946 [ 7638]: Release freeze handler for prio 3 2014/10/30 02:21:26.568922 [ 7638]: Freeze priority 1 2014/10/30 02:21:26.575640 [ 7638]: Freeze priority 1 2014/10/30 02:21:26.578669 [ 7638]: Freeze priority 2 2014/10/30 02:21:26.582962 [ 7638]: Freeze priority 2 2014/10/30 02:21:26.583195 [ 7638]: Freeze priority 3 2014/10/30 02:21:26.587978 [ 7638]: Freeze priority 3 2014/10/30 02:21:30.110244 [recoverd: 7923]: Taking out recovery lock from recovery daemon 2014/10/30 02:21:30.110329 [recoverd: 7923]: Take the recovery lock 2014/10/30 02:21:30.123476 [ 7638]: Freeze priority 1 2014/10/30 02:21:30.123922 [ 7638]: Freeze priority 2 2014/10/30 02:21:30.124297 [ 7638]: Freeze priority 3 2014/10/30 02:21:31.964760 [ 7638]: Thawing priority 1 2014/10/30 02:21:31.964812 [ 7638]: Release freeze handler for prio 1 2014/10/30 02:21:31.964849 [ 7638]: Thawing priority 2 2014/10/30 02:21:31.964874 [ 7638]: Release freeze handler for prio 2 2014/10/30 02:21:31.964928 [ 7638]: Thawing priority 3 2014/10/30 02:21:31.964952 [ 7638]: Release freeze handler for prio 3 2014/10/30 02:21:32.566902 [recoverd: 7923]: Resetting ban count to 0 for all nodes 2014/10/30 02:28:12.098849 [recoverd: 7923]: Taking out recovery lock from recovery daemon 2014/10/30 02:28:12.148209 [recoverd: 7923]: Take the recovery lock 2014/10/30 02:28:12.256201 [ 7638]: Freeze priority 1 2014/10/30 02:28:12.321698 [ 7638]: Freeze priority 2 2014/10/30 02:28:12.322949 [ 7638]: Freeze priority 3 2014/10/30 02:28:17.158071 [ 7638]: Thawing priority 1 2014/10/30 02:28:17.158138 [ 7638]: Release freeze handler for prio 1 2014/10/30 02:28:17.158182 [ 7638]: Thawing priority 2 2014/10/30 02:28:17.158203 [ 7638]: Release freeze handler for prio 2 2014/10/30 02:28:17.158231 [ 7638]: Thawing priority 3 2014/10/30 02:28:17.158248 [ 7638]: Release freeze handler for prio 3 2014/10/30 02:28:17.505566 [recoverd: 7923]: Resetting ban count to 0 for all nodes 2014/10/30 02:31:34.235331 [ 7638]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 02:31:34.247756 [ 7638]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 02:33:01.297452 [ 1515]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/30 02:33:01.297551 [ 1515]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 02:33:01.297565 [ 1515]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 02:37:00.913407 [ 7503]: Starting CTDBD (Version 2.5.3) as PID: 7503 2014/10/30 02:37:02.645312 [ 7503]: Vacuuming is disabled for persistent database registry.tdb 2014/10/30 02:37:02.669055 [ 7503]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/30 02:37:02.684931 [ 7503]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/30 02:37:02.699111 [ 7503]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/30 02:37:02.699137 [ 7503]: Ignoring persistent database 'account_policy.tdb.1' 2014/10/30 02:37:02.699148 [ 7503]: Ignoring persistent database 'ctdb.tdb.1' 2014/10/30 02:37:02.699157 [ 7503]: Ignoring persistent database 'group_mapping.tdb.1' 2014/10/30 02:37:02.699167 [ 7503]: Ignoring persistent database 'secrets.tdb.1' 2014/10/30 02:37:02.699176 [ 7503]: Ignoring persistent database 'share_info.tdb.1' 2014/10/30 02:37:02.713943 [ 7503]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/30 02:37:02.728574 [ 7503]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/30 02:37:02.728618 [ 7503]: Ignoring persistent database 'passdb.tdb.1' 2014/10/30 02:37:02.728628 [ 7503]: Ignoring persistent database 'registry.tdb.1' 2014/10/30 02:37:02.743180 [ 7503]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/30 02:37:02.743280 [ 7503]: Freeze priority 1 2014/10/30 02:37:02.765381 [ 7503]: Freeze priority 2 2014/10/30 02:37:02.773712 [ 7503]: Freeze priority 3 2014/10/30 02:37:02.954602 [ 7503]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/30 02:37:02.962452 [ 7503]: 00.ctdb: Set RecoverTimeout to 60 2014/10/30 02:37:02.965885 [ 7503]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/30 02:37:03.075670 [ 7503]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 02:37:03.085898 [ 7503]: Freeze priority 1 2014/10/30 02:37:03.085980 [ 7503]: Freeze priority 2 2014/10/30 02:37:03.086033 [ 7503]: Freeze priority 3 2014/10/30 02:37:07.093960 [recoverd: 7786]: server/ctdb_recoverd.c:3692 Current recmaster node 3 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/30 02:37:07.094040 [ 7503]: Freeze priority 1 2014/10/30 02:37:07.094101 [ 7503]: Freeze priority 2 2014/10/30 02:37:07.094155 [ 7503]: Freeze priority 3 2014/10/30 02:37:11.264358 [ 7503]: Freeze priority 1 2014/10/30 02:37:11.273756 [ 7503]: Freeze priority 2 2014/10/30 02:37:11.274653 [ 7503]: Freeze priority 3 2014/10/30 02:37:11.412515 [ 7503]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/30 02:37:11.413027 [ 7503]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/30 02:37:11.414361 [ 7503]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup 2014/10/30 02:37:13.793817 [ 7503]: Thawing priority 1 2014/10/30 02:37:13.793886 [ 7503]: Release freeze handler for prio 1 2014/10/30 02:37:13.793927 [ 7503]: Thawing priority 2 2014/10/30 02:37:13.793947 [ 7503]: Release freeze handler for prio 2 2014/10/30 02:37:13.793977 [ 7503]: Thawing priority 3 2014/10/30 02:37:13.793994 [ 7503]: Release freeze handler for prio 3 2014/10/30 02:37:15.795207 [recoverd: 7786]: Trigger takeoverrun 2014/10/30 02:37:15.926775 [ 7503]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/30 02:37:16.215936 [ 7503]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 02:37:16.226772 [ 7503]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 02:37:16.248890 [ 7503]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/30 02:37:16.421381 [ 7503]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/30 02:37:18.811484 [ 7503]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation 2014/10/30 02:37:24.772302 [ 7503]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 02:41:38.129001 [ 7503]: Freeze priority 1 2014/10/30 02:41:38.130501 [ 7503]: Freeze priority 1 2014/10/30 02:41:38.130909 [ 7503]: Freeze priority 2 2014/10/30 02:41:38.131379 [ 7503]: Freeze priority 2 2014/10/30 02:41:38.132702 [ 7503]: Freeze priority 3 2014/10/30 02:41:38.133255 [ 7503]: Freeze priority 3 2014/10/30 02:41:41.153191 [ 7503]: Freeze priority 1 2014/10/30 02:41:41.153555 [ 7503]: Freeze priority 2 2014/10/30 02:41:41.153934 [ 7503]: Freeze priority 3 2014/10/30 02:41:43.693639 [ 7503]: Thawing priority 1 2014/10/30 02:41:43.693693 [ 7503]: Release freeze handler for prio 1 2014/10/30 02:41:43.693727 [ 7503]: Thawing priority 2 2014/10/30 02:41:43.693749 [ 7503]: Release freeze handler for prio 2 2014/10/30 02:41:43.693779 [ 7503]: Thawing priority 3 2014/10/30 02:41:43.693798 [ 7503]: Release freeze handler for prio 3 2014/10/30 02:46:42.735558 [ 7503]: Freeze priority 1 2014/10/30 02:46:42.749472 [ 7503]: Freeze priority 2 2014/10/30 02:46:42.751270 [ 7503]: Freeze priority 3 2014/10/30 02:46:46.714760 [ 7503]: Thawing priority 1 2014/10/30 02:46:46.714841 [ 7503]: Release freeze handler for prio 1 2014/10/30 02:46:46.714880 [ 7503]: Thawing priority 2 2014/10/30 02:46:46.714898 [ 7503]: Release freeze handler for prio 2 2014/10/30 02:46:46.714926 [ 7503]: Thawing priority 3 2014/10/30 02:46:46.714941 [ 7503]: Release freeze handler for prio 3 2014/10/30 02:47:04.546243 [ 7503]: Monitoring event was cancelled 2014/10/30 02:47:04.546286 [ 7503]: server/eventscript.c:569 Sending SIGTERM to child pid:6864 2014/10/30 02:56:45.497013 [ 7503]: Freeze priority 1 2014/10/30 02:56:45.526888 [ 7503]: Freeze priority 1 2014/10/30 02:56:45.841148 [ 7503]: Freeze priority 1 2014/10/30 02:56:46.273086 [ 7503]: Skip monitoring since databases are frozen 2014/10/30 02:57:01.273590 [ 7503]: Skip monitoring since databases are frozen 2014/10/30 02:57:16.273885 [ 7503]: Skip monitoring since databases are frozen 2014/10/30 02:57:31.274837 [ 7503]: Skip monitoring since databases are frozen 2014/10/30 02:57:45.500574 [ 7503]: Freeze priority 1 2014/10/30 02:57:45.528732 [ 7503]: Freeze priority 1 2014/10/30 02:57:45.839845 [ 7503]: Recovery daemon ping timeout. Count : 0 2014/10/30 02:57:45.842052 [recoverd: 7786]: ctdb_control error: 'ctdb_control timed out' 2014/10/30 02:57:45.842105 [recoverd: 7786]: ctdb_control error: 'ctdb_control timed out' 2014/10/30 02:57:45.842131 [recoverd: 7786]: Async operation failed with ret=-1 res=-1 opcode=33 2014/10/30 02:57:45.842150 [recoverd: 7786]: Failed to freeze node 1 during recovery. Set it as ban culprit for 4 credits 2014/10/30 02:57:45.842171 [recoverd: 7786]: Async wait failed - fail_count=1 2014/10/30 02:57:45.842187 [recoverd: 7786]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed. 2014/10/30 02:57:45.842205 [recoverd: 7786]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster 2014/10/30 02:57:45.844141 [ 7503]: Freeze priority 1 2014/10/30 02:57:46.275940 [ 7503]: Skip monitoring since databases are frozen 2014/10/30 02:58:01.277022 [ 7503]: Skip monitoring since databases are frozen 2014/10/30 02:58:16.278077 [ 7503]: Skip monitoring since databases are frozen 2014/10/30 02:58:31.278506 [ 7503]: Skip monitoring since databases are frozen 2014/10/30 02:58:45.609319 [ 7503]: Freeze priority 1 2014/10/30 02:58:45.611461 [ 7503]: pnn 2 Invalid reqid 52689 in ctdb_reply_control 2014/10/30 02:58:45.611592 [ 7503]: Freeze priority 2 2014/10/30 02:58:45.611773 [ 7503]: Freeze priority 2 2014/10/30 02:58:45.612647 [ 7503]: Freeze priority 3 2014/10/30 02:58:45.612839 [ 7503]: Freeze priority 3 2014/10/30 02:58:45.842603 [ 7503]: Recovery daemon ping timeout. Count : 0 2014/10/30 02:58:48.621329 [recoverd: 7786]: Taking out recovery lock from recovery daemon 2014/10/30 02:58:48.621392 [recoverd: 7786]: Take the recovery lock 2014/10/30 02:58:48.633333 [ 7503]: Freeze priority 1 2014/10/30 02:58:48.633616 [ 7503]: Freeze priority 2 2014/10/30 02:58:48.633941 [ 7503]: Freeze priority 3 2014/10/30 02:58:50.360145 [ 7503]: Thawing priority 1 2014/10/30 02:58:50.360185 [ 7503]: Release freeze handler for prio 1 2014/10/30 02:58:50.360230 [ 7503]: Thawing priority 2 2014/10/30 02:58:50.360264 [ 7503]: Release freeze handler for prio 2 2014/10/30 02:58:50.360299 [ 7503]: Thawing priority 3 2014/10/30 02:58:50.360320 [ 7503]: Release freeze handler for prio 3 2014/10/30 02:58:50.362363 [ 7503]: server/ctdb_call.c:1005 reqid 53362 not found 2014/10/30 02:58:50.362409 [ 7503]: server/ctdb_call.c:1005 reqid 53363 not found 2014/10/30 02:58:50.747988 [ 7503]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 02:58:50.947847 [recoverd: 7786]: Resetting ban count to 0 for all nodes 2014/10/30 02:59:15.981154 [recoverd: 7786]: Taking out recovery lock from recovery daemon 2014/10/30 02:59:15.981213 [recoverd: 7786]: Take the recovery lock 2014/10/30 02:59:16.009161 [ 7503]: Freeze priority 1 2014/10/30 02:59:16.027925 [ 7503]: Freeze priority 2 2014/10/30 02:59:16.029506 [ 7503]: Freeze priority 3 2014/10/30 02:59:19.235865 [ 7503]: Thawing priority 1 2014/10/30 02:59:19.235929 [ 7503]: Release freeze handler for prio 1 2014/10/30 02:59:19.235970 [ 7503]: Thawing priority 2 2014/10/30 02:59:19.235990 [ 7503]: Release freeze handler for prio 2 2014/10/30 02:59:19.236024 [ 7503]: Thawing priority 3 2014/10/30 02:59:19.236044 [ 7503]: Release freeze handler for prio 3 2014/10/30 02:59:19.927863 [recoverd: 7786]: Resetting ban count to 0 for all nodes 2014/10/30 03:02:08.156267 [recoverd: 7786]: Taking out recovery lock from recovery daemon 2014/10/30 03:02:08.156350 [recoverd: 7786]: Take the recovery lock 2014/10/30 03:02:08.305469 [ 7503]: Freeze priority 1 2014/10/30 03:02:08.320271 [ 7503]: Freeze priority 2 2014/10/30 03:02:08.324911 [ 7503]: Freeze priority 3 2014/10/30 03:02:11.180106 [ 7503]: Thawing priority 1 2014/10/30 03:02:11.180158 [ 7503]: Release freeze handler for prio 1 2014/10/30 03:02:11.180192 [ 7503]: Thawing priority 2 2014/10/30 03:02:11.180228 [ 7503]: Release freeze handler for prio 2 2014/10/30 03:02:11.180260 [ 7503]: Thawing priority 3 2014/10/30 03:02:11.180279 [ 7503]: Release freeze handler for prio 3 2014/10/30 03:02:11.563575 [recoverd: 7786]: Resetting ban count to 0 for all nodes 2014/10/30 03:02:29.694994 [ 7503]: 10.interface: Killing TCP connection 10.10.10.205:54728 10.10.10.182:445 2014/10/30 03:02:29.708249 [ 7503]: 10.interface: Killed 1 TCP connections to released IP 10.10.10.182 2014/10/30 03:02:29.718685 [ 7503]: 10.interface: Re-adding secondary address 10.10.10.22/24 to dev bond1 2014/10/30 03:02:30.117155 [ 7503]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 03:06:47.925902 [recoverd: 7786]: server/ctdb_recoverd.c:3960 The vnnmap count is different from the number of active lmaster nodes: 4 vs 3 2014/10/30 03:06:47.925984 [recoverd: 7786]: Taking out recovery lock from recovery daemon 2014/10/30 03:06:47.926000 [recoverd: 7786]: Take the recovery lock 2014/10/30 03:06:47.942371 [ 7503]: Freeze priority 1 2014/10/30 03:06:57.570492 [ 7503]: Skip monitoring since databases are frozen 2014/10/30 03:07:12.570763 [ 7503]: Skip monitoring since databases are frozen 2014/10/30 03:07:27.571127 [ 7503]: Skip monitoring since databases are frozen 2014/10/30 03:07:42.572238 [ 7503]: Skip monitoring since databases are frozen 2014/10/30 03:07:47.917346 [ 7503]: Recovery daemon ping timeout. Count : 0 2014/10/30 03:07:47.942548 [recoverd: 7786]: ctdb_control error: 'ctdb_control timed out' 2014/10/30 03:07:47.942598 [recoverd: 7786]: ctdb_control error: 'ctdb_control timed out' 2014/10/30 03:07:47.942620 [recoverd: 7786]: Async operation failed with ret=-1 res=-1 opcode=33 2014/10/30 03:07:47.942634 [recoverd: 7786]: Failed to freeze node 0 during recovery. Set it as ban culprit for 4 credits 2014/10/30 03:07:47.942652 [recoverd: 7786]: Async wait failed - fail_count=1 2014/10/30 03:07:47.942666 [recoverd: 7786]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed. 2014/10/30 03:07:47.942680 [recoverd: 7786]: server/ctdb_recoverd.c:1833 Unable to set recovery mode to active on cluster 2014/10/30 03:07:47.946158 [recoverd: 7786]: Taking out recovery lock from recovery daemon 2014/10/30 03:07:47.946210 [recoverd: 7786]: Take the recovery lock 2014/10/30 03:07:47.960600 [ 7503]: Freeze priority 1 2014/10/30 03:07:57.573259 [ 7503]: Skip monitoring since databases are frozen 2014/10/30 03:08:12.574041 [ 7503]: Skip monitoring since databases are frozen 2014/10/30 03:08:27.574320 [ 7503]: Skip monitoring since databases are frozen 2014/10/30 03:08:42.575168 [ 7503]: Skip monitoring since databases are frozen 2014/10/30 03:08:47.943575 [ 7503]: Recovery daemon ping timeout. Count : 0 2014/10/30 03:08:47.960779 [recoverd: 7786]: ctdb_control error: 'ctdb_control timed out' 2014/10/30 03:08:47.960836 [recoverd: 7786]: ctdb_control error: 'ctdb_control timed out' 2014/10/30 03:08:47.960857 [recoverd: 7786]: Async operation failed with ret=-1 res=-1 opcode=33 2014/10/30 03:08:47.960870 [recoverd: 7786]: Failed to freeze node 0 during recovery. Set it as ban culprit for 4 credits 2014/10/30 03:08:47.960885 [recoverd: 7786]: Async wait failed - fail_count=1 2014/10/30 03:08:47.960898 [recoverd: 7786]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed. 2014/10/30 03:08:47.960910 [recoverd: 7786]: server/ctdb_recoverd.c:1833 Unable to set recovery mode to active on cluster 2014/10/30 03:08:48.181408 [ 7503]: pnn 2 Invalid reqid 84199 in ctdb_reply_control 2014/10/30 03:08:48.181452 [ 7503]: pnn 2 Invalid reqid 83859 in ctdb_reply_control 2014/10/30 03:08:48.183494 [recoverd: 7786]: Taking out recovery lock from recovery daemon 2014/10/30 03:08:48.183543 [recoverd: 7786]: Take the recovery lock 2014/10/30 03:08:48.197266 [ 7503]: Freeze priority 1 2014/10/30 03:08:48.197593 [ 7503]: Freeze priority 2 2014/10/30 03:08:48.198573 [ 7503]: Freeze priority 3 2014/10/30 03:08:49.975840 [ 7503]: Thawing priority 1 2014/10/30 03:08:49.975891 [ 7503]: Release freeze handler for prio 1 2014/10/30 03:08:49.975939 [ 7503]: Thawing priority 2 2014/10/30 03:08:49.975975 [ 7503]: Release freeze handler for prio 2 2014/10/30 03:08:49.976006 [ 7503]: Thawing priority 3 2014/10/30 03:08:49.976037 [ 7503]: Release freeze handler for prio 3 2014/10/30 03:08:50.334773 [ 7503]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 03:08:50.555091 [recoverd: 7786]: Resetting ban count to 0 for all nodes 2014/10/30 03:09:18.593542 [recoverd: 7786]: Taking out recovery lock from recovery daemon 2014/10/30 03:09:18.593595 [recoverd: 7786]: Take the recovery lock 2014/10/30 03:09:18.662372 [ 7503]: Freeze priority 1 2014/10/30 03:09:18.679631 [ 7503]: Freeze priority 2 2014/10/30 03:09:18.682498 [ 7503]: Freeze priority 3 2014/10/30 03:09:20.515392 [ 7503]: Thawing priority 1 2014/10/30 03:09:20.515454 [ 7503]: Release freeze handler for prio 1 2014/10/30 03:09:20.515494 [ 7503]: Thawing priority 2 2014/10/30 03:09:20.515512 [ 7503]: Release freeze handler for prio 2 2014/10/30 03:09:20.515538 [ 7503]: Thawing priority 3 2014/10/30 03:09:20.515563 [ 7503]: Release freeze handler for prio 3 2014/10/30 03:09:22.294252 [recoverd: 7786]: Resetting ban count to 0 for all nodes 2014/10/30 03:13:26.620019 [recoverd: 7786]: Taking out recovery lock from recovery daemon 2014/10/30 03:13:26.620069 [recoverd: 7786]: Take the recovery lock 2014/10/30 03:13:26.752064 [ 7503]: Freeze priority 1 2014/10/30 03:13:26.769881 [ 7503]: Freeze priority 2 2014/10/30 03:13:26.774293 [ 7503]: Freeze priority 3 2014/10/30 03:13:30.744914 [ 7503]: Thawing priority 1 2014/10/30 03:13:30.745021 [ 7503]: Release freeze handler for prio 1 2014/10/30 03:13:30.745103 [ 7503]: Thawing priority 2 2014/10/30 03:13:30.745124 [ 7503]: Release freeze handler for prio 2 2014/10/30 03:13:30.745163 [ 7503]: Thawing priority 3 2014/10/30 03:13:30.745179 [ 7503]: Release freeze handler for prio 3 2014/10/30 03:13:31.118725 [recoverd: 7786]: Resetting ban count to 0 for all nodes 2014/10/30 03:13:41.353980 [recoverd: 7786]: server/ctdb_recoverd.c:3933 Remote node:1 has different flags for node 0. It has 0x02 vs our 0x00 2014/10/30 03:13:41.354026 [recoverd: 7786]: Use flags 0x00 from local recmaster node for cluster update of node 0 flags 2014/10/30 03:13:41.359547 [recoverd: 7786]: Taking out recovery lock from recovery daemon 2014/10/30 03:13:41.359574 [recoverd: 7786]: Take the recovery lock 2014/10/30 03:13:41.479985 [ 7503]: Freeze priority 1 2014/10/30 03:13:41.486114 [ 7503]: Freeze priority 2 2014/10/30 03:13:41.491669 [ 7503]: Freeze priority 3 2014/10/30 03:13:44.303373 [ 7503]: Thawing priority 1 2014/10/30 03:13:44.303438 [ 7503]: Release freeze handler for prio 1 2014/10/30 03:13:44.303477 [ 7503]: Thawing priority 2 2014/10/30 03:13:44.303495 [ 7503]: Release freeze handler for prio 2 2014/10/30 03:13:44.303522 [ 7503]: Thawing priority 3 2014/10/30 03:13:44.303537 [ 7503]: Release freeze handler for prio 3 2014/10/30 03:13:44.723345 [recoverd: 7786]: Resetting ban count to 0 for all nodes 2014/10/30 03:14:03.370953 [ 7503]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 03:16:51.010354 [ 7503]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 03:16:51.022907 [ 7503]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 03:18:15.406944 [ 1533]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/30 03:18:15.407046 [ 1533]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 03:18:15.407060 [ 1533]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 03:22:17.671433 [ 7588]: Starting CTDBD (Version 2.5.3) as PID: 7588 2014/10/30 03:22:19.234406 [ 7588]: Vacuuming is disabled for persistent database registry.tdb 2014/10/30 03:22:19.258034 [ 7588]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/30 03:22:19.272258 [ 7588]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/30 03:22:19.286191 [ 7588]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/30 03:22:19.286210 [ 7588]: Ignoring persistent database 'account_policy.tdb.1' 2014/10/30 03:22:19.286219 [ 7588]: Ignoring persistent database 'ctdb.tdb.1' 2014/10/30 03:22:19.286242 [ 7588]: Ignoring persistent database 'group_mapping.tdb.1' 2014/10/30 03:22:19.286251 [ 7588]: Ignoring persistent database 'secrets.tdb.1' 2014/10/30 03:22:19.286259 [ 7588]: Ignoring persistent database 'share_info.tdb.1' 2014/10/30 03:22:19.300836 [ 7588]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/30 03:22:19.315182 [ 7588]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/30 03:22:19.315220 [ 7588]: Ignoring persistent database 'passdb.tdb.1' 2014/10/30 03:22:19.315229 [ 7588]: Ignoring persistent database 'registry.tdb.1' 2014/10/30 03:22:19.329221 [ 7588]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/30 03:22:19.329289 [ 7588]: Freeze priority 1 2014/10/30 03:22:19.346548 [ 7588]: Freeze priority 2 2014/10/30 03:22:19.346937 [ 7588]: Freeze priority 3 2014/10/30 03:22:19.510306 [ 7588]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 03:22:19.511273 [ 7588]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/30 03:22:19.515314 [ 7588]: 00.ctdb: Set RecoverTimeout to 60 2014/10/30 03:22:19.519011 [ 7588]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/30 03:22:19.639953 [ 7588]: Freeze priority 1 2014/10/30 03:22:19.640029 [ 7588]: Freeze priority 2 2014/10/30 03:22:19.640083 [ 7588]: Freeze priority 3 2014/10/30 03:22:20.074367 [ 7588]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 03:22:25.264022 [ 7588]: Freeze priority 1 2014/10/30 03:22:25.268474 [ 7588]: Freeze priority 2 2014/10/30 03:22:25.271899 [ 7588]: Freeze priority 3 2014/10/30 03:22:25.474461 [ 7588]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/30 03:22:25.477839 [ 7588]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/30 03:22:25.487353 [ 7588]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup 2014/10/30 03:22:29.050428 [ 7588]: Thawing priority 1 2014/10/30 03:22:29.050471 [ 7588]: Release freeze handler for prio 1 2014/10/30 03:22:29.050499 [ 7588]: Thawing priority 2 2014/10/30 03:22:29.050509 [ 7588]: Release freeze handler for prio 2 2014/10/30 03:22:29.050525 [ 7588]: Thawing priority 3 2014/10/30 03:22:29.050546 [ 7588]: Release freeze handler for prio 3 2014/10/30 03:22:43.066351 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:22:43.348291 [ 7588]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/30 03:22:43.606709 [ 7588]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 03:22:43.620142 [ 7588]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 03:22:43.647203 [ 7588]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/30 03:22:43.847006 [ 7588]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/30 03:22:46.276453 [ 7588]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation 2014/10/30 03:22:47.039955 [ 7588]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 03:26:55.376077 [ 7588]: Freeze priority 1 2014/10/30 03:26:55.382059 [ 7588]: Freeze priority 1 2014/10/30 03:26:55.392137 [ 7588]: Freeze priority 1 2014/10/30 03:26:55.394287 [ 7588]: Freeze priority 2 2014/10/30 03:26:55.394613 [ 7588]: Freeze priority 2 2014/10/30 03:26:55.394662 [ 7588]: Freeze priority 2 2014/10/30 03:26:55.395532 [ 7588]: Freeze priority 3 2014/10/30 03:26:55.395684 [ 7588]: Freeze priority 3 2014/10/30 03:26:55.395778 [ 7588]: Freeze priority 3 2014/10/30 03:26:58.916488 [ 7588]: Freeze priority 1 2014/10/30 03:26:58.916825 [ 7588]: Freeze priority 2 2014/10/30 03:26:58.917196 [ 7588]: Freeze priority 3 2014/10/30 03:27:01.618601 [ 7588]: Thawing priority 1 2014/10/30 03:27:01.618657 [ 7588]: Release freeze handler for prio 1 2014/10/30 03:27:01.618704 [ 7588]: Thawing priority 2 2014/10/30 03:27:01.618738 [ 7588]: Release freeze handler for prio 2 2014/10/30 03:27:01.618776 [ 7588]: Thawing priority 3 2014/10/30 03:27:01.618798 [ 7588]: Release freeze handler for prio 3 2014/10/30 03:27:01.621713 [ 7588]: pnn 2 Invalid reqid 14251 in ctdb_become_dmaster from node 0 2014/10/30 03:27:01.621814 [ 7588]: server/ctdb_call.c:1005 reqid 14253 not found 2014/10/30 03:27:01.621967 [ 7588]: pnn 2 Invalid reqid 14252 in ctdb_become_dmaster from node 1 2014/10/30 03:27:01.622025 [ 7588]: server/ctdb_call.c:1005 reqid 14254 not found 2014/10/30 03:31:59.045042 [recoverd: 7955]: Taking out recovery lock from recovery daemon 2014/10/30 03:31:59.099112 [recoverd: 7955]: Take the recovery lock 2014/10/30 03:31:59.257530 [ 7588]: Freeze priority 1 2014/10/30 03:31:59.335449 [ 7588]: Freeze priority 2 2014/10/30 03:31:59.338024 [ 7588]: Freeze priority 3 2014/10/30 03:32:04.090335 [ 7588]: Thawing priority 1 2014/10/30 03:32:04.090402 [ 7588]: Release freeze handler for prio 1 2014/10/30 03:32:04.090451 [ 7588]: Thawing priority 2 2014/10/30 03:32:04.090473 [ 7588]: Release freeze handler for prio 2 2014/10/30 03:32:04.090504 [ 7588]: Thawing priority 3 2014/10/30 03:32:04.090523 [ 7588]: Release freeze handler for prio 3 2014/10/30 03:32:04.103670 [ 7588]: Freeze priority 1 2014/10/30 03:32:04.105126 [ 7588]: Freeze priority 2 2014/10/30 03:32:04.106378 [ 7588]: Freeze priority 3 2014/10/30 03:32:04.300413 [ 7588]: Refusing to run event scripts call 'recovered' while in recovery 2014/10/30 03:32:04.300460 [ 7588]: server/ctdb_recover.c:952 Failed to end recovery 2014/10/30 03:32:04.300496 [recoverd: 7955]: Async operation failed with ret=0 res=-1 opcode=71 2014/10/30 03:32:04.300527 [recoverd: 7955]: server/ctdb_recoverd.c:239 Node 2 failed the recovered event. Setting it as recovery fail culprit 2014/10/30 03:32:04.300710 [recoverd: 7955]: Async operation failed with ret=0 res=-1 opcode=71 2014/10/30 03:32:04.300722 [recoverd: 7955]: server/ctdb_recoverd.c:239 Node 0 failed the recovered event. Setting it as recovery fail culprit 2014/10/30 03:32:04.300743 [recoverd: 7955]: Async operation failed with ret=0 res=-1 opcode=71 2014/10/30 03:32:04.300754 [recoverd: 7955]: server/ctdb_recoverd.c:239 Node 3 failed the recovered event. Setting it as recovery fail culprit 2014/10/30 03:32:04.300767 [recoverd: 7955]: Async operation failed with ret=0 res=-1 opcode=71 2014/10/30 03:32:04.300776 [recoverd: 7955]: server/ctdb_recoverd.c:239 Node 1 failed the recovered event. Setting it as recovery fail culprit 2014/10/30 03:32:04.300787 [recoverd: 7955]: Async wait failed - fail_count=4 2014/10/30 03:32:04.300797 [recoverd: 7955]: server/ctdb_recoverd.c:262 Unable to run the 'recovered' event when called from do_recovery 2014/10/30 03:32:04.300806 [recoverd: 7955]: server/ctdb_recoverd.c:2016 Unable to run the 'recovered' event on cluster. Recovery process failed. 2014/10/30 03:32:04.319310 [recoverd: 7955]: Taking out recovery lock from recovery daemon 2014/10/30 03:32:04.319372 [recoverd: 7955]: Take the recovery lock 2014/10/30 03:32:04.322003 [recoverd: 7955]: ctdb_recovery_lock: Failed to get recovery lock on '/mnt/lock/lockfile' 2014/10/30 03:32:04.322034 [recoverd: 7955]: Unable to get recovery lock - aborting recovery and ban ourself for 30 seconds 2014/10/30 03:32:04.322090 [ 7588]: Banning this node for 30 seconds 2014/10/30 03:32:04.497384 [ 7588]: 10.interface: Killing TCP connection 10.10.10.208:57594 10.10.10.183:445 2014/10/30 03:32:04.497498 [ 7588]: 10.interface: Killing TCP connection 10.10.10.206:49708 10.10.10.183:445 2014/10/30 03:32:04.517070 [ 7588]: 10.interface: Killed 2 TCP connections to released IP 10.10.10.183 2014/10/30 03:32:06.891513 [ 7588]: Thawing priority 1 2014/10/30 03:32:06.891553 [ 7588]: Release freeze handler for prio 1 2014/10/30 03:32:06.891584 [ 7588]: Thawing priority 2 2014/10/30 03:32:06.891597 [ 7588]: Release freeze handler for prio 2 2014/10/30 03:32:06.891617 [ 7588]: Thawing priority 3 2014/10/30 03:32:06.891628 [ 7588]: Release freeze handler for prio 3 2014/10/30 03:32:07.226393 [ 7588]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 03:32:08.304351 [recoverd: 7955]: Node is stopped or banned but recovery mode is not active. Activate recovery mode and lock databases 2014/10/30 03:32:08.304437 [ 7588]: Freeze priority 1 2014/10/30 03:32:10.102476 [ 7588]: DB Attach to database ctdb.tdb refused since node is inactive (flags=0x8) 2014/10/30 03:32:17.432653 [ 7588]: Freeze priority 2 2014/10/30 03:32:17.432838 [ 7588]: Freeze priority 3 2014/10/30 03:32:17.476718 [ 7588]: 10.interface: Killing TCP connection 10.10.10.208:57788 10.10.10.183:445 2014/10/30 03:32:17.482592 [ 7588]: 10.interface: Killed 1 TCP connections to released IP 10.10.10.183 2014/10/30 03:32:34.322951 [ 7588]: Banning timedout 2014/10/30 03:32:34.343571 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:34.343714 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:35.344859 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:35.344996 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:36.347636 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:36.347816 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:37.349615 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:37.349817 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:38.350890 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:38.351115 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:39.352767 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:39.352990 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:40.354511 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:40.354745 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:41.355307 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:41.355448 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:42.358255 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:42.358465 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:43.359842 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:43.360015 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:44.361524 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:44.361683 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:45.363477 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:45.363635 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:46.364399 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:46.364513 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:47.366990 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:47.367118 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:48.367994 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:48.368186 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:49.369828 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:49.370001 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:50.371810 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:50.371957 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:51.372780 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:51.372886 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:52.375139 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:52.375250 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:53.377136 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:53.377254 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:54.378530 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:54.378715 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:55.380459 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:55.380636 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:56.382373 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:56.382511 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:57.384189 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:57.384322 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:58.385483 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:58.385606 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:32:59.387322 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:32:59.387500 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:00.389156 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:00.389315 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:01.391140 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:01.391339 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:02.392959 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:02.393144 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:03.395003 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:03.395214 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:04.395798 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:04.395974 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:05.397532 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:05.397675 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:06.399294 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:06.399452 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:07.400647 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:07.400883 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:08.401717 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:08.401919 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:09.404185 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:09.404342 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:10.405469 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:10.405582 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:11.407345 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:11.407541 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:12.409329 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:12.409534 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:13.411189 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:13.411378 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:14.412247 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:14.412444 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:15.413163 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:15.413309 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:16.414874 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:16.415025 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:17.416997 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:17.417192 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:18.418613 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:18.418818 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:18.472498 [ 7588]: Freeze priority 1 2014/10/30 03:33:19.420103 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:19.420279 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:20.421870 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:20.422052 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:21.423959 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:21.424165 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:22.424305 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:22.424415 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:23.426092 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:23.426226 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:24.428039 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:24.428227 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:25.429934 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:25.430134 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:26.601429 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:26.601623 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:27.432826 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:27.432963 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:28.434699 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:28.434920 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:29.436054 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:29.436177 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:30.437574 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:30.437724 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:31.439025 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:31.439154 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:32.440731 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:32.440844 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:33.442712 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:33.442893 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:34.444529 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:34.444694 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:35.446108 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:35.446263 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:36.447759 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:36.447940 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:37.450006 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:37.450205 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:38.451004 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:38.451178 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:39.452875 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:39.453057 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:40.454592 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:40.454756 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:41.455187 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:41.455398 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:42.457234 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:42.457438 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:43.459649 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:43.459790 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:44.461537 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:44.461758 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:45.463211 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:45.463376 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:46.465040 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:46.465246 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:47.466843 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:47.467039 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:48.467611 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:48.467837 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:49.469478 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:49.469711 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:50.470409 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:50.470514 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:51.472879 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:51.473020 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:52.473280 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:52.473364 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:53.475945 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:53.476150 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:54.477872 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:54.478044 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:55.479638 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:55.479779 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:56.481257 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:56.481404 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:57.483065 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:57.483256 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:58.484610 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:58.484742 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:33:59.486371 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:33:59.486581 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:34:00.487767 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:34:00.487911 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:34:01.489396 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:34:01.489546 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:34:02.491169 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:34:02.491316 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:34:03.492951 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:34:03.493087 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:34:04.494404 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:34:04.494531 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:34:05.496204 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:34:05.496358 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:34:06.498060 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:34:06.498183 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:34:07.499578 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:34:07.499733 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:34:08.306355 [ 7588]: server/ctdb_recover.c:562 Been in recovery mode for too long. Dropping all IPS 2014/10/30 03:34:08.501924 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:34:08.502083 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:34:09.503735 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:34:09.503946 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:34:10.505610 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:34:10.505741 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:34:11.507143 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:34:11.507296 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:34:12.509282 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:34:12.509442 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:34:13.510987 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:34:13.511153 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:34:14.511876 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:34:14.512048 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:34:15.512955 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:34:15.513103 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:34:16.514449 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:34:16.514574 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:34:17.515783 [recoverd: 7955]: Public IP '10.10.10.183' is not assigned and we could serve it 2014/10/30 03:34:17.516016 [recoverd: 7955]: Trigger takeoverrun 2014/10/30 03:34:18.517832 [ 7588]: Freeze priority 1 2014/10/30 03:34:18.518371 [ 7588]: Freeze priority 2 2014/10/30 03:34:18.519562 [ 7588]: Freeze priority 3 2014/10/30 03:34:22.100215 [ 7588]: Freeze priority 1 2014/10/30 03:34:22.100630 [ 7588]: Freeze priority 2 2014/10/30 03:34:22.100944 [ 7588]: Freeze priority 3 2014/10/30 03:34:24.700205 [ 7588]: Thawing priority 1 2014/10/30 03:34:24.700247 [ 7588]: Release freeze handler for prio 1 2014/10/30 03:34:24.700287 [ 7588]: Thawing priority 2 2014/10/30 03:34:24.700310 [ 7588]: Release freeze handler for prio 2 2014/10/30 03:34:24.700343 [ 7588]: Thawing priority 3 2014/10/30 03:34:24.700362 [ 7588]: Release freeze handler for prio 3 2014/10/30 03:34:24.709824 [ 7588]: pnn 2 Invalid reqid 32094 in ctdb_become_dmaster from node 1 2014/10/30 03:34:24.709990 [ 7588]: server/ctdb_call.c:1005 reqid 32095 not found 2014/10/30 03:34:25.298089 [ 7588]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 03:34:36.082492 [ 7588]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 03:34:48.770118 [ 7588]: Freeze priority 1 2014/10/30 03:34:48.778944 [ 7588]: Freeze priority 2 2014/10/30 03:34:48.785468 [ 7588]: Freeze priority 3 2014/10/30 03:34:52.036874 [ 7588]: Thawing priority 1 2014/10/30 03:34:52.036923 [ 7588]: Release freeze handler for prio 1 2014/10/30 03:34:52.036960 [ 7588]: Thawing priority 2 2014/10/30 03:34:52.036980 [ 7588]: Release freeze handler for prio 2 2014/10/30 03:34:52.037010 [ 7588]: Thawing priority 3 2014/10/30 03:34:52.037029 [ 7588]: Release freeze handler for prio 3 2014/10/30 03:42:01.561496 [ 7588]: Freeze priority 1 2014/10/30 03:42:01.662581 [ 7588]: Freeze priority 2 2014/10/30 03:42:01.668700 [ 7588]: Freeze priority 3 2014/10/30 03:42:04.914056 [ 7588]: Thawing priority 1 2014/10/30 03:42:04.914113 [ 7588]: Release freeze handler for prio 1 2014/10/30 03:42:04.914142 [ 7588]: Thawing priority 2 2014/10/30 03:42:04.914169 [ 7588]: Release freeze handler for prio 2 2014/10/30 03:42:04.914194 [ 7588]: Thawing priority 3 2014/10/30 03:42:04.914209 [ 7588]: Release freeze handler for prio 3 2014/10/30 03:47:22.946969 [ 7588]: Freeze priority 1 2014/10/30 03:47:23.009728 [ 7588]: Freeze priority 2 2014/10/30 03:47:23.012149 [ 7588]: Freeze priority 3 2014/10/30 03:47:28.389886 [ 7588]: Thawing priority 1 2014/10/30 03:47:28.389937 [ 7588]: Release freeze handler for prio 1 2014/10/30 03:47:28.389971 [ 7588]: Thawing priority 2 2014/10/30 03:47:28.389990 [ 7588]: Release freeze handler for prio 2 2014/10/30 03:47:28.390017 [ 7588]: Thawing priority 3 2014/10/30 03:47:28.390033 [ 7588]: Release freeze handler for prio 3 2014/10/30 03:52:06.238108 [ 7588]: Freeze priority 1 2014/10/30 03:52:06.325681 [ 7588]: Freeze priority 2 2014/10/30 03:52:06.327546 [ 7588]: Freeze priority 3 2014/10/30 03:52:06.761278 [ 7588]: Freeze priority 1 2014/10/30 03:52:06.761736 [ 7588]: Freeze priority 2 2014/10/30 03:52:06.762447 [ 7588]: Freeze priority 3 2014/10/30 03:52:09.785923 [ 7588]: Freeze priority 1 2014/10/30 03:52:09.786240 [ 7588]: Freeze priority 2 2014/10/30 03:52:09.786581 [ 7588]: Freeze priority 3 2014/10/30 03:52:12.148658 [ 7588]: Thawing priority 1 2014/10/30 03:52:12.148699 [ 7588]: Release freeze handler for prio 1 2014/10/30 03:52:12.148734 [ 7588]: Thawing priority 2 2014/10/30 03:52:12.148755 [ 7588]: Release freeze handler for prio 2 2014/10/30 03:52:12.148785 [ 7588]: Thawing priority 3 2014/10/30 03:52:12.148804 [ 7588]: Release freeze handler for prio 3 2014/10/30 03:58:50.345012 [ 7588]: Freeze priority 1 2014/10/30 03:58:50.436617 [ 7588]: Freeze priority 2 2014/10/30 03:58:50.438775 [ 7588]: Freeze priority 3 2014/10/30 03:58:53.746872 [ 7588]: Thawing priority 1 2014/10/30 03:58:53.746966 [ 7588]: Release freeze handler for prio 1 2014/10/30 03:58:53.747024 [ 7588]: Thawing priority 2 2014/10/30 03:58:53.747043 [ 7588]: Release freeze handler for prio 2 2014/10/30 03:58:53.747081 [ 7588]: Thawing priority 3 2014/10/30 03:58:53.747098 [ 7588]: Release freeze handler for prio 3 2014/10/30 03:59:04.441232 [ 7588]: Freeze priority 1 2014/10/30 03:59:04.463043 [ 7588]: Freeze priority 2 2014/10/30 03:59:04.465833 [ 7588]: Freeze priority 3 2014/10/30 03:59:07.613588 [ 7588]: Thawing priority 1 2014/10/30 03:59:07.613626 [ 7588]: Release freeze handler for prio 1 2014/10/30 03:59:07.613659 [ 7588]: Thawing priority 2 2014/10/30 03:59:07.613680 [ 7588]: Release freeze handler for prio 2 2014/10/30 03:59:07.613710 [ 7588]: Thawing priority 3 2014/10/30 03:59:07.613729 [ 7588]: Release freeze handler for prio 3 2014/10/30 04:02:15.338221 [ 7588]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 04:02:15.414907 [ 7588]: common/ctdb_fork.c:131 waitpid() returned error. errno:10 2014/10/30 04:02:15.414943 [ 7588]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 04:03:44.616786 [ 1551]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/30 04:03:44.616903 [ 1551]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 04:03:44.616917 [ 1551]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 04:07:46.192240 [ 7647]: Starting CTDBD (Version 2.5.3) as PID: 7647 2014/10/30 04:07:47.785871 [ 7647]: Vacuuming is disabled for persistent database registry.tdb 2014/10/30 04:07:47.811655 [ 7647]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/30 04:07:47.826072 [ 7647]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/30 04:07:47.840102 [ 7647]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/30 04:07:47.840121 [ 7647]: Ignoring persistent database 'account_policy.tdb.1' 2014/10/30 04:07:47.840131 [ 7647]: Ignoring persistent database 'ctdb.tdb.1' 2014/10/30 04:07:47.840140 [ 7647]: Ignoring persistent database 'group_mapping.tdb.1' 2014/10/30 04:07:47.840149 [ 7647]: Ignoring persistent database 'secrets.tdb.1' 2014/10/30 04:07:47.840168 [ 7647]: Ignoring persistent database 'share_info.tdb.1' 2014/10/30 04:07:47.854144 [ 7647]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/30 04:07:47.868074 [ 7647]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/30 04:07:47.868093 [ 7647]: Ignoring persistent database 'passdb.tdb.1' 2014/10/30 04:07:47.868102 [ 7647]: Ignoring persistent database 'registry.tdb.1' 2014/10/30 04:07:47.882256 [ 7647]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/30 04:07:47.882292 [ 7647]: Freeze priority 1 2014/10/30 04:07:47.902607 [ 7647]: Freeze priority 2 2014/10/30 04:07:47.903021 [ 7647]: Freeze priority 3 2014/10/30 04:07:48.067160 [ 7647]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/30 04:07:48.068217 [ 7647]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 04:07:48.071283 [ 7647]: 00.ctdb: Set RecoverTimeout to 60 2014/10/30 04:07:48.074612 [ 7647]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/30 04:07:48.194681 [ 7647]: Freeze priority 1 2014/10/30 04:07:48.194779 [ 7647]: Freeze priority 2 2014/10/30 04:07:48.194867 [ 7647]: Freeze priority 3 2014/10/30 04:07:53.398416 [ 7647]: Freeze priority 1 2014/10/30 04:07:53.409735 [ 7647]: Freeze priority 2 2014/10/30 04:07:53.412494 [ 7647]: Freeze priority 3 2014/10/30 04:07:53.591233 [ 7647]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/30 04:07:53.592423 [ 7647]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/30 04:07:53.596675 [ 7647]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup 2014/10/30 04:07:55.940766 [ 7647]: Thawing priority 1 2014/10/30 04:07:55.940819 [ 7647]: Release freeze handler for prio 1 2014/10/30 04:07:55.940853 [ 7647]: Thawing priority 2 2014/10/30 04:07:55.940882 [ 7647]: Release freeze handler for prio 2 2014/10/30 04:07:55.940907 [ 7647]: Thawing priority 3 2014/10/30 04:07:55.940922 [ 7647]: Release freeze handler for prio 3 2014/10/30 04:08:09.960816 [recoverd: 7985]: Trigger takeoverrun 2014/10/30 04:08:10.439709 [ 7647]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/30 04:08:10.752987 [ 7647]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 04:08:10.763745 [ 7647]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 04:08:10.811286 [ 7647]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/30 04:08:10.974705 [ 7647]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/30 04:08:13.376196 [ 7647]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation 2014/10/30 04:08:14.896121 [ 7647]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 04:12:18.715733 [ 7647]: Freeze priority 1 2014/10/30 04:12:18.716787 [ 7647]: Freeze priority 2 2014/10/30 04:12:18.717632 [ 7647]: Freeze priority 3 2014/10/30 04:12:20.947401 [ 7647]: Thawing priority 1 2014/10/30 04:12:20.947445 [ 7647]: Release freeze handler for prio 1 2014/10/30 04:12:20.947475 [ 7647]: Thawing priority 2 2014/10/30 04:12:20.947491 [ 7647]: Release freeze handler for prio 2 2014/10/30 04:12:20.947527 [ 7647]: Thawing priority 3 2014/10/30 04:12:20.947543 [ 7647]: Release freeze handler for prio 3 2014/10/30 04:17:33.022404 [ 7647]: Freeze priority 1 2014/10/30 04:17:33.032998 [ 7647]: Freeze priority 2 2014/10/30 04:17:33.034769 [ 7647]: Freeze priority 3 2014/10/30 04:17:36.086530 [ 7647]: Thawing priority 1 2014/10/30 04:17:36.086598 [ 7647]: Release freeze handler for prio 1 2014/10/30 04:17:36.086635 [ 7647]: Thawing priority 2 2014/10/30 04:17:36.086655 [ 7647]: Release freeze handler for prio 2 2014/10/30 04:17:36.086684 [ 7647]: Thawing priority 3 2014/10/30 04:17:36.086714 [ 7647]: Release freeze handler for prio 3 2014/10/30 04:27:31.878797 [ 7647]: Freeze priority 1 2014/10/30 04:27:31.899677 [ 7647]: Freeze priority 1 2014/10/30 04:27:31.956293 [ 7647]: Freeze priority 2 2014/10/30 04:27:31.957087 [ 7647]: Freeze priority 2 2014/10/30 04:27:31.961223 [ 7647]: Freeze priority 3 2014/10/30 04:27:31.962039 [ 7647]: Freeze priority 3 2014/10/30 04:27:34.981999 [ 7647]: Freeze priority 1 2014/10/30 04:27:34.982392 [ 7647]: Freeze priority 2 2014/10/30 04:27:34.982681 [ 7647]: Freeze priority 3 2014/10/30 04:27:39.839753 [ 7647]: Thawing priority 1 2014/10/30 04:27:39.839820 [ 7647]: Release freeze handler for prio 1 2014/10/30 04:27:39.839862 [ 7647]: Thawing priority 2 2014/10/30 04:27:39.839900 [ 7647]: Release freeze handler for prio 2 2014/10/30 04:27:39.839939 [ 7647]: Thawing priority 3 2014/10/30 04:27:39.839971 [ 7647]: Release freeze handler for prio 3 2014/10/30 04:32:54.023580 [ 7647]: Freeze priority 1 2014/10/30 04:32:54.143548 [ 7647]: Freeze priority 2 2014/10/30 04:32:54.146889 [ 7647]: Freeze priority 3 2014/10/30 04:32:59.106296 [ 7647]: Thawing priority 1 2014/10/30 04:32:59.106355 [ 7647]: Release freeze handler for prio 1 2014/10/30 04:32:59.106401 [ 7647]: Thawing priority 2 2014/10/30 04:32:59.106426 [ 7647]: Release freeze handler for prio 2 2014/10/30 04:32:59.106460 [ 7647]: Thawing priority 3 2014/10/30 04:32:59.106484 [ 7647]: Release freeze handler for prio 3 2014/10/30 04:37:42.159685 [ 7647]: Freeze priority 1 2014/10/30 04:37:42.161784 [ 7647]: Freeze priority 1 2014/10/30 04:37:42.176155 [ 7647]: Freeze priority 2 2014/10/30 04:37:42.178375 [ 7647]: Freeze priority 2 2014/10/30 04:37:42.181139 [ 7647]: Freeze priority 3 2014/10/30 04:37:42.184336 [ 7647]: Freeze priority 3 2014/10/30 04:37:45.707207 [recoverd: 7985]: Taking out recovery lock from recovery daemon 2014/10/30 04:37:45.707300 [recoverd: 7985]: Take the recovery lock 2014/10/30 04:37:45.719799 [ 7647]: Freeze priority 1 2014/10/30 04:37:45.720139 [ 7647]: Freeze priority 2 2014/10/30 04:37:45.720435 [ 7647]: Freeze priority 3 2014/10/30 04:37:47.636054 [ 7647]: Thawing priority 1 2014/10/30 04:37:47.636101 [ 7647]: Release freeze handler for prio 1 2014/10/30 04:37:47.636135 [ 7647]: Thawing priority 2 2014/10/30 04:37:47.636164 [ 7647]: Release freeze handler for prio 2 2014/10/30 04:37:47.636189 [ 7647]: Thawing priority 3 2014/10/30 04:37:47.636215 [ 7647]: Release freeze handler for prio 3 2014/10/30 04:37:48.232306 [recoverd: 7985]: Resetting ban count to 0 for all nodes 2014/10/30 04:44:25.247222 [ 7647]: Freeze priority 1 2014/10/30 04:44:25.395776 [ 7647]: Freeze priority 2 2014/10/30 04:44:25.397074 [ 7647]: Freeze priority 3 2014/10/30 04:44:28.875769 [ 7647]: Thawing priority 1 2014/10/30 04:44:28.875838 [ 7647]: Release freeze handler for prio 1 2014/10/30 04:44:28.875891 [ 7647]: Thawing priority 2 2014/10/30 04:44:28.875912 [ 7647]: Release freeze handler for prio 2 2014/10/30 04:44:28.875942 [ 7647]: Thawing priority 3 2014/10/30 04:44:28.875961 [ 7647]: Release freeze handler for prio 3 2014/10/30 04:44:32.036422 [ 7647]: Freeze priority 1 2014/10/30 04:44:32.041913 [ 7647]: Freeze priority 2 2014/10/30 04:44:32.045477 [ 7647]: Freeze priority 3 2014/10/30 04:44:35.136655 [ 7647]: Thawing priority 1 2014/10/30 04:44:35.136724 [ 7647]: Release freeze handler for prio 1 2014/10/30 04:44:35.136772 [ 7647]: Thawing priority 2 2014/10/30 04:44:35.136791 [ 7647]: Release freeze handler for prio 2 2014/10/30 04:44:35.136818 [ 7647]: Thawing priority 3 2014/10/30 04:44:35.136835 [ 7647]: Release freeze handler for prio 3 2014/10/30 04:47:47.417472 [ 7647]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 04:47:47.428464 [ 7647]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 04:49:12.932961 [ 1531]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/30 04:49:12.933103 [ 1531]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 04:49:12.933125 [ 1531]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 04:53:13.680727 [ 7771]: Starting CTDBD (Version 2.5.3) as PID: 7771 2014/10/30 04:53:15.121045 [ 7771]: Vacuuming is disabled for persistent database registry.tdb 2014/10/30 04:53:15.145409 [ 7771]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/30 04:53:15.159569 [ 7771]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/30 04:53:15.173540 [ 7771]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/30 04:53:15.173559 [ 7771]: Ignoring persistent database 'account_policy.tdb.1' 2014/10/30 04:53:15.173567 [ 7771]: Ignoring persistent database 'ctdb.tdb.1' 2014/10/30 04:53:15.173576 [ 7771]: Ignoring persistent database 'group_mapping.tdb.1' 2014/10/30 04:53:15.173584 [ 7771]: Ignoring persistent database 'secrets.tdb.1' 2014/10/30 04:53:15.173592 [ 7771]: Ignoring persistent database 'share_info.tdb.1' 2014/10/30 04:53:15.187917 [ 7771]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/30 04:53:15.202114 [ 7771]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/30 04:53:15.202137 [ 7771]: Ignoring persistent database 'passdb.tdb.1' 2014/10/30 04:53:15.202146 [ 7771]: Ignoring persistent database 'registry.tdb.1' 2014/10/30 04:53:15.216102 [ 7771]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/30 04:53:15.216134 [ 7771]: Freeze priority 1 2014/10/30 04:53:15.232896 [ 7771]: Freeze priority 2 2014/10/30 04:53:15.233244 [ 7771]: Freeze priority 3 2014/10/30 04:53:15.397452 [ 7771]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/30 04:53:15.401512 [ 7771]: 00.ctdb: Set RecoverTimeout to 60 2014/10/30 04:53:15.405162 [ 7771]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/30 04:53:15.527117 [ 7771]: Freeze priority 1 2014/10/30 04:53:15.527191 [ 7771]: Freeze priority 2 2014/10/30 04:53:15.527245 [ 7771]: Freeze priority 3 2014/10/30 04:53:15.855657 [ 7771]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 04:53:15.857534 [ 7771]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 04:53:15.858167 [ 7771]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 04:53:15.858488 [ 7771]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 04:53:15.900048 [recoverd: 8056]: server/ctdb_recoverd.c:1058 Unable to find db_id 0x6afb8c09 on local node 2014/10/30 04:53:15.900116 [recoverd: 8056]: server/ctdb_recoverd.c:1058 Unable to find db_id 0xaf029e9d on local node 2014/10/30 04:53:15.900133 [recoverd: 8056]: server/ctdb_recoverd.c:1058 Unable to find db_id 0x4e66c2b2 on local node 2014/10/30 04:53:15.900146 [recoverd: 8056]: server/ctdb_recoverd.c:1058 Unable to find db_id 0x6afb8c09 on local node 2014/10/30 04:53:15.900158 [recoverd: 8056]: server/ctdb_recoverd.c:1058 Unable to find db_id 0xaf029e9d on local node 2014/10/30 04:53:15.900173 [recoverd: 8056]: server/ctdb_recoverd.c:1058 Unable to find db_id 0x4e66c2b2 on local node 2014/10/30 04:53:15.902425 [ 7771]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 04:53:19.534430 [recoverd: 8056]: server/ctdb_recoverd.c:3692 Current recmaster node 0 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/30 04:53:19.534487 [ 7771]: Freeze priority 1 2014/10/30 04:53:19.534549 [ 7771]: Freeze priority 2 2014/10/30 04:53:19.534602 [ 7771]: Freeze priority 3 2014/10/30 04:53:24.032569 [ 7771]: Freeze priority 1 2014/10/30 04:53:24.043522 [recoverd: 8056]: Trigger takeoverrun 2014/10/30 04:53:24.045707 [ 7771]: Freeze priority 2 2014/10/30 04:53:24.047282 [ 7771]: Freeze priority 3 2014/10/30 04:53:24.222051 [ 7771]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/30 04:53:24.224411 [ 7771]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/30 04:53:24.228830 [ 7771]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup 2014/10/30 04:53:26.626588 [ 7771]: Thawing priority 1 2014/10/30 04:53:26.626637 [ 7771]: Release freeze handler for prio 1 2014/10/30 04:53:26.626695 [ 7771]: Thawing priority 2 2014/10/30 04:53:26.626717 [ 7771]: Release freeze handler for prio 2 2014/10/30 04:53:26.626745 [ 7771]: Thawing priority 3 2014/10/30 04:53:26.626762 [ 7771]: Release freeze handler for prio 3 2014/10/30 04:53:28.633226 [recoverd: 8056]: Trigger takeoverrun 2014/10/30 04:53:28.785751 [ 7771]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/30 04:53:29.043945 [ 7771]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 04:53:29.057538 [ 7771]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 04:53:29.083971 [ 7771]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/30 04:53:29.272008 [ 7771]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/30 04:53:31.724700 [ 7771]: Node became HEALTHY. Ask recovery master 0 to perform ip reallocation 2014/10/30 04:53:37.567001 [ 7771]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 04:57:50.473555 [ 7771]: Freeze priority 1 2014/10/30 04:57:50.499675 [ 7771]: Freeze priority 2 2014/10/30 04:57:50.500750 [ 7771]: Freeze priority 3 2014/10/30 04:57:52.919220 [ 7771]: Thawing priority 1 2014/10/30 04:57:52.919265 [ 7771]: Release freeze handler for prio 1 2014/10/30 04:57:52.919304 [ 7771]: Thawing priority 2 2014/10/30 04:57:52.919327 [ 7771]: Release freeze handler for prio 2 2014/10/30 04:57:52.919372 [ 7771]: Thawing priority 3 2014/10/30 04:57:52.919390 [ 7771]: Release freeze handler for prio 3 2014/10/30 05:02:52.883034 [ 7771]: Freeze priority 1 2014/10/30 05:02:52.987170 [ 7771]: Freeze priority 2 2014/10/30 05:02:52.988812 [ 7771]: Freeze priority 3 2014/10/30 05:02:58.771496 [ 7771]: Thawing priority 1 2014/10/30 05:02:58.771570 [ 7771]: Release freeze handler for prio 1 2014/10/30 05:02:58.771608 [ 7771]: Thawing priority 2 2014/10/30 05:02:58.771635 [ 7771]: Release freeze handler for prio 2 2014/10/30 05:02:58.771684 [ 7771]: Thawing priority 3 2014/10/30 05:02:58.771703 [ 7771]: Release freeze handler for prio 3 2014/10/30 05:03:09.183111 [ 7771]: Freeze priority 1 2014/10/30 05:03:09.185291 [ 7771]: Freeze priority 2 2014/10/30 05:03:09.187527 [ 7771]: Freeze priority 3 2014/10/30 05:03:11.867896 [ 7771]: Thawing priority 1 2014/10/30 05:03:11.867950 [ 7771]: Release freeze handler for prio 1 2014/10/30 05:03:11.867988 [ 7771]: Thawing priority 2 2014/10/30 05:03:11.868008 [ 7771]: Release freeze handler for prio 2 2014/10/30 05:03:11.868038 [ 7771]: Thawing priority 3 2014/10/30 05:03:11.868056 [ 7771]: Release freeze handler for prio 3 2014/10/30 05:12:59.002841 [ 7771]: Freeze priority 1 2014/10/30 05:12:59.003116 [ 7771]: Freeze priority 1 2014/10/30 05:12:59.048410 [ 7771]: Freeze priority 1 2014/10/30 05:12:59.179316 [ 7771]: Freeze priority 2 2014/10/30 05:12:59.179524 [ 7771]: Freeze priority 2 2014/10/30 05:12:59.180441 [ 7771]: Freeze priority 2 2014/10/30 05:12:59.180683 [ 7771]: Freeze priority 3 2014/10/30 05:12:59.181563 [ 7771]: Freeze priority 3 2014/10/30 05:12:59.182681 [ 7771]: Freeze priority 3 2014/10/30 05:13:02.245321 [ 7771]: Freeze priority 1 2014/10/30 05:13:02.245697 [ 7771]: Freeze priority 2 2014/10/30 05:13:02.245992 [ 7771]: Freeze priority 3 2014/10/30 05:13:07.586542 [ 7771]: Thawing priority 1 2014/10/30 05:13:07.586607 [ 7771]: Release freeze handler for prio 1 2014/10/30 05:13:07.586643 [ 7771]: Thawing priority 2 2014/10/30 05:13:07.586676 [ 7771]: Release freeze handler for prio 2 2014/10/30 05:13:07.586705 [ 7771]: Thawing priority 3 2014/10/30 05:13:07.586736 [ 7771]: Release freeze handler for prio 3 2014/10/30 05:18:21.744705 [ 7771]: Freeze priority 1 2014/10/30 05:18:21.858673 [ 7771]: Freeze priority 2 2014/10/30 05:18:21.862773 [ 7771]: Freeze priority 3 2014/10/30 05:18:28.282940 [ 7771]: Thawing priority 1 2014/10/30 05:18:28.283003 [ 7771]: Release freeze handler for prio 1 2014/10/30 05:18:28.283040 [ 7771]: Thawing priority 2 2014/10/30 05:18:28.283060 [ 7771]: Release freeze handler for prio 2 2014/10/30 05:18:28.283089 [ 7771]: Thawing priority 3 2014/10/30 05:18:28.283108 [ 7771]: Release freeze handler for prio 3 2014/10/30 05:23:10.398652 [ 7771]: Freeze priority 1 2014/10/30 05:23:10.417348 [ 7771]: Freeze priority 1 2014/10/30 05:23:10.419947 [ 7771]: Freeze priority 2 2014/10/30 05:23:10.420109 [ 7771]: Freeze priority 2 2014/10/30 05:23:10.421070 [ 7771]: Freeze priority 3 2014/10/30 05:23:10.421293 [ 7771]: Freeze priority 3 2014/10/30 05:23:13.933107 [recoverd: 8056]: Taking out recovery lock from recovery daemon 2014/10/30 05:23:13.933178 [recoverd: 8056]: Take the recovery lock 2014/10/30 05:23:13.945108 [ 7771]: Freeze priority 1 2014/10/30 05:23:13.945478 [ 7771]: Freeze priority 2 2014/10/30 05:23:13.946153 [ 7771]: Freeze priority 3 2014/10/30 05:23:15.813040 [ 7771]: Thawing priority 1 2014/10/30 05:23:15.813085 [ 7771]: Release freeze handler for prio 1 2014/10/30 05:23:15.813131 [ 7771]: Thawing priority 2 2014/10/30 05:23:15.813150 [ 7771]: Release freeze handler for prio 2 2014/10/30 05:23:15.813177 [ 7771]: Thawing priority 3 2014/10/30 05:23:15.813194 [ 7771]: Release freeze handler for prio 3 2014/10/30 05:23:16.368044 [recoverd: 8056]: Resetting ban count to 0 for all nodes 2014/10/30 05:29:57.927301 [recoverd: 8056]: Taking out recovery lock from recovery daemon 2014/10/30 05:29:57.927377 [recoverd: 8056]: Take the recovery lock 2014/10/30 05:29:58.056676 [ 7771]: Freeze priority 1 2014/10/30 05:29:58.072482 [ 7771]: Freeze priority 2 2014/10/30 05:29:58.076227 [ 7771]: Freeze priority 3 2014/10/30 05:30:00.671686 [ 7771]: Thawing priority 1 2014/10/30 05:30:00.671727 [ 7771]: Release freeze handler for prio 1 2014/10/30 05:30:00.671758 [ 7771]: Thawing priority 2 2014/10/30 05:30:00.671775 [ 7771]: Release freeze handler for prio 2 2014/10/30 05:30:00.671810 [ 7771]: Thawing priority 3 2014/10/30 05:30:00.671829 [ 7771]: Release freeze handler for prio 3 2014/10/30 05:30:01.029404 [recoverd: 8056]: Resetting ban count to 0 for all nodes 2014/10/30 05:33:11.044634 [ 7771]: common/ctdb_fork.c:131 waitpid() returned error. errno:10 2014/10/30 05:33:11.128218 [ 7771]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 05:33:11.139454 [ 7771]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 05:34:36.323395 [ 1540]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/30 05:34:36.323539 [ 1540]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 05:34:36.323563 [ 1540]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 05:38:38.353089 [ 7506]: Starting CTDBD (Version 2.5.3) as PID: 7506 2014/10/30 05:38:40.043395 [ 7506]: Vacuuming is disabled for persistent database registry.tdb 2014/10/30 05:38:40.067059 [ 7506]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/30 05:38:40.081267 [ 7506]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/30 05:38:40.095497 [ 7506]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/30 05:38:40.095517 [ 7506]: Ignoring persistent database 'account_policy.tdb.1' 2014/10/30 05:38:40.095527 [ 7506]: Ignoring persistent database 'ctdb.tdb.1' 2014/10/30 05:38:40.095536 [ 7506]: Ignoring persistent database 'group_mapping.tdb.1' 2014/10/30 05:38:40.095545 [ 7506]: Ignoring persistent database 'secrets.tdb.1' 2014/10/30 05:38:40.095555 [ 7506]: Ignoring persistent database 'share_info.tdb.1' 2014/10/30 05:38:40.109613 [ 7506]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/30 05:38:40.123674 [ 7506]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/30 05:38:40.123693 [ 7506]: Ignoring persistent database 'passdb.tdb.1' 2014/10/30 05:38:40.123704 [ 7506]: Ignoring persistent database 'registry.tdb.1' 2014/10/30 05:38:40.137713 [ 7506]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/30 05:38:40.137745 [ 7506]: Freeze priority 1 2014/10/30 05:38:40.155074 [ 7506]: Freeze priority 2 2014/10/30 05:38:40.155412 [ 7506]: Freeze priority 3 2014/10/30 05:38:40.343792 [ 7506]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 05:38:40.343933 [ 7506]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 05:38:40.343956 [ 7506]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 05:38:40.344169 [ 7506]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 05:38:40.344972 [ 7506]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/30 05:38:40.348992 [ 7506]: 00.ctdb: Set RecoverTimeout to 60 2014/10/30 05:38:40.352577 [ 7506]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/30 05:38:40.490609 [ 7506]: Freeze priority 1 2014/10/30 05:38:40.490695 [ 7506]: Freeze priority 2 2014/10/30 05:38:40.490754 [ 7506]: Freeze priority 3 2014/10/30 05:38:44.197140 [ 7506]: Freeze priority 1 2014/10/30 05:38:44.321422 [ 7506]: Freeze priority 2 2014/10/30 05:38:44.322816 [ 7506]: Freeze priority 3 2014/10/30 05:38:44.497289 [recoverd: 7843]: server/ctdb_recoverd.c:3692 Current recmaster node 3 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/30 05:38:44.497376 [ 7506]: Freeze priority 1 2014/10/30 05:38:44.497436 [ 7506]: Freeze priority 2 2014/10/30 05:38:44.497488 [ 7506]: Freeze priority 3 2014/10/30 05:38:48.460242 [ 7506]: Thawing priority 1 2014/10/30 05:38:48.460291 [ 7506]: Release freeze handler for prio 1 2014/10/30 05:38:48.460324 [ 7506]: Thawing priority 2 2014/10/30 05:38:48.460341 [ 7506]: Release freeze handler for prio 2 2014/10/30 05:38:48.460377 [ 7506]: Thawing priority 3 2014/10/30 05:38:48.460393 [ 7506]: Release freeze handler for prio 3 2014/10/30 05:38:49.505487 [recoverd: 7843]: Trigger takeoverrun 2014/10/30 05:38:59.033521 [ 7506]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/30 05:38:59.086695 [ 7506]: Freeze priority 1 2014/10/30 05:38:59.090976 [ 7506]: Freeze priority 2 2014/10/30 05:38:59.093362 [ 7506]: Freeze priority 3 2014/10/30 05:38:59.269732 [ 7506]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/30 05:38:59.273421 [ 7506]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup 2014/10/30 05:39:01.508278 [ 7506]: Thawing priority 1 2014/10/30 05:39:01.508376 [ 7506]: Release freeze handler for prio 1 2014/10/30 05:39:01.508435 [ 7506]: Thawing priority 2 2014/10/30 05:39:01.508455 [ 7506]: Release freeze handler for prio 2 2014/10/30 05:39:01.508493 [ 7506]: Thawing priority 3 2014/10/30 05:39:01.508512 [ 7506]: Release freeze handler for prio 3 2014/10/30 05:39:15.545467 [recoverd: 7843]: Trigger takeoverrun 2014/10/30 05:39:15.880479 [ 7506]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/30 05:39:16.236680 [ 7506]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 05:39:16.248322 [ 7506]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 05:39:16.271276 [ 7506]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/30 05:39:16.485987 [ 7506]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/30 05:39:18.872663 [ 7506]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation 2014/10/30 05:39:19.407094 [ 7506]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 05:43:13.855039 [ 7506]: Freeze priority 1 2014/10/30 05:43:13.867353 [ 7506]: Freeze priority 1 2014/10/30 05:43:13.873364 [ 7506]: Freeze priority 1 2014/10/30 05:43:13.906402 [ 7506]: Freeze priority 2 2014/10/30 05:43:13.906604 [ 7506]: Freeze priority 2 2014/10/30 05:43:13.906673 [ 7506]: Freeze priority 2 2014/10/30 05:43:13.907455 [ 7506]: Freeze priority 3 2014/10/30 05:43:13.908298 [ 7506]: Freeze priority 3 2014/10/30 05:43:13.908371 [ 7506]: Freeze priority 3 2014/10/30 05:43:16.927830 [ 7506]: Freeze priority 1 2014/10/30 05:43:16.928260 [ 7506]: Freeze priority 2 2014/10/30 05:43:16.928552 [ 7506]: Freeze priority 3 2014/10/30 05:43:21.547204 [ 7506]: Thawing priority 1 2014/10/30 05:43:21.547277 [ 7506]: Release freeze handler for prio 1 2014/10/30 05:43:21.547324 [ 7506]: Thawing priority 2 2014/10/30 05:43:21.547340 [ 7506]: Release freeze handler for prio 2 2014/10/30 05:43:21.547377 [ 7506]: Thawing priority 3 2014/10/30 05:43:21.547392 [ 7506]: Release freeze handler for prio 3 2014/10/30 05:48:21.663997 [ 7506]: Freeze priority 1 2014/10/30 05:48:21.685963 [ 7506]: Freeze priority 2 2014/10/30 05:48:21.690421 [ 7506]: Freeze priority 3 2014/10/30 05:48:24.857674 [ 7506]: Thawing priority 1 2014/10/30 05:48:24.857751 [ 7506]: Release freeze handler for prio 1 2014/10/30 05:48:24.857789 [ 7506]: Thawing priority 2 2014/10/30 05:48:24.857836 [ 7506]: Release freeze handler for prio 2 2014/10/30 05:48:24.857874 [ 7506]: Thawing priority 3 2014/10/30 05:48:24.857894 [ 7506]: Release freeze handler for prio 3 2014/10/30 05:58:21.993530 [ 7506]: Freeze priority 1 2014/10/30 05:58:21.995952 [ 7506]: Freeze priority 1 2014/10/30 05:58:21.997938 [ 7506]: Freeze priority 1 2014/10/30 05:58:22.007652 [ 7506]: Freeze priority 2 2014/10/30 05:58:22.008899 [ 7506]: Freeze priority 3 2014/10/30 05:58:22.009747 [ 7506]: Freeze priority 2 2014/10/30 05:58:22.010155 [ 7506]: Freeze priority 2 2014/10/30 05:58:22.012996 [ 7506]: Freeze priority 3 2014/10/30 05:58:22.013409 [ 7506]: Freeze priority 3 2014/10/30 05:58:25.038498 [ 7506]: Freeze priority 1 2014/10/30 05:58:25.038829 [ 7506]: Freeze priority 2 2014/10/30 05:58:25.039155 [ 7506]: Freeze priority 3 2014/10/30 05:58:28.618570 [ 7506]: Thawing priority 1 2014/10/30 05:58:28.618687 [ 7506]: Release freeze handler for prio 1 2014/10/30 05:58:28.618747 [ 7506]: Thawing priority 2 2014/10/30 05:58:28.618770 [ 7506]: Release freeze handler for prio 2 2014/10/30 05:58:28.618813 [ 7506]: Thawing priority 3 2014/10/30 05:58:28.618844 [ 7506]: Release freeze handler for prio 3 2014/10/30 06:03:44.900153 [ 7506]: Freeze priority 1 2014/10/30 06:03:44.968881 [ 7506]: Freeze priority 2 2014/10/30 06:03:44.973331 [ 7506]: Freeze priority 3 2014/10/30 06:03:49.812350 [ 7506]: Thawing priority 1 2014/10/30 06:03:49.812417 [ 7506]: Release freeze handler for prio 1 2014/10/30 06:03:49.812453 [ 7506]: Thawing priority 2 2014/10/30 06:03:49.812473 [ 7506]: Release freeze handler for prio 2 2014/10/30 06:03:49.812501 [ 7506]: Thawing priority 3 2014/10/30 06:03:49.812518 [ 7506]: Release freeze handler for prio 3 2014/10/30 06:08:29.886867 [ 7506]: Freeze priority 1 2014/10/30 06:08:29.891091 [ 7506]: Freeze priority 1 2014/10/30 06:08:29.908299 [ 7506]: Freeze priority 2 2014/10/30 06:08:29.909390 [ 7506]: Freeze priority 2 2014/10/30 06:08:29.912770 [ 7506]: Freeze priority 3 2014/10/30 06:08:29.915378 [ 7506]: Freeze priority 3 2014/10/30 06:08:33.430903 [recoverd: 7843]: Taking out recovery lock from recovery daemon 2014/10/30 06:08:33.430978 [recoverd: 7843]: Take the recovery lock 2014/10/30 06:08:33.442472 [ 7506]: Freeze priority 1 2014/10/30 06:08:33.442815 [ 7506]: Freeze priority 2 2014/10/30 06:08:33.443101 [ 7506]: Freeze priority 3 2014/10/30 06:08:35.251880 [ 7506]: Thawing priority 1 2014/10/30 06:08:35.251947 [ 7506]: Release freeze handler for prio 1 2014/10/30 06:08:35.251996 [ 7506]: Thawing priority 2 2014/10/30 06:08:35.252025 [ 7506]: Release freeze handler for prio 2 2014/10/30 06:08:35.252054 [ 7506]: Thawing priority 3 2014/10/30 06:08:35.252070 [ 7506]: Release freeze handler for prio 3 2014/10/30 06:08:35.859673 [recoverd: 7843]: Resetting ban count to 0 for all nodes 2014/10/30 06:15:12.431357 [recoverd: 7843]: Taking out recovery lock from recovery daemon 2014/10/30 06:15:12.431415 [recoverd: 7843]: Take the recovery lock 2014/10/30 06:15:12.569408 [ 7506]: Freeze priority 1 2014/10/30 06:15:12.579404 [ 7506]: Freeze priority 2 2014/10/30 06:15:12.583450 [ 7506]: Freeze priority 3 2014/10/30 06:15:15.440627 [ 7506]: Thawing priority 1 2014/10/30 06:15:15.440687 [ 7506]: Release freeze handler for prio 1 2014/10/30 06:15:15.440727 [ 7506]: Thawing priority 2 2014/10/30 06:15:15.440748 [ 7506]: Release freeze handler for prio 2 2014/10/30 06:15:15.440779 [ 7506]: Thawing priority 3 2014/10/30 06:15:15.440798 [ 7506]: Release freeze handler for prio 3 2014/10/30 06:15:15.816301 [recoverd: 7843]: Resetting ban count to 0 for all nodes 2014/10/30 06:15:26.059699 [recoverd: 7843]: server/ctdb_recoverd.c:3933 Remote node:1 has different flags for node 0. It has 0x02 vs our 0x00 2014/10/30 06:15:26.059755 [recoverd: 7843]: Use flags 0x00 from local recmaster node for cluster update of node 0 flags 2014/10/30 06:15:26.061228 [recoverd: 7843]: Taking out recovery lock from recovery daemon 2014/10/30 06:15:26.061262 [recoverd: 7843]: Take the recovery lock 2014/10/30 06:15:26.120270 [ 7506]: Freeze priority 1 2014/10/30 06:15:26.123676 [ 7506]: Freeze priority 2 2014/10/30 06:15:26.126432 [ 7506]: Freeze priority 3 2014/10/30 06:15:29.033628 [ 7506]: Thawing priority 1 2014/10/30 06:15:29.033682 [ 7506]: Release freeze handler for prio 1 2014/10/30 06:15:29.033724 [ 7506]: Thawing priority 2 2014/10/30 06:15:29.033760 [ 7506]: Release freeze handler for prio 2 2014/10/30 06:15:29.033790 [ 7506]: Thawing priority 3 2014/10/30 06:15:29.033809 [ 7506]: Release freeze handler for prio 3 2014/10/30 06:15:29.431851 [recoverd: 7843]: Resetting ban count to 0 for all nodes 2014/10/30 06:18:33.436332 [ 7506]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 06:18:33.447338 [ 7506]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 06:20:00.686246 [ 1524]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/30 06:20:00.686345 [ 1524]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 06:20:00.686359 [ 1524]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 06:24:01.760413 [ 7740]: Starting CTDBD (Version 2.5.3) as PID: 7740 2014/10/30 06:24:03.209028 [ 7740]: Vacuuming is disabled for persistent database registry.tdb 2014/10/30 06:24:03.232597 [ 7740]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/30 06:24:03.246818 [ 7740]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/30 06:24:03.261033 [ 7740]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/30 06:24:03.261051 [ 7740]: Ignoring persistent database 'account_policy.tdb.1' 2014/10/30 06:24:03.261060 [ 7740]: Ignoring persistent database 'ctdb.tdb.1' 2014/10/30 06:24:03.261068 [ 7740]: Ignoring persistent database 'group_mapping.tdb.1' 2014/10/30 06:24:03.261077 [ 7740]: Ignoring persistent database 'secrets.tdb.1' 2014/10/30 06:24:03.261086 [ 7740]: Ignoring persistent database 'share_info.tdb.1' 2014/10/30 06:24:03.275028 [ 7740]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/30 06:24:03.289051 [ 7740]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/30 06:24:03.289070 [ 7740]: Ignoring persistent database 'passdb.tdb.1' 2014/10/30 06:24:03.289079 [ 7740]: Ignoring persistent database 'registry.tdb.1' 2014/10/30 06:24:03.302991 [ 7740]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/30 06:24:03.303023 [ 7740]: Freeze priority 1 2014/10/30 06:24:03.314474 [ 7740]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 06:24:03.321069 [ 7740]: Freeze priority 2 2014/10/30 06:24:03.321419 [ 7740]: Freeze priority 3 2014/10/30 06:24:03.485305 [ 7740]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/30 06:24:03.489111 [ 7740]: 00.ctdb: Set RecoverTimeout to 60 2014/10/30 06:24:03.492929 [ 7740]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/30 06:24:03.612759 [ 7740]: Freeze priority 1 2014/10/30 06:24:03.612840 [ 7740]: Freeze priority 2 2014/10/30 06:24:03.612898 [ 7740]: Freeze priority 3 2014/10/30 06:24:04.146435 [ 7740]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 06:24:08.817452 [ 7740]: Freeze priority 1 2014/10/30 06:24:08.832580 [ 7740]: Freeze priority 2 2014/10/30 06:24:08.834854 [ 7740]: Freeze priority 3 2014/10/30 06:24:09.012037 [ 7740]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/30 06:24:09.016217 [ 7740]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/30 06:24:09.019699 [ 7740]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup 2014/10/30 06:24:11.228067 [ 7740]: Thawing priority 1 2014/10/30 06:24:11.228098 [ 7740]: Release freeze handler for prio 1 2014/10/30 06:24:11.228127 [ 7740]: Thawing priority 2 2014/10/30 06:24:11.228156 [ 7740]: Release freeze handler for prio 2 2014/10/30 06:24:11.228179 [ 7740]: Thawing priority 3 2014/10/30 06:24:11.228206 [ 7740]: Release freeze handler for prio 3 2014/10/30 06:24:26.242720 [recoverd: 8014]: Trigger takeoverrun 2014/10/30 06:24:26.449108 [ 7740]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/30 06:24:26.715391 [ 7740]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 06:24:26.728906 [ 7740]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 06:24:26.765692 [ 7740]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/30 06:24:26.961194 [ 7740]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/30 06:24:29.417969 [ 7740]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation 2014/10/30 06:24:30.179131 [ 7740]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 06:28:37.580154 [ 7740]: Freeze priority 1 2014/10/30 06:28:37.598806 [ 7740]: Freeze priority 1 2014/10/30 06:28:37.613160 [ 7740]: Freeze priority 1 2014/10/30 06:28:37.647510 [ 7740]: Freeze priority 2 2014/10/30 06:28:37.647737 [ 7740]: Freeze priority 2 2014/10/30 06:28:37.648610 [ 7740]: Freeze priority 2 2014/10/30 06:28:37.649073 [ 7740]: Freeze priority 3 2014/10/30 06:28:37.650173 [ 7740]: Freeze priority 3 2014/10/30 06:28:37.651324 [ 7740]: Freeze priority 3 2014/10/30 06:28:40.688247 [ 7740]: Freeze priority 1 2014/10/30 06:28:40.688620 [ 7740]: Freeze priority 2 2014/10/30 06:28:40.688998 [ 7740]: Freeze priority 3 2014/10/30 06:28:44.228453 [ 7740]: Thawing priority 1 2014/10/30 06:28:44.228515 [ 7740]: Release freeze handler for prio 1 2014/10/30 06:28:44.228552 [ 7740]: Thawing priority 2 2014/10/30 06:28:44.228585 [ 7740]: Release freeze handler for prio 2 2014/10/30 06:28:44.228620 [ 7740]: Thawing priority 3 2014/10/30 06:28:44.228653 [ 7740]: Release freeze handler for prio 3 2014/10/30 06:33:45.372666 [ 7740]: Freeze priority 1 2014/10/30 06:33:45.473092 [ 7740]: Freeze priority 2 2014/10/30 06:33:45.477459 [ 7740]: Freeze priority 3 2014/10/30 06:33:49.677700 [ 7740]: Thawing priority 1 2014/10/30 06:33:49.677765 [ 7740]: Release freeze handler for prio 1 2014/10/30 06:33:49.677801 [ 7740]: Thawing priority 2 2014/10/30 06:33:49.677822 [ 7740]: Release freeze handler for prio 2 2014/10/30 06:33:49.677884 [ 7740]: Thawing priority 3 2014/10/30 06:33:49.677907 [ 7740]: Release freeze handler for prio 3 2014/10/30 06:34:00.093115 [ 7740]: Freeze priority 1 2014/10/30 06:34:00.121706 [ 7740]: Freeze priority 2 2014/10/30 06:34:00.124975 [ 7740]: Freeze priority 3 2014/10/30 06:34:04.124920 [ 7740]: Thawing priority 1 2014/10/30 06:34:04.124966 [ 7740]: Release freeze handler for prio 1 2014/10/30 06:34:04.124998 [ 7740]: Thawing priority 2 2014/10/30 06:34:04.125018 [ 7740]: Release freeze handler for prio 2 2014/10/30 06:34:04.125049 [ 7740]: Thawing priority 3 2014/10/30 06:34:04.125067 [ 7740]: Release freeze handler for prio 3 2014/10/30 06:43:49.971404 [ 7740]: Freeze priority 1 2014/10/30 06:43:50.083436 [ 7740]: Freeze priority 2 2014/10/30 06:43:50.084581 [ 7740]: Freeze priority 3 2014/10/30 06:43:53.931395 [ 7740]: Freeze priority 1 2014/10/30 06:43:53.931793 [ 7740]: Freeze priority 2 2014/10/30 06:43:53.932142 [ 7740]: Freeze priority 3 2014/10/30 06:43:57.699025 [ 7740]: Thawing priority 1 2014/10/30 06:43:57.699066 [ 7740]: Release freeze handler for prio 1 2014/10/30 06:43:57.699099 [ 7740]: Thawing priority 2 2014/10/30 06:43:57.699120 [ 7740]: Release freeze handler for prio 2 2014/10/30 06:43:57.699148 [ 7740]: Thawing priority 3 2014/10/30 06:43:57.699167 [ 7740]: Release freeze handler for prio 3 2014/10/30 06:49:07.908515 [ 7740]: Freeze priority 1 2014/10/30 06:49:07.930351 [ 7740]: Freeze priority 2 2014/10/30 06:49:07.933707 [ 7740]: Freeze priority 3 2014/10/30 06:49:13.989635 [ 7740]: Thawing priority 1 2014/10/30 06:49:13.989686 [ 7740]: Release freeze handler for prio 1 2014/10/30 06:49:13.989736 [ 7740]: Thawing priority 2 2014/10/30 06:49:13.989752 [ 7740]: Release freeze handler for prio 2 2014/10/30 06:49:13.989778 [ 7740]: Thawing priority 3 2014/10/30 06:49:13.989794 [ 7740]: Release freeze handler for prio 3 2014/10/30 06:49:24.434469 [ 7740]: Freeze priority 1 2014/10/30 06:49:24.622295 [ 7740]: Freeze priority 2 2014/10/30 06:49:24.624051 [ 7740]: Freeze priority 3 2014/10/30 06:49:28.572269 [ 7740]: Thawing priority 1 2014/10/30 06:49:28.572313 [ 7740]: Release freeze handler for prio 1 2014/10/30 06:49:28.572341 [ 7740]: Thawing priority 2 2014/10/30 06:49:28.572369 [ 7740]: Release freeze handler for prio 2 2014/10/30 06:49:28.572393 [ 7740]: Thawing priority 3 2014/10/30 06:49:28.572408 [ 7740]: Release freeze handler for prio 3 2014/10/30 06:53:59.942698 [ 7740]: Freeze priority 1 2014/10/30 06:53:59.949313 [ 7740]: Freeze priority 1 2014/10/30 06:53:59.958253 [ 7740]: Freeze priority 1 2014/10/30 06:53:59.965143 [ 7740]: Freeze priority 2 2014/10/30 06:53:59.965362 [ 7740]: Freeze priority 2 2014/10/30 06:53:59.965405 [ 7740]: Freeze priority 2 2014/10/30 06:53:59.966903 [ 7740]: Freeze priority 3 2014/10/30 06:53:59.967093 [ 7740]: Freeze priority 3 2014/10/30 06:53:59.968248 [ 7740]: Freeze priority 3 2014/10/30 06:54:03.477701 [recoverd: 8014]: Taking out recovery lock from recovery daemon 2014/10/30 06:54:03.477746 [recoverd: 8014]: Take the recovery lock 2014/10/30 06:54:03.490690 [ 7740]: Freeze priority 1 2014/10/30 06:54:03.491048 [ 7740]: Freeze priority 2 2014/10/30 06:54:03.491368 [ 7740]: Freeze priority 3 2014/10/30 06:54:05.169504 [ 7740]: Thawing priority 1 2014/10/30 06:54:05.169559 [ 7740]: Release freeze handler for prio 1 2014/10/30 06:54:05.169610 [ 7740]: Thawing priority 2 2014/10/30 06:54:05.169632 [ 7740]: Release freeze handler for prio 2 2014/10/30 06:54:05.169676 [ 7740]: Thawing priority 3 2014/10/30 06:54:05.169699 [ 7740]: Release freeze handler for prio 3 2014/10/30 06:54:05.701963 [recoverd: 8014]: Resetting ban count to 0 for all nodes 2014/10/30 07:00:49.252909 [recoverd: 8014]: Taking out recovery lock from recovery daemon 2014/10/30 07:00:49.252971 [recoverd: 8014]: Take the recovery lock 2014/10/30 07:00:49.310761 [ 7740]: Freeze priority 1 2014/10/30 07:00:49.327202 [ 7740]: Freeze priority 2 2014/10/30 07:00:49.330718 [ 7740]: Freeze priority 3 2014/10/30 07:00:51.833262 [ 7740]: Thawing priority 1 2014/10/30 07:00:51.833314 [ 7740]: Release freeze handler for prio 1 2014/10/30 07:00:51.833349 [ 7740]: Thawing priority 2 2014/10/30 07:00:51.833370 [ 7740]: Release freeze handler for prio 2 2014/10/30 07:00:51.833401 [ 7740]: Thawing priority 3 2014/10/30 07:00:51.833420 [ 7740]: Release freeze handler for prio 3 2014/10/30 07:00:52.205210 [recoverd: 8014]: Resetting ban count to 0 for all nodes 2014/10/30 07:04:02.298016 [ 7740]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 07:04:02.308978 [ 7740]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 07:05:26.786064 [ 1520]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/30 07:05:26.786168 [ 1520]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 07:05:26.786186 [ 1520]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 07:09:29.236277 [ 7610]: Starting CTDBD (Version 2.5.3) as PID: 7610 2014/10/30 07:09:30.841650 [ 7610]: Vacuuming is disabled for persistent database registry.tdb 2014/10/30 07:09:30.865220 [ 7610]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/30 07:09:30.879845 [ 7610]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/30 07:09:30.893995 [ 7610]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/30 07:09:30.894015 [ 7610]: Ignoring persistent database 'account_policy.tdb.1' 2014/10/30 07:09:30.894025 [ 7610]: Ignoring persistent database 'ctdb.tdb.1' 2014/10/30 07:09:30.894034 [ 7610]: Ignoring persistent database 'group_mapping.tdb.1' 2014/10/30 07:09:30.894043 [ 7610]: Ignoring persistent database 'secrets.tdb.1' 2014/10/30 07:09:30.894052 [ 7610]: Ignoring persistent database 'share_info.tdb.1' 2014/10/30 07:09:30.908049 [ 7610]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/30 07:09:30.921990 [ 7610]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/30 07:09:30.922011 [ 7610]: Ignoring persistent database 'passdb.tdb.1' 2014/10/30 07:09:30.922021 [ 7610]: Ignoring persistent database 'registry.tdb.1' 2014/10/30 07:09:30.936065 [ 7610]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/30 07:09:30.936115 [ 7610]: Freeze priority 1 2014/10/30 07:09:30.949730 [ 7610]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 07:09:30.953431 [ 7610]: Freeze priority 2 2014/10/30 07:09:30.953839 [ 7610]: Freeze priority 3 2014/10/30 07:09:31.117913 [ 7610]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/30 07:09:31.121654 [ 7610]: 00.ctdb: Set RecoverTimeout to 60 2014/10/30 07:09:31.125068 [ 7610]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/30 07:09:31.243666 [ 7610]: Freeze priority 1 2014/10/30 07:09:31.243741 [ 7610]: Freeze priority 2 2014/10/30 07:09:31.243818 [ 7610]: Freeze priority 3 2014/10/30 07:09:31.378081 [ 7610]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 07:09:31.378142 [ 7610]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 07:09:31.773661 [ 7610]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 07:09:35.218080 [ 7610]: Freeze priority 1 2014/10/30 07:09:35.251389 [recoverd: 7904]: server/ctdb_recoverd.c:3692 Current recmaster node 1 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/30 07:09:35.251462 [ 7610]: Freeze priority 1 2014/10/30 07:09:35.251518 [ 7610]: Freeze priority 2 2014/10/30 07:09:35.251569 [ 7610]: Freeze priority 3 2014/10/30 07:09:35.287411 [ 7610]: Freeze priority 2 2014/10/30 07:09:35.294649 [ 7610]: Freeze priority 3 2014/10/30 07:09:42.004406 [ 7610]: Freeze priority 1 2014/10/30 07:09:42.007031 [ 7610]: Thawing priority 1 2014/10/30 07:09:42.007065 [ 7610]: Release freeze handler for prio 1 2014/10/30 07:09:42.007106 [ 7610]: Thawing priority 2 2014/10/30 07:09:42.007125 [ 7610]: Release freeze handler for prio 2 2014/10/30 07:09:42.007153 [ 7610]: Thawing priority 3 2014/10/30 07:09:42.007170 [ 7610]: Release freeze handler for prio 3 2014/10/30 07:09:42.009966 [ 7610]: Freeze priority 2 2014/10/30 07:09:42.011084 [ 7610]: Freeze priority 3 2014/10/30 07:09:42.014010 [set_recmode: 9303]: ERROR: recovery lock file /mnt/lock/lockfile not locked when recovering! 2014/10/30 07:09:45.030081 [ 7610]: Freeze priority 1 2014/10/30 07:09:45.031157 [ 7610]: Freeze priority 2 2014/10/30 07:09:45.031481 [ 7610]: Freeze priority 3 2014/10/30 07:09:45.190508 [ 7610]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/30 07:09:45.191365 [ 7610]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/30 07:09:45.192560 [ 7610]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup 2014/10/30 07:09:47.202078 [ 7610]: Thawing priority 1 2014/10/30 07:09:47.202133 [ 7610]: Release freeze handler for prio 1 2014/10/30 07:09:47.202168 [ 7610]: Thawing priority 2 2014/10/30 07:09:47.202189 [ 7610]: Release freeze handler for prio 2 2014/10/30 07:09:47.202232 [ 7610]: Thawing priority 3 2014/10/30 07:09:47.202251 [ 7610]: Release freeze handler for prio 3 2014/10/30 07:10:01.227523 [recoverd: 7904]: Trigger takeoverrun 2014/10/30 07:10:01.683618 [ 7610]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/30 07:10:02.248791 [ 7610]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 07:10:02.262318 [ 7610]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 07:10:02.290866 [ 7610]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/30 07:10:02.477620 [ 7610]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/30 07:10:04.943567 [ 7610]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation 2014/10/30 07:10:06.094236 [ 7610]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 07:14:06.609719 [ 7610]: Freeze priority 1 2014/10/30 07:14:06.609934 [ 7610]: Freeze priority 1 2014/10/30 07:14:06.610059 [ 7610]: Freeze priority 1 2014/10/30 07:14:06.779425 [ 7610]: Freeze priority 2 2014/10/30 07:14:06.779641 [ 7610]: Freeze priority 2 2014/10/30 07:14:06.779689 [ 7610]: Freeze priority 2 2014/10/30 07:14:06.780507 [ 7610]: Freeze priority 3 2014/10/30 07:14:06.780661 [ 7610]: Freeze priority 3 2014/10/30 07:14:06.780798 [ 7610]: Freeze priority 3 2014/10/30 07:14:09.802458 [ 7610]: Freeze priority 1 2014/10/30 07:14:09.802840 [ 7610]: Freeze priority 2 2014/10/30 07:14:09.803214 [ 7610]: Freeze priority 3 2014/10/30 07:14:14.742248 [ 7610]: Thawing priority 1 2014/10/30 07:14:14.742287 [ 7610]: Release freeze handler for prio 1 2014/10/30 07:14:14.742321 [ 7610]: Thawing priority 2 2014/10/30 07:14:14.742343 [ 7610]: Release freeze handler for prio 2 2014/10/30 07:14:14.742375 [ 7610]: Thawing priority 3 2014/10/30 07:14:14.742396 [ 7610]: Release freeze handler for prio 3 2014/10/30 07:19:13.733797 [ 7610]: Freeze priority 1 2014/10/30 07:19:13.846895 [ 7610]: Freeze priority 2 2014/10/30 07:19:13.848199 [ 7610]: Freeze priority 3 2014/10/30 07:19:16.638831 [ 7610]: Thawing priority 1 2014/10/30 07:19:16.638897 [ 7610]: Release freeze handler for prio 1 2014/10/30 07:19:16.638947 [ 7610]: Thawing priority 2 2014/10/30 07:19:16.638966 [ 7610]: Release freeze handler for prio 2 2014/10/30 07:19:16.638994 [ 7610]: Thawing priority 3 2014/10/30 07:19:16.639021 [ 7610]: Release freeze handler for prio 3 2014/10/30 07:19:27.107699 [ 7610]: Freeze priority 1 2014/10/30 07:19:27.122193 [ 7610]: Freeze priority 2 2014/10/30 07:19:27.124853 [ 7610]: Freeze priority 3 2014/10/30 07:19:29.752674 [ 7610]: Thawing priority 1 2014/10/30 07:19:29.752727 [ 7610]: Release freeze handler for prio 1 2014/10/30 07:19:29.752778 [ 7610]: Thawing priority 2 2014/10/30 07:19:29.752800 [ 7610]: Release freeze handler for prio 2 2014/10/30 07:19:29.752839 [ 7610]: Thawing priority 3 2014/10/30 07:19:29.752854 [ 7610]: Release freeze handler for prio 3 2014/10/30 07:29:16.527007 [ 7610]: Freeze priority 1 2014/10/30 07:29:16.556096 [ 7610]: Freeze priority 1 2014/10/30 07:29:16.600786 [ 7610]: Freeze priority 1 2014/10/30 07:29:27.062102 [ 7610]: Skip monitoring since databases are frozen 2014/10/30 07:29:42.062395 [ 7610]: Skip monitoring since databases are frozen 2014/10/30 07:29:57.062537 [ 7610]: Skip monitoring since databases are frozen 2014/10/30 07:30:12.062961 [ 7610]: Skip monitoring since databases are frozen 2014/10/30 07:30:16.530216 [ 7610]: Freeze priority 1 2014/10/30 07:30:16.540339 [ 7610]: Recovery daemon ping timeout. Count : 0 2014/10/30 07:30:16.556515 [recoverd: 7904]: ctdb_control error: 'ctdb_control timed out' 2014/10/30 07:30:16.556561 [recoverd: 7904]: ctdb_control error: 'ctdb_control timed out' 2014/10/30 07:30:16.556587 [recoverd: 7904]: Async operation failed with ret=-1 res=-1 opcode=33 2014/10/30 07:30:16.556603 [recoverd: 7904]: Failed to freeze node 1 during recovery. Set it as ban culprit for 4 credits 2014/10/30 07:30:16.556623 [recoverd: 7904]: Async wait failed - fail_count=1 2014/10/30 07:30:16.556638 [recoverd: 7904]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed. 2014/10/30 07:30:16.556655 [recoverd: 7904]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster 2014/10/30 07:30:16.558255 [ 7610]: Freeze priority 1 2014/10/30 07:30:16.602992 [ 7610]: Freeze priority 1 2014/10/30 07:30:27.063305 [ 7610]: Skip monitoring since databases are frozen ===== Start of debug locks PID=8307 ===== 5505 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 7989 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 184876 184876 W 5505 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=5505 ----- #0 0x00007f90287ac890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f90287ac744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fffd09b9f18) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=8307 ===== ===== Start of debug locks PID=8859 ===== 5505 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 7989 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 184876 184876 W 5505 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=5505 ----- #0 0x00007f90287ac890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f90287ac744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fffd09b9f18) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=8859 ===== 2014/10/30 07:30:42.063778 [ 7610]: Skip monitoring since databases are frozen ===== Start of debug locks PID=9201 ===== 5505 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 7989 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 184876 184876 W 5505 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=5505 ----- #0 0x00007f90287ac890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f90287ac744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fffd09b9f18) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=9201 ===== 2014/10/30 07:30:57.064304 [ 7610]: Skip monitoring since databases are frozen ===== Start of debug locks PID=9672 ===== 5505 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 7989 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 184876 184876 W 5505 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=5505 ----- #0 0x00007f90287ac890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f90287ac744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fffd09b9f18) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=9672 ===== ===== Start of debug locks PID=10101 ===== 5505 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 10259 /usr/sbin/smbd printer_list.tdb.2 2984 2984 W 5505 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 5505 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 7989 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 184876 184876 W 5505 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=5505 ----- #0 0x00007f90287ac890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f90287ac744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fffd09b9f18) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=10101 ===== 2014/10/30 07:31:12.064709 [ 7610]: Skip monitoring since databases are frozen 2014/10/30 07:31:16.557795 [ 7610]: Recovery daemon ping timeout. Count : 0 2014/10/30 07:31:16.559004 [recoverd: 7904]: ctdb_control error: 'ctdb_control timed out' 2014/10/30 07:31:16.559053 [recoverd: 7904]: ctdb_control error: 'ctdb_control timed out' 2014/10/30 07:31:16.559077 [recoverd: 7904]: Async operation failed with ret=-1 res=-1 opcode=33 2014/10/30 07:31:16.559092 [recoverd: 7904]: Failed to freeze node 1 during recovery. Set it as ban culprit for 4 credits 2014/10/30 07:31:16.559112 [recoverd: 7904]: Async wait failed - fail_count=1 2014/10/30 07:31:16.559128 [recoverd: 7904]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed. 2014/10/30 07:31:16.559144 [recoverd: 7904]: server/ctdb_recoverd.c:2720 Unable to set recovery mode to active on cluster 2014/10/30 07:31:16.773102 [ 7610]: Freeze priority 1 2014/10/30 07:31:16.773203 [ 7610]: Freeze priority 1 2014/10/30 07:31:16.823226 [ 7610]: pnn 2 Invalid reqid 45672 in ctdb_reply_control 2014/10/30 07:31:16.823262 [ 7610]: pnn 2 Invalid reqid 45455 in ctdb_reply_control 2014/10/30 07:31:16.823335 [ 7610]: Freeze priority 2 2014/10/30 07:31:16.823479 [ 7610]: Freeze priority 2 2014/10/30 07:31:16.824328 [ 7610]: Freeze priority 3 2014/10/30 07:31:16.824473 [ 7610]: Freeze priority 3 2014/10/30 07:31:19.832708 [recoverd: 7904]: Taking out recovery lock from recovery daemon 2014/10/30 07:31:19.832749 [recoverd: 7904]: Take the recovery lock 2014/10/30 07:31:19.845824 [ 7610]: Freeze priority 1 2014/10/30 07:31:19.846127 [ 7610]: Freeze priority 2 2014/10/30 07:31:19.846445 [ 7610]: Freeze priority 3 2014/10/30 07:31:21.530602 [ 7610]: Thawing priority 1 2014/10/30 07:31:21.530655 [ 7610]: Release freeze handler for prio 1 2014/10/30 07:31:21.530698 [ 7610]: Thawing priority 2 2014/10/30 07:31:21.530720 [ 7610]: Release freeze handler for prio 2 2014/10/30 07:31:21.530761 [ 7610]: Thawing priority 3 2014/10/30 07:31:21.530786 [ 7610]: Release freeze handler for prio 3 ===== Start of debug locks PID=10541 ===== ===== End of debug locks PID=10541 ===== 2014/10/30 07:31:26.879924 [ 7610]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 07:31:27.081816 [recoverd: 7904]: Resetting ban count to 0 for all nodes 2014/10/30 07:31:47.105533 [recoverd: 7904]: Taking out recovery lock from recovery daemon 2014/10/30 07:31:47.105571 [recoverd: 7904]: Take the recovery lock 2014/10/30 07:31:47.117516 [ 7610]: Freeze priority 1 2014/10/30 07:31:47.118872 [ 7610]: Freeze priority 2 2014/10/30 07:31:47.119920 [ 7610]: Freeze priority 3 2014/10/30 07:31:50.735331 [ 7610]: Thawing priority 1 2014/10/30 07:31:50.735373 [ 7610]: Release freeze handler for prio 1 2014/10/30 07:31:50.735405 [ 7610]: Thawing priority 2 2014/10/30 07:31:50.735426 [ 7610]: Release freeze handler for prio 2 2014/10/30 07:31:50.735457 [ 7610]: Thawing priority 3 2014/10/30 07:31:50.735476 [ 7610]: Release freeze handler for prio 3 2014/10/30 07:31:56.373950 [recoverd: 7904]: Resetting ban count to 0 for all nodes 2014/10/30 07:34:33.032954 [ 7610]: Freeze priority 1 2014/10/30 07:34:33.034431 [ 7610]: Freeze priority 2 2014/10/30 07:34:33.035341 [ 7610]: Freeze priority 3 2014/10/30 07:34:37.416416 [recoverd: 7904]: Taking out recovery lock from recovery daemon 2014/10/30 07:34:37.416541 [recoverd: 7904]: Take the recovery lock 2014/10/30 07:34:37.416673 [ 7610]: Thawing priority 1 2014/10/30 07:34:37.416717 [ 7610]: Release freeze handler for prio 1 2014/10/30 07:34:37.416806 [ 7610]: Thawing priority 2 2014/10/30 07:34:37.416833 [ 7610]: Release freeze handler for prio 2 2014/10/30 07:34:37.416869 [ 7610]: Thawing priority 3 2014/10/30 07:34:37.416888 [ 7610]: Release freeze handler for prio 3 2014/10/30 07:34:37.429293 [ 7610]: Freeze priority 1 2014/10/30 07:34:37.431502 [ 7610]: Freeze priority 2 2014/10/30 07:34:37.433209 [ 7610]: Freeze priority 3 2014/10/30 07:34:37.609490 [ 7610]: Refusing to run event scripts call 'recovered' while in recovery 2014/10/30 07:34:37.609527 [ 7610]: server/ctdb_recover.c:952 Failed to end recovery 2014/10/30 07:34:40.526609 [ 7610]: Thawing priority 1 2014/10/30 07:34:40.526652 [ 7610]: Release freeze handler for prio 1 2014/10/30 07:34:40.526681 [ 7610]: Thawing priority 2 2014/10/30 07:34:40.526700 [ 7610]: Release freeze handler for prio 2 2014/10/30 07:34:40.526726 [ 7610]: Thawing priority 3 2014/10/30 07:34:40.526743 [ 7610]: Release freeze handler for prio 3 2014/10/30 07:34:40.528143 [ 7610]: Freeze priority 1 2014/10/30 07:34:40.532745 [ 7610]: Freeze priority 2 2014/10/30 07:34:40.534611 [ 7610]: Freeze priority 3 2014/10/30 07:34:40.786081 [recoverd: 7904]: Async operation failed with ret=0 res=-1 opcode=71 2014/10/30 07:34:40.786131 [recoverd: 7904]: server/ctdb_recoverd.c:239 Node 1 failed the recovered event. Setting it as recovery fail culprit 2014/10/30 07:34:40.927466 [recoverd: 7904]: Async wait failed - fail_count=1 2014/10/30 07:34:40.927520 [recoverd: 7904]: server/ctdb_recoverd.c:262 Unable to run the 'recovered' event when called from do_recovery 2014/10/30 07:34:40.927533 [recoverd: 7904]: server/ctdb_recoverd.c:2016 Unable to run the 'recovered' event on cluster. Recovery process failed. 2014/10/30 07:34:43.934177 [recoverd: 7904]: Taking out recovery lock from recovery daemon 2014/10/30 07:34:43.934229 [recoverd: 7904]: Take the recovery lock 2014/10/30 07:34:43.945996 [ 7610]: Freeze priority 1 2014/10/30 07:34:46.308336 [ 7610]: Skip monitoring since databases are frozen ===== Start of debug locks PID=22007 ===== 21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF 21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W 21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=21453 ----- #0 0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f008d132744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=22007 ===== 2014/10/30 07:34:51.309222 [ 7610]: Skip monitoring since databases are frozen 2014/10/30 07:34:56.309861 [ 7610]: Skip monitoring since databases are frozen ===== Start of debug locks PID=22378 ===== 21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF 21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W 21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=21453 ----- #0 0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f008d132744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=22378 ===== 2014/10/30 07:35:01.310906 [ 7610]: Skip monitoring since databases are frozen 2014/10/30 07:35:06.311314 [ 7610]: Skip monitoring since databases are frozen ===== Start of debug locks PID=22878 ===== 21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF 21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W 21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=21453 ----- #0 0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f008d132744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=22878 ===== 2014/10/30 07:35:11.312360 [ 7610]: Skip monitoring since databases are frozen 2014/10/30 07:35:16.312911 [ 7610]: Skip monitoring since databases are frozen ===== Start of debug locks PID=23263 ===== 21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF 21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W 21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=21453 ----- #0 0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f008d132744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=23263 ===== 2014/10/30 07:35:21.313408 [ 7610]: Skip monitoring since databases are frozen 2014/10/30 07:35:26.313560 [ 7610]: Skip monitoring since databases are frozen ===== Start of debug locks PID=23618 ===== 21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF 21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W 21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=21453 ----- #0 0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f008d132744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=23618 ===== 2014/10/30 07:35:31.314102 [ 7610]: Skip monitoring since databases are frozen 2014/10/30 07:35:36.314651 [ 7610]: Skip monitoring since databases are frozen ===== Start of debug locks PID=24143 ===== 21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF 21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W 21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=21453 ----- #0 0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f008d132744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=24143 ===== 2014/10/30 07:35:41.315706 [ 7610]: Skip monitoring since databases are frozen 2014/10/30 07:35:43.932392 [ 7610]: Recovery daemon ping timeout. Count : 0 2014/10/30 07:35:43.946543 [recoverd: 7904]: ctdb_control error: 'ctdb_control timed out' 2014/10/30 07:35:43.946604 [recoverd: 7904]: ctdb_control error: 'ctdb_control timed out' 2014/10/30 07:35:43.946640 [recoverd: 7904]: Async operation failed with ret=-1 res=-1 opcode=33 2014/10/30 07:35:43.946667 [recoverd: 7904]: Failed to freeze node 3 during recovery. Set it as ban culprit for 4 credits 2014/10/30 07:35:43.946688 [recoverd: 7904]: Async wait failed - fail_count=1 2014/10/30 07:35:43.946700 [recoverd: 7904]: server/ctdb_recoverd.c:395 Unable to freeze nodes. Recovery failed. 2014/10/30 07:35:43.946713 [recoverd: 7904]: server/ctdb_recoverd.c:1833 Unable to set recovery mode to active on cluster 2014/10/30 07:35:43.958196 [recoverd: 7904]: Taking out recovery lock from recovery daemon 2014/10/30 07:35:43.958235 [recoverd: 7904]: Take the recovery lock 2014/10/30 07:35:43.994158 [ 7610]: Freeze priority 1 2014/10/30 07:35:46.315985 [ 7610]: Skip monitoring since databases are frozen ===== Start of debug locks PID=24780 ===== 21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF 21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W 21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=21453 ----- #0 0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f008d132744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=24780 ===== 2014/10/30 07:35:51.316892 [ 7610]: Skip monitoring since databases are frozen 2014/10/30 07:35:56.318008 [ 7610]: Skip monitoring since databases are frozen ===== Start of debug locks PID=25143 ===== 21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF 21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W 21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=21453 ----- #0 0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f008d132744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=25143 ===== 2014/10/30 07:36:01.318718 [ 7610]: Skip monitoring since databases are frozen 2014/10/30 07:36:06.319414 [ 7610]: Skip monitoring since databases are frozen ===== Start of debug locks PID=25650 ===== 21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF 21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W 25569 /usr/bin/ctdb_lock_helper locking.tdb.2 168196 168196 W 21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=21453 ----- #0 0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f008d132744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=25650 ===== 2014/10/30 07:36:11.320130 [ 7610]: Skip monitoring since databases are frozen 2014/10/30 07:36:16.320764 [ 7610]: Skip monitoring since databases are frozen ===== Start of debug locks PID=26021 ===== 21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF 21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W 25569 /usr/bin/ctdb_lock_helper locking.tdb.2 168196 168196 W 21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=21453 ----- #0 0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f008d132744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=26021 ===== ===== Start of debug locks PID=26142 ===== 21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF 21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W 25569 /usr/bin/ctdb_lock_helper locking.tdb.2 168196 168196 W 21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=21453 ----- #0 0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f008d132744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=26142 ===== 2014/10/30 07:36:21.321265 [ 7610]: Skip monitoring since databases are frozen 2014/10/30 07:36:26.322244 [ 7610]: Skip monitoring since databases are frozen ===== Start of debug locks PID=26503 ===== 21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF 21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W 25569 /usr/bin/ctdb_lock_helper locking.tdb.2 168196 168196 W 21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=21453 ----- #0 0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f008d132744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=26503 ===== ===== Start of debug locks PID=26626 ===== 21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF 21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W 25569 /usr/bin/ctdb_lock_helper locking.tdb.2 168196 168196 W 21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=21453 ----- #0 0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f008d132744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=26626 ===== 2014/10/30 07:36:31.322929 [ 7610]: Skip monitoring since databases are frozen 2014/10/30 07:36:36.324011 [ 7610]: Skip monitoring since databases are frozen ===== Start of debug locks PID=27048 ===== 21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF 21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W 25569 /usr/bin/ctdb_lock_helper locking.tdb.2 168196 168196 W 21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=21453 ----- #0 0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f008d132744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=27048 ===== 2014/10/30 07:36:40.538616 [ 7610]: server/ctdb_recover.c:562 Been in recovery mode for too long. Dropping all IPS ===== Start of debug locks PID=27242 ===== 21460 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF 21460 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF 21456 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/registry.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/passdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/secrets.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/share_info.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/ctdb.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/account_policy.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper persistent/group_mapping.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper locking.tdb.2 168 EOF 21454 /usr/bin/ctdb_lock_helper locking.tdb.2 298336 298336 W 25569 /usr/bin/ctdb_lock_helper locking.tdb.2 168196 168196 W 21453 /usr/bin/ctdb_lock_helper smbXsrv_tcon_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_session_global.tdb.2 168 EOF 21453 /usr/bin/ctdb_lock_helper smbXsrv_version_global.tdb.2 168 EOF ----- Stack trace for PID=21453 ----- #0 0x00007f008d132890 in __nanosleep_nocancel () from /lib64/libc.so.6 #1 0x00007f008d132744 in sleep () from /lib64/libc.so.6 #2 0x0000000000401ef7 in main (argc=17, argv=0x7fff84b56bc8) at server/ctdb_lock_helper.c:145 ===== End of debug locks PID=27242 ===== 2014/10/30 07:36:40.704508 [ 7610]: pnn 2 Invalid reqid 53706 in ctdb_reply_control 2014/10/30 07:36:40.704817 [ 7610]: Freeze priority 2 2014/10/30 07:36:40.705312 [ 7610]: Freeze priority 3 2014/10/30 07:36:43.313289 [ 7610]: Thawing priority 1 2014/10/30 07:36:43.313326 [ 7610]: Release freeze handler for prio 1 2014/10/30 07:36:43.313363 [ 7610]: Thawing priority 2 2014/10/30 07:36:43.313384 [ 7610]: Release freeze handler for prio 2 2014/10/30 07:36:43.313414 [ 7610]: Thawing priority 3 2014/10/30 07:36:43.313433 [ 7610]: Release freeze handler for prio 3 2014/10/30 07:36:43.617253 [ 7610]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 07:36:43.799970 [recoverd: 7904]: Resetting ban count to 0 for all nodes 2014/10/30 07:36:43.947685 [ 7610]: Recovery daemon ping timeout. Count : 0 2014/10/30 07:39:25.026633 [recoverd: 7904]: server/ctdb_recoverd.c:3960 The vnnmap count is different from the number of active lmaster nodes: 4 vs 3 2014/10/30 07:39:25.067517 [recoverd: 7904]: Taking out recovery lock from recovery daemon 2014/10/30 07:39:25.067551 [recoverd: 7904]: Take the recovery lock 2014/10/30 07:39:25.233241 [ 7610]: Freeze priority 1 2014/10/30 07:39:25.334714 [ 7610]: Freeze priority 2 2014/10/30 07:39:25.336778 [ 7610]: Freeze priority 3 2014/10/30 07:39:25.339993 [ 7610]: Monitoring event was cancelled 2014/10/30 07:39:25.340032 [ 7610]: server/eventscript.c:569 Sending SIGTERM to child pid:3866 2014/10/30 07:39:28.097089 [ 7610]: Thawing priority 1 2014/10/30 07:39:28.097132 [ 7610]: Release freeze handler for prio 1 2014/10/30 07:39:28.097164 [ 7610]: Thawing priority 2 2014/10/30 07:39:28.097184 [ 7610]: Release freeze handler for prio 2 2014/10/30 07:39:28.097213 [ 7610]: Thawing priority 3 2014/10/30 07:39:28.097231 [ 7610]: Release freeze handler for prio 3 2014/10/30 07:39:28.835411 [recoverd: 7904]: Resetting ban count to 0 for all nodes 2014/10/30 07:46:38.435309 [recoverd: 7904]: Taking out recovery lock from recovery daemon 2014/10/30 07:46:38.435363 [recoverd: 7904]: Take the recovery lock 2014/10/30 07:46:38.486660 [ 7610]: Freeze priority 1 2014/10/30 07:46:38.507490 [ 7610]: Freeze priority 2 2014/10/30 07:46:38.508776 [ 7610]: Freeze priority 3 2014/10/30 07:46:41.464641 [ 7610]: Thawing priority 1 2014/10/30 07:46:41.464679 [ 7610]: Release freeze handler for prio 1 2014/10/30 07:46:41.464706 [ 7610]: Thawing priority 2 2014/10/30 07:46:41.464725 [ 7610]: Release freeze handler for prio 2 2014/10/30 07:46:41.464762 [ 7610]: Thawing priority 3 2014/10/30 07:46:41.464782 [ 7610]: Release freeze handler for prio 3 2014/10/30 07:46:41.835633 [recoverd: 7904]: Resetting ban count to 0 for all nodes 2014/10/30 07:49:34.964116 [ 7610]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 07:49:35.021317 [ 7610]: common/ctdb_fork.c:131 waitpid() returned error. errno:10 2014/10/30 07:49:35.021351 [ 7610]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 07:51:00.837394 [ 1487]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/30 07:51:00.837489 [ 1487]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 07:51:00.837503 [ 1487]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 07:55:00.654581 [ 7845]: Starting CTDBD (Version 2.5.3) as PID: 7845 2014/10/30 07:55:02.528277 [ 7845]: Vacuuming is disabled for persistent database registry.tdb 2014/10/30 07:55:02.552227 [ 7845]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/30 07:55:02.567580 [ 7845]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/30 07:55:02.582547 [ 7845]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/30 07:55:02.582573 [ 7845]: Ignoring persistent database 'account_policy.tdb.1' 2014/10/30 07:55:02.582582 [ 7845]: Ignoring persistent database 'ctdb.tdb.1' 2014/10/30 07:55:02.582590 [ 7845]: Ignoring persistent database 'group_mapping.tdb.1' 2014/10/30 07:55:02.582599 [ 7845]: Ignoring persistent database 'secrets.tdb.1' 2014/10/30 07:55:02.582607 [ 7845]: Ignoring persistent database 'share_info.tdb.1' 2014/10/30 07:55:02.597961 [ 7845]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/30 07:55:02.613320 [ 7845]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/30 07:55:02.613347 [ 7845]: Ignoring persistent database 'passdb.tdb.1' 2014/10/30 07:55:02.613357 [ 7845]: Ignoring persistent database 'registry.tdb.1' 2014/10/30 07:55:02.628585 [ 7845]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/30 07:55:02.631128 [ 7845]: Freeze priority 1 2014/10/30 07:55:02.665985 [ 7845]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 07:55:02.681036 [ 7845]: Freeze priority 2 2014/10/30 07:55:02.681416 [ 7845]: Freeze priority 3 2014/10/30 07:55:02.862259 [ 7845]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/30 07:55:02.871303 [ 7845]: 00.ctdb: Set RecoverTimeout to 60 2014/10/30 07:55:02.874920 [ 7845]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/30 07:55:02.965682 [ 7845]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 07:55:03.005942 [ 7845]: Freeze priority 1 2014/10/30 07:55:03.006024 [ 7845]: Freeze priority 2 2014/10/30 07:55:03.006081 [ 7845]: Freeze priority 3 2014/10/30 07:55:03.217031 [ 7845]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 07:55:07.015239 [recoverd: 8565]: server/ctdb_recoverd.c:3692 Current recmaster node 3 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/30 07:55:07.015319 [ 7845]: Freeze priority 1 2014/10/30 07:55:07.015378 [ 7845]: Freeze priority 2 2014/10/30 07:55:07.015429 [ 7845]: Freeze priority 3 2014/10/30 07:55:11.513694 [ 7845]: Freeze priority 1 2014/10/30 07:55:11.522696 [ 7845]: Freeze priority 2 2014/10/30 07:55:11.523607 [ 7845]: Freeze priority 3 2014/10/30 07:55:11.693095 [ 7845]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/30 07:55:11.693624 [ 7845]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/30 07:55:11.694440 [ 7845]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup 2014/10/30 07:55:14.835355 [ 7845]: Thawing priority 1 2014/10/30 07:55:14.835406 [ 7845]: Release freeze handler for prio 1 2014/10/30 07:55:14.835440 [ 7845]: Thawing priority 2 2014/10/30 07:55:14.835461 [ 7845]: Release freeze handler for prio 2 2014/10/30 07:55:14.835490 [ 7845]: Thawing priority 3 2014/10/30 07:55:14.835508 [ 7845]: Release freeze handler for prio 3 2014/10/30 07:55:28.860157 [recoverd: 8565]: Trigger takeoverrun 2014/10/30 07:55:29.422806 [ 7845]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/30 07:55:29.746964 [ 7845]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 07:55:29.760258 [ 7845]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 07:55:29.788972 [ 7845]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/30 07:55:29.967240 [ 7845]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/30 07:55:32.433374 [ 7845]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation 2014/10/30 07:55:33.851886 [ 7845]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 07:59:38.204235 [ 7845]: Freeze priority 1 2014/10/30 07:59:38.218143 [ 7845]: Freeze priority 1 2014/10/30 07:59:38.220797 [ 7845]: Freeze priority 1 2014/10/30 07:59:38.221414 [ 7845]: Freeze priority 2 2014/10/30 07:59:38.221804 [ 7845]: Freeze priority 2 2014/10/30 07:59:38.221891 [ 7845]: Freeze priority 2 2014/10/30 07:59:38.223066 [ 7845]: Freeze priority 3 2014/10/30 07:59:38.223372 [ 7845]: Freeze priority 3 2014/10/30 07:59:38.223988 [ 7845]: Freeze priority 3 2014/10/30 07:59:41.266544 [ 7845]: Freeze priority 1 2014/10/30 07:59:41.267703 [ 7845]: Freeze priority 2 2014/10/30 07:59:41.268624 [ 7845]: Freeze priority 3 2014/10/30 07:59:45.249536 [ 7845]: Thawing priority 1 2014/10/30 07:59:45.249588 [ 7845]: Release freeze handler for prio 1 2014/10/30 07:59:45.249624 [ 7845]: Thawing priority 2 2014/10/30 07:59:45.249645 [ 7845]: Release freeze handler for prio 2 2014/10/30 07:59:45.249676 [ 7845]: Thawing priority 3 2014/10/30 07:59:45.249694 [ 7845]: Release freeze handler for prio 3 2014/10/30 08:04:45.491472 [ 7845]: Freeze priority 1 2014/10/30 08:04:45.512095 [ 7845]: Freeze priority 2 2014/10/30 08:04:45.516010 [ 7845]: Freeze priority 3 2014/10/30 08:04:48.839050 [ 7845]: Thawing priority 1 2014/10/30 08:04:48.839108 [ 7845]: Release freeze handler for prio 1 2014/10/30 08:04:48.839159 [ 7845]: Thawing priority 2 2014/10/30 08:04:48.839182 [ 7845]: Release freeze handler for prio 2 2014/10/30 08:04:48.839225 [ 7845]: Thawing priority 3 2014/10/30 08:04:48.839246 [ 7845]: Release freeze handler for prio 3 2014/10/30 08:14:43.633345 [ 7845]: Freeze priority 1 2014/10/30 08:14:43.642585 [ 7845]: Freeze priority 2 2014/10/30 08:14:43.644376 [ 7845]: Freeze priority 3 2014/10/30 08:14:46.716939 [ 7845]: Freeze priority 1 2014/10/30 08:14:46.717895 [ 7845]: Freeze priority 2 2014/10/30 08:14:46.718923 [ 7845]: Freeze priority 3 2014/10/30 08:14:48.791472 [ 7845]: Thawing priority 1 2014/10/30 08:14:48.791517 [ 7845]: Release freeze handler for prio 1 2014/10/30 08:14:48.791544 [ 7845]: Thawing priority 2 2014/10/30 08:14:48.791560 [ 7845]: Release freeze handler for prio 2 2014/10/30 08:14:48.791585 [ 7845]: Thawing priority 3 2014/10/30 08:14:48.791600 [ 7845]: Release freeze handler for prio 3 2014/10/30 08:20:01.439712 [ 7845]: Freeze priority 1 2014/10/30 08:20:01.582705 [ 7845]: Freeze priority 2 2014/10/30 08:20:01.587253 [ 7845]: Freeze priority 3 2014/10/30 08:20:04.899225 [ 7845]: Thawing priority 1 2014/10/30 08:20:04.899272 [ 7845]: Release freeze handler for prio 1 2014/10/30 08:20:04.899306 [ 7845]: Thawing priority 2 2014/10/30 08:20:04.899324 [ 7845]: Release freeze handler for prio 2 2014/10/30 08:20:04.899365 [ 7845]: Thawing priority 3 2014/10/30 08:20:04.899381 [ 7845]: Release freeze handler for prio 3 2014/10/30 08:24:46.847320 [ 7845]: Freeze priority 1 2014/10/30 08:24:49.269653 [ 7845]: Skip monitoring since databases are frozen ===== Start of debug locks PID=4673 ===== 5800 /usr/sbin/smbd locking.tdb.2 353844 353844 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843 5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W 4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=4206 ----- #0 0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537 #8 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129 ----- Stack trace for PID=5800 ----- #0 0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #29 0x00007f9d18ee91b4 in smbd_accept_connection () #30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0 #31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #33 0x00007f9d18ee5d01 in main () ===== End of debug locks PID=4673 ===== 2014/10/30 08:25:04.270366 [ 7845]: Skip monitoring since databases are frozen ===== Start of debug locks PID=5541 ===== 5800 /usr/sbin/smbd locking.tdb.2 353844 353844 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843 5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W 4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=4206 ----- #0 0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537 #8 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129 ----- Stack trace for PID=5800 ----- #0 0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #29 0x00007f9d18ee91b4 in smbd_accept_connection () #30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0 #31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #33 0x00007f9d18ee5d01 in main () ===== End of debug locks PID=5541 ===== ===== Start of debug locks PID=6277 ===== 5800 /usr/sbin/smbd locking.tdb.2 353844 353844 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843 5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W 4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=4206 ----- #0 0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537 #8 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129 ----- Stack trace for PID=5800 ----- #0 0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #29 0x00007f9d18ee91b4 in smbd_accept_connection () #30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0 #31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #33 0x00007f9d18ee5d01 in main () ===== End of debug locks PID=6277 ===== 2014/10/30 08:25:19.271062 [ 7845]: Skip monitoring since databases are frozen ===== Start of debug locks PID=6646 ===== 5800 /usr/sbin/smbd locking.tdb.2 353844 353844 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843 5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W 4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=4206 ----- #0 0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537 #8 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129 ----- Stack trace for PID=5800 ----- #0 0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #29 0x00007f9d18ee91b4 in smbd_accept_connection () #30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0 #31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #33 0x00007f9d18ee5d01 in main () ===== End of debug locks PID=6646 ===== 2014/10/30 08:25:34.271397 [ 7845]: Skip monitoring since databases are frozen ===== Start of debug locks PID=7132 ===== 5800 /usr/sbin/smbd locking.tdb.2 353844 353844 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843 5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W 4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=4206 ----- #0 0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537 #8 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129 ----- Stack trace for PID=5800 ----- #0 0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #29 0x00007f9d18ee91b4 in smbd_accept_connection () #30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0 #31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #33 0x00007f9d18ee5d01 in main () ===== End of debug locks PID=7132 ===== 2014/10/30 08:25:46.865237 [ 7845]: Freeze priority 1 ===== Start of debug locks PID=7589 ===== 5800 /usr/sbin/smbd locking.tdb.2 353844 353844 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843 5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W 4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=4206 ----- #0 0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537 #8 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129 ----- Stack trace for PID=5800 ----- #0 0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #29 0x00007f9d18ee91b4 in smbd_accept_connection () #30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0 #31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #33 0x00007f9d18ee5d01 in main () ===== End of debug locks PID=7589 ===== 2014/10/30 08:25:49.271899 [ 7845]: Skip monitoring since databases are frozen ===== Start of debug locks PID=7955 ===== 5800 /usr/sbin/smbd locking.tdb.2 353844 353844 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843 5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W 4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=4206 ----- #0 0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537 #8 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129 ----- Stack trace for PID=5800 ----- #0 0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #29 0x00007f9d18ee91b4 in smbd_accept_connection () #30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0 #31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #33 0x00007f9d18ee5d01 in main () ===== End of debug locks PID=7955 ===== 2014/10/30 08:26:04.272070 [ 7845]: Skip monitoring since databases are frozen ===== Start of debug locks PID=8429 ===== 5800 /usr/sbin/smbd locking.tdb.2 353844 353844 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843 5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W 4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=4206 ----- #0 0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537 #8 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129 ----- Stack trace for PID=5800 ----- #0 0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #29 0x00007f9d18ee91b4 in smbd_accept_connection () #30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0 #31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #33 0x00007f9d18ee5d01 in main () ===== End of debug locks PID=8429 ===== ===== Start of debug locks PID=9001 ===== 5800 /usr/sbin/smbd locking.tdb.2 353844 353844 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843 5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W 4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=4206 ----- #0 0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537 #8 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129 ----- Stack trace for PID=5800 ----- #0 0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #29 0x00007f9d18ee91b4 in smbd_accept_connection () #30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0 #31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #33 0x00007f9d18ee5d01 in main () ===== End of debug locks PID=9001 ===== 2014/10/30 08:26:19.272136 [ 7845]: Skip monitoring since databases are frozen ===== Start of debug locks PID=9405 ===== 5800 /usr/sbin/smbd locking.tdb.2 353844 353844 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843 5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W 4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=4206 ----- #0 0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537 #8 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129 ----- Stack trace for PID=5800 ----- #0 0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #29 0x00007f9d18ee91b4 in smbd_accept_connection () #30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0 #31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #33 0x00007f9d18ee5d01 in main () ===== End of debug locks PID=9405 ===== 2014/10/30 08:26:34.273157 [ 7845]: Skip monitoring since databases are frozen ===== Start of debug locks PID=9848 ===== 5800 /usr/sbin/smbd locking.tdb.2 353844 353844 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843 5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W 4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=4206 ----- #0 0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537 #8 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129 ----- Stack trace for PID=5800 ----- #0 0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6 #1 0x00007f9d17118db9 in poll_one_fd () from /lib64/libsmbconf.so.0 #2 0x00007f9d1711c3bb in ctdb_packet_fd_read_sync_timeout () from /lib64/libsmbconf.so.0 #3 0x00007f9d1711d5ca in ctdb_read_req () from /lib64/libsmbconf.so.0 #4 0x00007f9d1712010f in ctdbd_parse () from /lib64/libsmbconf.so.0 #5 0x00007f9d171259ab in db_ctdb_parse_record () from /lib64/libsmbconf.so.0 #6 0x00007f9d13a21afb in dbwrap_parse_record () from /usr/lib64/samba/libdbwrap.so #7 0x00007f9d13a21b2e in dbwrap_fetch () from /usr/lib64/samba/libdbwrap.so #8 0x00007f9d171275e1 in dbwrap_watch_record_stored () from /lib64/libsmbconf.so.0 #9 0x00007f9d13a218a6 in dbwrap_record_store () from /usr/lib64/samba/libdbwrap.so #10 0x00007f9d184d8077 in share_mode_data_destructor () from /usr/lib64/samba/libsmbd_base.so #11 0x00007f9d15d35e38 in _talloc_free_internal () from /lib64/libtalloc.so.2 #12 0x00007f9d15d35c33 in _talloc_free_internal () from /lib64/libtalloc.so.2 #13 0x00007f9d15d327db in _talloc_free () from /lib64/libtalloc.so.2 #14 0x00007f9d184d1617 in set_delete_on_close () from /usr/lib64/samba/libsmbd_base.so #15 0x00007f9d18438000 in smb_set_file_disposition_info () from /usr/lib64/samba/libsmbd_base.so #16 0x00007f9d184469db in smbd_do_setfilepathinfo () from /usr/lib64/samba/libsmbd_base.so #17 0x00007f9d184994bf in smbd_smb2_request_process_setinfo () from /usr/lib64/samba/libsmbd_base.so #18 0x00007f9d18487911 in smbd_smb2_request_dispatch () from /usr/lib64/samba/libsmbd_base.so #19 0x00007f9d1848819f in smbd_smb2_request_incoming () from /usr/lib64/samba/libsmbd_base.so #20 0x00007f9d1848509c in smbd_smb2_request_read_done () from /usr/lib64/samba/libsmbd_base.so #21 0x00007f9d16ee1534 in tstream_readv_pdu_queue_done () from /usr/lib64/samba/libsamba-sockets.so #22 0x00007f9d16ee1069 in tstream_readv_pdu_readv_done () from /usr/lib64/samba/libsamba-sockets.so #23 0x00007f9d16edff46 in tstream_readv_done () from /usr/lib64/samba/libsamba-sockets.so #24 0x00007f9d15b273f4 in tevent_common_loop_immediate () from /lib64/libtevent.so.0 #25 0x00007f9d1713171c in run_events_poll () from /lib64/libsmbconf.so.0 #26 0x00007f9d17131a04 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #27 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #28 0x00007f9d18473bb0 in smbd_process () from /usr/lib64/samba/libsmbd_base.so #29 0x00007f9d18ee91b4 in smbd_accept_connection () #30 0x00007f9d1713184c in run_events_poll () from /lib64/libsmbconf.so.0 #31 0x00007f9d17131aa0 in s3_event_loop_once () from /lib64/libsmbconf.so.0 #32 0x00007f9d15b26bcd in _tevent_loop_once () from /lib64/libtevent.so.0 #33 0x00007f9d18ee5d01 in main () ===== End of debug locks PID=9848 ===== 2014/10/30 08:26:46.866203 [ 7845]: Banning this node for 30 seconds 2014/10/30 08:26:46.866272 [ 7845]: Freeze priority 1 2014/10/30 08:26:46.866286 [ 7845]: Freeze priority 2 2014/10/30 08:26:46.866420 [ 7845]: Freeze priority 3 ===== Start of debug locks PID=10292 ===== 10282 /usr/bin/ctdb_lock_helper dbwrap_watchers.tdb.2 168 EOF 10282 /usr/bin/ctdb_lock_helper notify_index.tdb.2 168 EOF 5800 /usr/sbin/smbd locking.tdb.2 353844 353844 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 353844 353846 W 10281 /usr/bin/ctdb_lock_helper serverid.tdb.2 168 EOF 10281 /usr/bin/ctdb_lock_helper g_lock.tdb.2 168 EOF 10281 /usr/bin/ctdb_lock_helper brlock.tdb.2 168 EOF 4206 /usr/bin/ctdb_lock_helper locking.tdb.2 168 353843 5793 /usr/sbin/smbd locking.tdb.2 135724 135724 W 4206 /usr/bin/ctdb_lock_helper smbXsrv_open_global.tdb.2 168 EOF 4206 /usr/bin/ctdb_lock_helper printer_list.tdb.2 168 EOF ----- Stack trace for PID=4206 ----- 2014/10/30 08:26:47.008997 [ 7845]: 10.interface: Killing TCP connection 10.10.10.208:63875 10.10.10.183:445 2014/10/30 08:26:47.009171 [ 7845]: 10.interface: Killing TCP connection 10.10.10.206:49739 10.10.10.183:445 2014/10/30 08:26:47.009332 [ 7845]: 10.interface: Killing TCP connection 10.10.10.205:54766 10.10.10.183:445 #0 0x00007fe112e6f094 in fcntl () from /lib64/libc.so.6 #1 0x000000000040e034 in fcntl_lock (tdb=0x222b370, rw=1, off=353844, len=3, waitflag=true) at lib/tdb/common/lock.c:47 #2 0x000000000040e161 in tdb_brlock (tdb=0x222b370, rw_type=1, offset=353844, len=3, flags=TDB_LOCK_WAIT) at lib/tdb/common/lock.c:156 #3 0x000000000040ed1b in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=3) at lib/tdb/common/lock.c:527 #4 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=6) at lib/tdb/common/lock.c:537 #5 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353844, len=12) at lib/tdb/common/lock.c:537 #6 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=24) at lib/tdb/common/lock.c:541 #7 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353832, len=49) at lib/tdb/common/lock.c:537 #8 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353783, len=98) at lib/tdb/common/lock.c:541 #9 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=195) at lib/tdb/common/lock.c:541 #10 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353686, len=391) at lib/tdb/common/lock.c:537 #11 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=781) at lib/tdb/common/lock.c:541 #12 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=1562) at lib/tdb/common/lock.c:537 #13 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=353296, len=3125) at lib/tdb/common/lock.c:537 #14 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=6250) at lib/tdb/common/lock.c:541 #15 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=12500) at lib/tdb/common/lock.c:537 #16 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=25000) at lib/tdb/common/lock.c:537 #17 0x000000000040ed75 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=350171, len=50001) at lib/tdb/common/lock.c:537 #18 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=300171, len=100001) at lib/tdb/common/lock.c:541 #19 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=200170, len=200002) at lib/tdb/common/lock.c:541 #20 0x000000000040edc0 in tdb_chainlock_gradual (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, off=168, len=400004) at lib/tdb/common/lock.c:541 #21 0x000000000040ee68 in tdb_allrecord_lock (tdb=0x222b370, ltype=1, flags=TDB_LOCK_WAIT, upgradable=false) at lib/tdb/common/lock.c:570 #22 0x000000000040f117 in tdb_lockall (tdb=0x222b370) at lib/tdb/common/lock.c:650 #23 0x0000000000401d14 in lock_db (dbpath=0x7fff2b920d80 "/var/lib/ctdb/locking.tdb.2") at server/ctdb_lock_helper.c:86 #24 0x0000000000401e8a in main (argc=17, argv=0x7fff2b920ae8) at server/ctdb_lock_helper.c:129 ----- Stack trace for PID=5800 ----- 2014/10/30 08:26:47.033499 [ 7845]: 10.interface: Killed 3 TCP connections to released IP 10.10.10.183 2014/10/30 08:26:47.156700 [ 7845]: Freeze priority 1 #0 0x00007f9d1584ddf0 in __poll_nocancel () from /lib64/libc.so.6 ===== End of debug locks PID=10292 ===== 2014/10/30 08:26:49.139034 [ 7845]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 08:26:51.976804 [ 7845]: DB Attach to database ctdb.tdb refused since node is inactive (flags=0x8) 2014/10/30 08:27:16.866346 [ 7845]: Banning timedout 2014/10/30 08:27:17.596298 [ 7845]: Freeze priority 1 2014/10/30 08:27:17.602515 [ 7845]: Freeze priority 2 2014/10/30 08:27:17.606564 [ 7845]: Freeze priority 3 2014/10/30 08:27:20.159126 [ 7845]: Thawing priority 1 2014/10/30 08:27:20.159183 [ 7845]: Release freeze handler for prio 1 2014/10/30 08:27:20.159231 [ 7845]: Thawing priority 2 2014/10/30 08:27:20.159248 [ 7845]: Release freeze handler for prio 2 2014/10/30 08:27:20.159272 [ 7845]: Thawing priority 3 2014/10/30 08:27:20.159285 [ 7845]: Release freeze handler for prio 3 2014/10/30 08:27:20.162423 [ 7845]: pnn 2 Invalid reqid 60681 in ctdb_become_dmaster from node 3 2014/10/30 08:27:20.162505 [ 7845]: server/ctdb_call.c:1005 reqid 60682 not found 2014/10/30 08:27:20.693942 [ 7845]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 08:31:22.559339 [vacuum-smbXsrv_open_global.tdb:23765]: Error storing record copies on node 1: ret[0] res[-1] 2014/10/30 08:31:22.769552 [vacuum-locking.tdb:23774]: Error storing record copies on node 1: ret[0] res[-1] 2014/10/30 08:31:26.360084 [ 7845]: Freeze priority 1 2014/10/30 08:31:26.368578 [ 7845]: Freeze priority 2 2014/10/30 08:31:26.371792 [ 7845]: Freeze priority 3 2014/10/30 08:31:28.723884 [ 7845]: Thawing priority 1 2014/10/30 08:31:28.723944 [ 7845]: Release freeze handler for prio 1 2014/10/30 08:31:28.723983 [ 7845]: Thawing priority 2 2014/10/30 08:31:28.724004 [ 7845]: Release freeze handler for prio 2 2014/10/30 08:31:28.724035 [ 7845]: Thawing priority 3 2014/10/30 08:31:28.724052 [ 7845]: Release freeze handler for prio 3 2014/10/30 08:34:57.804563 [ 7845]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 08:34:57.849022 [ 7845]: common/ctdb_fork.c:131 waitpid() returned error. errno:10 2014/10/30 08:34:57.849096 [ 7845]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 08:36:25.589060 [ 1526]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/30 08:36:25.589166 [ 1526]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 08:36:25.589180 [ 1526]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 08:40:26.596007 [ 7759]: Starting CTDBD (Version 2.5.3) as PID: 7759 2014/10/30 08:40:28.044766 [ 7759]: Vacuuming is disabled for persistent database registry.tdb 2014/10/30 08:40:28.068204 [ 7759]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/30 08:40:28.082637 [ 7759]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/30 08:40:28.096741 [ 7759]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/30 08:40:28.096762 [ 7759]: Ignoring persistent database 'account_policy.tdb.1' 2014/10/30 08:40:28.096771 [ 7759]: Ignoring persistent database 'ctdb.tdb.1' 2014/10/30 08:40:28.096780 [ 7759]: Ignoring persistent database 'group_mapping.tdb.1' 2014/10/30 08:40:28.096789 [ 7759]: Ignoring persistent database 'secrets.tdb.1' 2014/10/30 08:40:28.096798 [ 7759]: Ignoring persistent database 'share_info.tdb.1' 2014/10/30 08:40:28.110786 [ 7759]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/30 08:40:28.124709 [ 7759]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/30 08:40:28.124729 [ 7759]: Ignoring persistent database 'passdb.tdb.1' 2014/10/30 08:40:28.124739 [ 7759]: Ignoring persistent database 'registry.tdb.1' 2014/10/30 08:40:28.138720 [ 7759]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/30 08:40:28.138751 [ 7759]: Freeze priority 1 2014/10/30 08:40:28.148213 [ 7759]: Freeze priority 2 2014/10/30 08:40:28.148565 [ 7759]: Freeze priority 3 2014/10/30 08:40:28.312516 [ 7759]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/30 08:40:28.316310 [ 7759]: 00.ctdb: Set RecoverTimeout to 60 2014/10/30 08:40:28.320031 [ 7759]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/30 08:40:28.348871 [ 7759]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 08:40:28.349040 [ 7759]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 08:40:28.349066 [ 7759]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 08:40:28.387846 [ 7759]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 08:40:28.387880 [ 7759]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 08:40:28.438846 [ 7759]: Freeze priority 1 2014/10/30 08:40:28.438916 [ 7759]: Freeze priority 2 2014/10/30 08:40:28.438971 [ 7759]: Freeze priority 3 2014/10/30 08:40:28.917467 [ 7759]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 08:40:32.447109 [recoverd: 8031]: server/ctdb_recoverd.c:3692 Current recmaster node 1 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/30 08:40:32.447491 [ 7759]: Freeze priority 1 2014/10/30 08:40:32.447619 [ 7759]: Freeze priority 2 2014/10/30 08:40:32.447687 [ 7759]: Freeze priority 3 2014/10/30 08:40:36.807698 [ 7759]: Freeze priority 1 2014/10/30 08:40:36.827438 [ 7759]: Freeze priority 2 2014/10/30 08:40:36.828699 [ 7759]: Freeze priority 3 2014/10/30 08:40:37.039072 [ 7759]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/30 08:40:37.043614 [ 7759]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/30 08:40:37.049247 [ 7759]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup 2014/10/30 08:40:38.708169 [ 7759]: Thawing priority 1 2014/10/30 08:40:38.708241 [ 7759]: Release freeze handler for prio 1 2014/10/30 08:40:38.708288 [ 7759]: Thawing priority 2 2014/10/30 08:40:38.708309 [ 7759]: Release freeze handler for prio 2 2014/10/30 08:40:38.708348 [ 7759]: Thawing priority 3 2014/10/30 08:40:38.708366 [ 7759]: Release freeze handler for prio 3 2014/10/30 08:40:53.491994 [recoverd: 8031]: Trigger takeoverrun 2014/10/30 08:40:53.641782 [ 7759]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/30 08:40:53.981692 [ 7759]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 08:40:53.994968 [ 7759]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 08:40:54.034498 [ 7759]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/30 08:40:54.213206 [ 7759]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/30 08:40:56.627598 [ 7759]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation 2014/10/30 08:40:57.544915 [ 7759]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 08:45:00.118554 [ 7759]: Freeze priority 1 2014/10/30 08:45:00.151997 [ 7759]: Freeze priority 1 2014/10/30 08:45:00.165676 [ 7759]: Freeze priority 2 2014/10/30 08:45:00.166043 [ 7759]: Freeze priority 2 2014/10/30 08:45:00.166668 [ 7759]: Freeze priority 3 2014/10/30 08:45:00.167604 [ 7759]: Freeze priority 3 2014/10/30 08:45:03.187457 [ 7759]: Freeze priority 1 2014/10/30 08:45:03.187852 [ 7759]: Freeze priority 2 2014/10/30 08:45:03.188121 [ 7759]: Freeze priority 3 2014/10/30 08:45:07.520410 [ 7759]: Thawing priority 1 2014/10/30 08:45:07.520468 [ 7759]: Release freeze handler for prio 1 2014/10/30 08:45:07.520501 [ 7759]: Thawing priority 2 2014/10/30 08:45:07.520538 [ 7759]: Release freeze handler for prio 2 2014/10/30 08:45:07.520584 [ 7759]: Thawing priority 3 2014/10/30 08:45:07.520603 [ 7759]: Release freeze handler for prio 3 2014/10/30 08:50:04.558334 [ 7759]: Freeze priority 1 2014/10/30 08:50:04.623344 [ 7759]: Freeze priority 2 2014/10/30 08:50:04.627513 [ 7759]: Freeze priority 3 2014/10/30 08:50:07.198975 [ 7759]: Thawing priority 1 2014/10/30 08:50:07.199019 [ 7759]: Release freeze handler for prio 1 2014/10/30 08:50:07.199053 [ 7759]: Thawing priority 2 2014/10/30 08:50:07.199071 [ 7759]: Release freeze handler for prio 2 2014/10/30 08:50:07.199108 [ 7759]: Thawing priority 3 2014/10/30 08:50:07.199127 [ 7759]: Release freeze handler for prio 3 2014/10/30 08:50:17.829321 [ 7759]: Freeze priority 1 2014/10/30 08:50:17.833224 [ 7759]: Freeze priority 2 2014/10/30 08:50:17.836824 [ 7759]: Freeze priority 3 2014/10/30 08:50:19.909921 [ 7759]: Thawing priority 1 2014/10/30 08:50:19.909955 [ 7759]: Release freeze handler for prio 1 2014/10/30 08:50:19.909984 [ 7759]: Thawing priority 2 2014/10/30 08:50:19.910002 [ 7759]: Release freeze handler for prio 2 2014/10/30 08:50:19.910028 [ 7759]: Thawing priority 3 2014/10/30 08:50:19.910045 [ 7759]: Release freeze handler for prio 3 2014/10/30 09:00:10.353603 [ 7759]: Freeze priority 1 2014/10/30 09:00:10.375699 [ 7759]: Freeze priority 2 2014/10/30 09:00:10.378329 [ 7759]: Freeze priority 3 2014/10/30 09:00:12.223107 [ 7759]: Thawing priority 1 2014/10/30 09:00:12.223145 [ 7759]: Release freeze handler for prio 1 2014/10/30 09:00:12.223174 [ 7759]: Thawing priority 2 2014/10/30 09:00:12.223191 [ 7759]: Release freeze handler for prio 2 2014/10/30 09:00:12.223216 [ 7759]: Thawing priority 3 2014/10/30 09:00:12.223231 [ 7759]: Release freeze handler for prio 3 2014/10/30 09:05:26.379123 [ 7759]: Freeze priority 1 2014/10/30 09:05:26.850015 [ 7759]: Freeze priority 2 2014/10/30 09:05:26.858254 [ 7759]: Freeze priority 3 2014/10/30 09:05:31.276777 [ 7759]: Thawing priority 1 2014/10/30 09:05:31.276837 [ 7759]: Release freeze handler for prio 1 2014/10/30 09:05:31.276892 [ 7759]: Thawing priority 2 2014/10/30 09:05:31.276913 [ 7759]: Release freeze handler for prio 2 2014/10/30 09:05:31.276941 [ 7759]: Thawing priority 3 2014/10/30 09:05:31.276959 [ 7759]: Release freeze handler for prio 3 2014/10/30 09:05:41.689545 [ 7759]: Freeze priority 1 2014/10/30 09:05:41.701314 [ 7759]: Freeze priority 2 2014/10/30 09:05:41.702487 [ 7759]: Freeze priority 3 2014/10/30 09:05:45.964701 [ 7759]: Thawing priority 1 2014/10/30 09:05:45.964745 [ 7759]: Release freeze handler for prio 1 2014/10/30 09:05:45.964781 [ 7759]: Thawing priority 2 2014/10/30 09:05:45.964802 [ 7759]: Release freeze handler for prio 2 2014/10/30 09:05:45.964831 [ 7759]: Thawing priority 3 2014/10/30 09:05:45.964850 [ 7759]: Release freeze handler for prio 3 2014/10/30 09:06:04.827134 [ 7759]: Monitoring event was cancelled 2014/10/30 09:10:17.338793 [ 7759]: Freeze priority 1 2014/10/30 09:10:17.343182 [ 7759]: Freeze priority 1 2014/10/30 09:10:17.345028 [ 7759]: Freeze priority 1 2014/10/30 09:10:17.347100 [ 7759]: Freeze priority 2 2014/10/30 09:10:17.347443 [ 7759]: Freeze priority 2 2014/10/30 09:10:17.348444 [ 7759]: Freeze priority 2 2014/10/30 09:10:17.348890 [ 7759]: Freeze priority 3 2014/10/30 09:10:17.349077 [ 7759]: Freeze priority 3 2014/10/30 09:10:17.349114 [ 7759]: Freeze priority 3 2014/10/30 09:10:20.358270 [recoverd: 8031]: Taking out recovery lock from recovery daemon 2014/10/30 09:10:20.358329 [recoverd: 8031]: Take the recovery lock 2014/10/30 09:10:20.370813 [ 7759]: Freeze priority 1 2014/10/30 09:10:20.371134 [ 7759]: Freeze priority 2 2014/10/30 09:10:20.371427 [ 7759]: Freeze priority 3 2014/10/30 09:10:22.084892 [ 7759]: Thawing priority 1 2014/10/30 09:10:22.084935 [ 7759]: Release freeze handler for prio 1 2014/10/30 09:10:22.084975 [ 7759]: Thawing priority 2 2014/10/30 09:10:22.084996 [ 7759]: Release freeze handler for prio 2 2014/10/30 09:10:22.085042 [ 7759]: Thawing priority 3 2014/10/30 09:10:22.085059 [ 7759]: Release freeze handler for prio 3 2014/10/30 09:10:22.086537 [ 7759]: server/ctdb_call.c:1005 reqid 50535 not found 2014/10/30 09:10:22.086601 [ 7759]: server/ctdb_call.c:1005 reqid 50536 not found 2014/10/30 09:10:22.617950 [recoverd: 8031]: Resetting ban count to 0 for all nodes 2014/10/30 09:17:21.218849 [recoverd: 8031]: Taking out recovery lock from recovery daemon 2014/10/30 09:17:21.218902 [recoverd: 8031]: Take the recovery lock 2014/10/30 09:17:21.315009 [ 7759]: Freeze priority 1 2014/10/30 09:17:21.327635 [ 7759]: Freeze priority 2 2014/10/30 09:17:21.329597 [ 7759]: Freeze priority 3 2014/10/30 09:17:23.213696 [ 7759]: Thawing priority 1 2014/10/30 09:17:23.213739 [ 7759]: Release freeze handler for prio 1 2014/10/30 09:17:23.213778 [ 7759]: Thawing priority 2 2014/10/30 09:17:23.213801 [ 7759]: Release freeze handler for prio 2 2014/10/30 09:17:23.213846 [ 7759]: Thawing priority 3 2014/10/30 09:17:23.213865 [ 7759]: Release freeze handler for prio 3 2014/10/30 09:17:23.539615 [recoverd: 8031]: Resetting ban count to 0 for all nodes 2014/10/30 09:17:33.749489 [recoverd: 8031]: server/ctdb_recoverd.c:3933 Remote node:1 has different flags for node 0. It has 0x02 vs our 0x00 2014/10/30 09:17:33.749545 [recoverd: 8031]: Use flags 0x00 from local recmaster node for cluster update of node 0 flags 2014/10/30 09:17:33.753336 [recoverd: 8031]: Taking out recovery lock from recovery daemon 2014/10/30 09:17:33.753368 [recoverd: 8031]: Take the recovery lock 2014/10/30 09:17:33.882711 [ 7759]: Freeze priority 1 2014/10/30 09:17:33.890523 [ 7759]: Freeze priority 2 2014/10/30 09:17:33.898881 [ 7759]: Freeze priority 3 2014/10/30 09:17:35.654818 [ 7759]: Thawing priority 1 2014/10/30 09:17:35.654916 [ 7759]: Release freeze handler for prio 1 2014/10/30 09:17:35.654984 [ 7759]: Thawing priority 2 2014/10/30 09:17:35.655005 [ 7759]: Release freeze handler for prio 2 2014/10/30 09:17:35.655034 [ 7759]: Thawing priority 3 2014/10/30 09:17:35.655051 [ 7759]: Release freeze handler for prio 3 2014/10/30 09:17:36.066372 [recoverd: 8031]: Resetting ban count to 0 for all nodes 2014/10/30 09:20:22.995568 [ 7759]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 09:20:23.008057 [ 7759]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 09:21:47.278429 [ 1518]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/30 09:21:47.278529 [ 1518]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 09:21:47.278543 [ 1518]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 09:25:48.759922 [ 8158]: Starting CTDBD (Version 2.5.3) as PID: 8158 2014/10/30 09:25:50.191843 [ 8158]: Vacuuming is disabled for persistent database registry.tdb 2014/10/30 09:25:50.215412 [ 8158]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/30 09:25:50.229591 [ 8158]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/30 09:25:50.243583 [ 8158]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/30 09:25:50.243601 [ 8158]: Ignoring persistent database 'account_policy.tdb.1' 2014/10/30 09:25:50.243610 [ 8158]: Ignoring persistent database 'ctdb.tdb.1' 2014/10/30 09:25:50.243619 [ 8158]: Ignoring persistent database 'group_mapping.tdb.1' 2014/10/30 09:25:50.243627 [ 8158]: Ignoring persistent database 'secrets.tdb.1' 2014/10/30 09:25:50.243635 [ 8158]: Ignoring persistent database 'share_info.tdb.1' 2014/10/30 09:25:50.257675 [ 8158]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/30 09:25:50.271652 [ 8158]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/30 09:25:50.271671 [ 8158]: Ignoring persistent database 'passdb.tdb.1' 2014/10/30 09:25:50.271680 [ 8158]: Ignoring persistent database 'registry.tdb.1' 2014/10/30 09:25:50.285592 [ 8158]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/30 09:25:50.285625 [ 8158]: Freeze priority 1 2014/10/30 09:25:50.303790 [ 8158]: Freeze priority 2 2014/10/30 09:25:50.304142 [ 8158]: Freeze priority 3 2014/10/30 09:25:50.468063 [ 8158]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/30 09:25:50.471820 [ 8158]: 00.ctdb: Set RecoverTimeout to 60 2014/10/30 09:25:50.475348 [ 8158]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/30 09:25:50.594568 [ 8158]: Freeze priority 1 2014/10/30 09:25:50.594659 [ 8158]: Freeze priority 2 2014/10/30 09:25:50.594732 [ 8158]: Freeze priority 3 2014/10/30 09:25:50.832826 [ 8158]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 09:25:50.937223 [ 8158]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 09:25:50.943320 [ 8158]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 09:25:54.602479 [recoverd: 8440]: server/ctdb_recoverd.c:3692 Current recmaster node 3 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/30 09:25:54.602559 [ 8158]: Freeze priority 1 2014/10/30 09:25:54.602621 [ 8158]: Freeze priority 2 2014/10/30 09:25:54.602685 [ 8158]: Freeze priority 3 2014/10/30 09:25:58.699242 [ 8158]: Freeze priority 1 2014/10/30 09:25:58.707178 [ 8158]: Freeze priority 2 2014/10/30 09:25:58.708009 [ 8158]: Freeze priority 3 2014/10/30 09:25:58.865421 [ 8158]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/30 09:25:58.866126 [ 8158]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/30 09:25:58.866957 [ 8158]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup 2014/10/30 09:26:01.408850 [ 8158]: Thawing priority 1 2014/10/30 09:26:01.408926 [ 8158]: Release freeze handler for prio 1 2014/10/30 09:26:01.408968 [ 8158]: Thawing priority 2 2014/10/30 09:26:01.409003 [ 8158]: Release freeze handler for prio 2 2014/10/30 09:26:01.409029 [ 8158]: Thawing priority 3 2014/10/30 09:26:01.409046 [ 8158]: Release freeze handler for prio 3 2014/10/30 09:26:16.404396 [ 8158]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/30 09:26:16.436195 [recoverd: 8440]: Trigger takeoverrun 2014/10/30 09:26:16.778567 [ 8158]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 09:26:16.791969 [ 8158]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 09:26:16.820562 [ 8158]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/30 09:26:17.003659 [ 8158]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/30 09:26:19.472638 [ 8158]: Node became HEALTHY. Ask recovery master 3 to perform ip reallocation 2014/10/30 09:26:20.387370 [ 8158]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 09:30:24.770985 [ 8158]: Freeze priority 1 2014/10/30 09:30:24.792197 [ 8158]: Freeze priority 1 2014/10/30 09:30:24.795917 [ 8158]: Freeze priority 1 2014/10/30 09:30:24.840950 [ 8158]: Freeze priority 2 2014/10/30 09:30:24.841073 [ 8158]: Freeze priority 2 2014/10/30 09:30:24.841110 [ 8158]: Freeze priority 2 2014/10/30 09:30:24.841784 [ 8158]: Freeze priority 3 2014/10/30 09:30:24.841917 [ 8158]: Freeze priority 3 2014/10/30 09:30:24.841965 [ 8158]: Freeze priority 3 2014/10/30 09:30:27.859902 [ 8158]: Freeze priority 1 2014/10/30 09:30:27.860211 [ 8158]: Freeze priority 2 2014/10/30 09:30:27.860422 [ 8158]: Freeze priority 3 2014/10/30 09:30:30.816961 [ 8158]: Thawing priority 1 2014/10/30 09:30:30.817006 [ 8158]: Release freeze handler for prio 1 2014/10/30 09:30:30.817035 [ 8158]: Thawing priority 2 2014/10/30 09:30:30.817054 [ 8158]: Release freeze handler for prio 2 2014/10/30 09:30:30.817082 [ 8158]: Thawing priority 3 2014/10/30 09:30:30.817101 [ 8158]: Release freeze handler for prio 3 2014/10/30 09:35:35.853417 [ 8158]: Freeze priority 1 2014/10/30 09:35:35.955426 [ 8158]: Freeze priority 2 2014/10/30 09:35:35.956890 [ 8158]: Freeze priority 3 2014/10/30 09:35:39.191937 [ 8158]: Thawing priority 1 2014/10/30 09:35:39.191973 [ 8158]: Release freeze handler for prio 1 2014/10/30 09:35:39.192007 [ 8158]: Thawing priority 2 2014/10/30 09:35:39.192028 [ 8158]: Release freeze handler for prio 2 2014/10/30 09:35:39.192061 [ 8158]: Thawing priority 3 2014/10/30 09:35:39.192081 [ 8158]: Release freeze handler for prio 3 2014/10/30 09:45:34.038341 [ 8158]: Freeze priority 1 2014/10/30 09:45:34.162299 [ 8158]: Freeze priority 2 2014/10/30 09:45:34.163782 [ 8158]: Freeze priority 3 2014/10/30 09:45:38.086931 [ 8158]: Freeze priority 1 2014/10/30 09:45:38.089472 [ 8158]: Freeze priority 2 2014/10/30 09:45:38.092109 [ 8158]: Freeze priority 3 2014/10/30 09:45:42.002518 [ 8158]: Thawing priority 1 2014/10/30 09:45:42.002566 [ 8158]: Release freeze handler for prio 1 2014/10/30 09:45:42.002597 [ 8158]: Thawing priority 2 2014/10/30 09:45:42.002617 [ 8158]: Release freeze handler for prio 2 2014/10/30 09:45:42.002657 [ 8158]: Thawing priority 3 2014/10/30 09:45:42.002679 [ 8158]: Release freeze handler for prio 3 2014/10/30 09:50:51.431212 [recoverd: 8440]: Taking out recovery lock from recovery daemon 2014/10/30 09:50:51.431265 [recoverd: 8440]: Take the recovery lock 2014/10/30 09:50:51.494324 [ 8158]: Freeze priority 1 2014/10/30 09:50:51.595151 [ 8158]: Freeze priority 2 2014/10/30 09:50:51.597727 [ 8158]: Freeze priority 3 2014/10/30 09:50:55.750899 [ 8158]: Thawing priority 1 2014/10/30 09:50:55.750959 [ 8158]: Release freeze handler for prio 1 2014/10/30 09:50:55.751019 [ 8158]: Thawing priority 2 2014/10/30 09:50:55.751043 [ 8158]: Release freeze handler for prio 2 2014/10/30 09:50:55.751079 [ 8158]: Thawing priority 3 2014/10/30 09:50:55.751116 [ 8158]: Release freeze handler for prio 3 2014/10/30 09:50:56.150909 [recoverd: 8440]: Resetting ban count to 0 for all nodes 2014/10/30 09:53:37.381277 [recoverd: 8440]: server/ctdb_recoverd.c:3960 The vnnmap count is different from the number of active lmaster nodes: 4 vs 3 2014/10/30 09:53:37.381348 [recoverd: 8440]: Taking out recovery lock from recovery daemon 2014/10/30 09:53:37.381362 [recoverd: 8440]: Take the recovery lock 2014/10/30 09:53:37.391351 [ 8158]: Freeze priority 1 2014/10/30 09:53:37.418153 [ 8158]: Freeze priority 2 2014/10/30 09:53:37.419106 [ 8158]: Freeze priority 3 2014/10/30 09:53:39.741314 [ 8158]: Thawing priority 1 2014/10/30 09:53:39.741354 [ 8158]: Release freeze handler for prio 1 2014/10/30 09:53:39.741394 [ 8158]: Thawing priority 2 2014/10/30 09:53:39.741428 [ 8158]: Release freeze handler for prio 2 2014/10/30 09:53:39.741469 [ 8158]: Thawing priority 3 2014/10/30 09:53:39.741490 [ 8158]: Release freeze handler for prio 3 2014/10/30 09:53:39.742528 [ 8158]: server/ctdb_call.c:1005 reqid 52646 not found 2014/10/30 09:53:39.742582 [ 8158]: server/ctdb_call.c:1005 reqid 52648 not found 2014/10/30 09:53:39.743615 [ 8158]: pnn 2 Invalid reqid 52645 in ctdb_become_dmaster from node 1 2014/10/30 09:53:39.743675 [ 8158]: server/ctdb_call.c:1005 reqid 52647 not found 2014/10/30 09:53:40.204678 [recoverd: 8440]: Resetting ban count to 0 for all nodes 2014/10/30 09:58:53.462851 [ 8158]: Freeze priority 1 2014/10/30 09:58:53.618452 [ 8158]: Freeze priority 2 2014/10/30 09:58:53.621452 [ 8158]: Freeze priority 3 2014/10/30 09:58:53.627339 [ 8158]: Monitoring event was cancelled 2014/10/30 09:58:53.627371 [ 8158]: server/eventscript.c:569 Sending SIGTERM to child pid:18191 2014/10/30 09:58:58.639383 [ 8158]: Thawing priority 1 2014/10/30 09:58:58.639427 [ 8158]: Release freeze handler for prio 1 2014/10/30 09:58:58.639462 [ 8158]: Thawing priority 2 2014/10/30 09:58:58.639480 [ 8158]: Release freeze handler for prio 2 2014/10/30 09:58:58.639509 [ 8158]: Thawing priority 3 2014/10/30 09:58:58.639526 [ 8158]: Release freeze handler for prio 3 2014/10/30 09:59:09.191231 [ 8158]: Freeze priority 1 2014/10/30 09:59:09.256792 [ 8158]: Freeze priority 2 2014/10/30 09:59:09.260339 [ 8158]: Freeze priority 3 2014/10/30 09:59:09.264789 [ 8158]: Monitoring event was cancelled 2014/10/30 09:59:09.264824 [ 8158]: server/eventscript.c:569 Sending SIGTERM to child pid:19107 2014/10/30 09:59:13.953342 [ 8158]: Thawing priority 1 2014/10/30 09:59:13.953397 [ 8158]: Release freeze handler for prio 1 2014/10/30 09:59:13.953443 [ 8158]: Thawing priority 2 2014/10/30 09:59:13.953462 [ 8158]: Release freeze handler for prio 2 2014/10/30 09:59:13.953498 [ 8158]: Thawing priority 3 2014/10/30 09:59:13.953514 [ 8158]: Release freeze handler for prio 3 2014/10/30 10:11:08.019433 [ 8158]: 60.nfs: ERROR: rquotad failed RPC check: 2014/10/30 10:11:08.046885 [ 8158]: 60.nfs: rpcinfo: RPC: Program not registered 2014/10/30 10:11:08.046917 [ 8158]: 60.nfs: program 100011 version 1 is not available 2014/10/30 10:11:08.046964 [ 8158]: 60.nfs: Trying to restart rquotad [rpc.rquotad] 2014/10/30 10:11:14.716921 [ 8158]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 10:11:14.729115 [ 8158]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 10:14:05.724484 [ 1502]: Recovery lock file set to "". Disabling recovery lock checking 2014/10/30 10:14:05.724603 [ 1502]: ctdb error: Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 10:14:05.724618 [ 1502]: ctdb_set_nlist failed - Failed to load nlist '/etc/ctdb/nodes' 2014/10/30 10:18:05.538175 [ 7589]: Starting CTDBD (Version 2.5.3) as PID: 7589 2014/10/30 10:18:07.378398 [ 7589]: Vacuuming is disabled for persistent database registry.tdb 2014/10/30 10:18:07.401642 [ 7589]: Vacuuming is disabled for persistent database passdb.tdb 2014/10/30 10:18:07.415797 [ 7589]: Vacuuming is disabled for persistent database secrets.tdb 2014/10/30 10:18:07.429977 [ 7589]: Vacuuming is disabled for persistent database share_info.tdb 2014/10/30 10:18:07.429996 [ 7589]: Ignoring persistent database 'account_policy.tdb.1' 2014/10/30 10:18:07.430006 [ 7589]: Ignoring persistent database 'ctdb.tdb.1' 2014/10/30 10:18:07.430015 [ 7589]: Ignoring persistent database 'group_mapping.tdb.1' 2014/10/30 10:18:07.430024 [ 7589]: Ignoring persistent database 'secrets.tdb.1' 2014/10/30 10:18:07.430033 [ 7589]: Ignoring persistent database 'share_info.tdb.1' 2014/10/30 10:18:07.444550 [ 7589]: Vacuuming is disabled for persistent database ctdb.tdb 2014/10/30 10:18:07.459083 [ 7589]: Vacuuming is disabled for persistent database account_policy.tdb 2014/10/30 10:18:07.459103 [ 7589]: Ignoring persistent database 'passdb.tdb.1' 2014/10/30 10:18:07.459112 [ 7589]: Ignoring persistent database 'registry.tdb.1' 2014/10/30 10:18:07.473599 [ 7589]: Vacuuming is disabled for persistent database group_mapping.tdb 2014/10/30 10:18:07.473642 [ 7589]: Freeze priority 1 2014/10/30 10:18:07.504497 [ 7589]: Freeze priority 2 2014/10/30 10:18:07.504916 [ 7589]: Freeze priority 3 2014/10/30 10:18:07.679590 [ 7589]: 00.ctdb: Set EventScriptTimeout to 60 2014/10/30 10:18:07.683268 [ 7589]: 00.ctdb: Set RecoverTimeout to 60 2014/10/30 10:18:07.686581 [ 7589]: 00.ctdb: Set RecoveryBanPeriod to 30 2014/10/30 10:18:07.813379 [ 7589]: Freeze priority 1 2014/10/30 10:18:07.813455 [ 7589]: Freeze priority 2 2014/10/30 10:18:07.813508 [ 7589]: Freeze priority 3 2014/10/30 10:18:08.227865 [ 7589]: Unknown db_id 0xaf029e9d in ctdb_ltdb_update_seqnum 2014/10/30 10:18:11.820204 [recoverd: 7872]: server/ctdb_recoverd.c:3692 Current recmaster node 1 does not have CAP_RECMASTER, but we (node 2) have - force an election 2014/10/30 10:18:11.820284 [ 7589]: Freeze priority 1 2014/10/30 10:18:11.820344 [ 7589]: Freeze priority 2 2014/10/30 10:18:11.820394 [ 7589]: Freeze priority 3 2014/10/30 10:18:16.150196 [ 7589]: Freeze priority 1 2014/10/30 10:18:16.161389 [ 7589]: Freeze priority 2 2014/10/30 10:18:16.162263 [ 7589]: Freeze priority 3 2014/10/30 10:18:16.297203 [ 7589]: server/ctdb_monitor.c:495 Node 0 became healthy - force recovery for startup 2014/10/30 10:18:16.297738 [ 7589]: server/ctdb_monitor.c:495 Node 1 became healthy - force recovery for startup 2014/10/30 10:18:16.298502 [ 7589]: server/ctdb_monitor.c:495 Node 3 became healthy - force recovery for startup 2014/10/30 10:18:19.087828 [ 7589]: Thawing priority 1 2014/10/30 10:18:19.087875 [ 7589]: Release freeze handler for prio 1 2014/10/30 10:18:19.087909 [ 7589]: Thawing priority 2 2014/10/30 10:18:19.087929 [ 7589]: Release freeze handler for prio 2 2014/10/30 10:18:19.087972 [ 7589]: Thawing priority 3 2014/10/30 10:18:19.087988 [ 7589]: Release freeze handler for prio 3 2014/10/30 10:18:34.032164 [ 7589]: 50.samba: Redirecting to /bin/systemctl start smb.service 2014/10/30 10:18:34.114149 [recoverd: 7872]: Trigger takeoverrun 2014/10/30 10:18:34.347415 [ 7589]: 60.nfs: Stopping nfslock (via systemctl): [ OK ] 2014/10/30 10:18:34.360712 [ 7589]: 60.nfs: Redirecting to /bin/systemctl stop nfs.service 2014/10/30 10:18:34.389691 [ 7589]: 60.nfs: Redirecting to /bin/systemctl start nfs.service 2014/10/30 10:18:34.614431 [ 7589]: 60.nfs: Starting nfslock (via systemctl): [ OK ] 2014/10/30 10:18:37.023882 [ 7589]: Node became HEALTHY. Ask recovery master 1 to perform ip reallocation 2014/10/30 10:18:37.904380 [ 7589]: 60.nfs: Reconfiguring service "nfs"... 2014/10/30 10:28:30.805726 [ 7589]: Freeze priority 1 2014/10/30 10:28:31.291855 [ 7589]: Freeze priority 1 2014/10/30 10:28:31.309339 [ 7589]: Freeze priority 2 2014/10/30 10:28:31.310567 [ 7589]: Freeze priority 3 2014/10/30 10:28:31.817334 [recoverd: 7872]: server/ctdb_recoverd.c:2343 Reload nodes file from recovery daemon 2014/10/30 10:28:32.243238 [ 7589]: Freeze priority 1 2014/10/30 10:28:32.243710 [ 7589]: Freeze priority 2 2014/10/30 10:28:32.244090 [ 7589]: Freeze priority 3 2014/10/30 10:28:33.974842 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0 2014/10/30 10:28:33.974906 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0 2014/10/30 10:28:34.137396 [ 7589]: Thawing priority 1 2014/10/30 10:28:34.137441 [ 7589]: Release freeze handler for prio 1 2014/10/30 10:28:34.137476 [ 7589]: Thawing priority 2 2014/10/30 10:28:34.137499 [ 7589]: Release freeze handler for prio 2 2014/10/30 10:28:34.137530 [ 7589]: Thawing priority 3 2014/10/30 10:28:34.137551 [ 7589]: Release freeze handler for prio 3 2014/10/30 10:28:56.156817 [ 7589]: Refused connection from unknown node 10.192.55.182 2014/10/30 10:28:59.159370 [ 7589]: Refused connection from unknown node 10.192.55.182 2014/10/30 10:29:02.161617 [ 7589]: Refused connection from unknown node 10.192.55.182 2014/10/30 10:29:05.162924 [ 7589]: Refused connection from unknown node 10.192.55.182 2014/10/30 10:29:08.164475 [ 7589]: Refused connection from unknown node 10.192.55.182 2014/10/30 10:29:11.168203 [ 7589]: Refused connection from unknown node 10.192.55.182 2014/10/30 10:29:14.170780 [ 7589]: Refused connection from unknown node 10.192.55.182 2014/10/30 10:29:17.171837 [ 7589]: Refused connection from unknown node 10.192.55.182 2014/10/30 10:29:20.174829 [ 7589]: Refused connection from unknown node 10.192.55.182 2014/10/30 10:29:23.177413 [ 7589]: Refused connection from unknown node 10.192.55.182 2014/10/30 10:29:26.178249 [ 7589]: Refused connection from unknown node 10.192.55.182 2014/10/30 10:29:29.179542 [ 7589]: Refused connection from unknown node 10.192.55.182 2014/10/30 10:39:35.274922 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0 2014/10/30 10:39:35.275819 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0 2014/10/30 10:39:35.277700 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0 2014/10/30 10:39:35.278954 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0 2014/10/30 10:39:35.280102 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0 2014/10/30 10:39:35.280431 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0 2014/10/30 10:52:35.989398 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0 2014/10/30 10:52:35.992143 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0 2014/10/30 10:52:36.000357 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0 2014/10/30 10:52:36.004212 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0 2014/10/30 10:52:36.006268 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0 2014/10/30 10:52:36.006639 [ 7589]: server/ctdb_server.c:554 Can not queue packet to DELETED node 0