Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SIGABRT When Polling Begins #44

Closed
jalcine opened this issue Jan 13, 2015 · 6 comments
Closed

SIGABRT When Polling Begins #44

jalcine opened this issue Jan 13, 2015 · 6 comments
Assignees
@jalcine
Labels
bug core plugins
Milestone
v0.1.x

Comments

@jalcine
Copy link
Owner

jalcine commented Jan 13, 2015

What: Once the loop is started from the test plugin-unit-zmq_tunnel, the
test segfaults with SIGABRT.

How: Build up the system (use Vagrant) and run the test
plugin-unit-zmq_tunnel (lives under bin/plugin-unit-zmq_tunnel).

Output

[wntr] Test suite starting...
Running cxxtest tests (1 test)0 [0x7ffff7fc37c0] INFO root null - Started logging session at Tue Jan 13 06:39:39 2015

88 [0x7ffff7fc37c0] DEBUG root null - 1.2.0
[New Thread 0x7ffff4932700 (LWP 2664)]
[New Thread 0x7ffff4131700 (LWP 2665)]
plugin-unit-zmq_tunnel: src/wintermutecore/event_poller.cpp:27: void wintermute_event_poller_callback(uv_poll_t*, int, int): Assertion `status' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff6a03e37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.

Backtrace

  #0  0x00007ffff6a03e37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
  #1  0x00007ffff6a05528 in __GI_abort () at abort.c:89
  #2  0x00007ffff69fcce6 in __assert_fail_base (fmt=0x7ffff6b4d788 "%s%s%"..., assertion=assertion@entry=0x7ffff78edfb7 "statu"...,
        file=file@entry=0x7ffff78edf60 "/home"..., line=line@entry=27,
        function=function@entry=0x7ffff78ee160 <wintermute_event_poller_callback::__PRETTY_FUNCTION__> "void "...) at assert.c:92
  #3  0x00007ffff69fcd92 in __GI___assert_fail (assertion=0x7ffff78edfb7 "statu"..., file=0x7ffff78edf60 "/home"..., line=27,
        function=0x7ffff78ee160 <wintermute_event_poller_callback::__PRETTY_FUNCTION__> "void "...) at assert.c:101
  #4  0x00007ffff78d4271 in wintermute_event_poller_callback (handle=0x7ffff7de9557 <_dl_fixup+247 at ../elf/dl-runtime.c:111>, status=32767, events=-27920)
        at src/wintermutecore/event_poller.cpp:27
  #5  0x00007ffff7bcf71a in uv__io_poll (loop=loop@entry=0x7ffff7dd9bc0 <default_loop_struct>, timeout=45) at src/unix/linux-core.c:300
  #6  0x00007ffff7bc2e47 in uv_run (loop=0x7ffff7dd9bc0 <default_loop_struct>, mode=UV_RUN_DEFAULT) at src/unix/core.c:324
  #7  0x00007ffff78c5ff1 in Wintermute::Events::Loop::run (this=0x67e2b0) at src/wintermutecore/event_loop.cpp:49
  #8  0x0000000000432fee in ZMQTunnelTestSuite::testSendOutMessage (this=0x7ffff7ffe1c8)
        at src/wintermute-transport-zeromq/test/tunnel.hh:165
  #9  0x0000000000434932 in TestDescription_suite_ZMQTunnelTestSuite_testSendOutMessage::runTest (
        this=0x65ae20 <testDescription_suite_ZMQTunnelTestSuite_testSendOutMessage>)
        at build/src/wintermute-transport-zeromq/test/plugin-unit-zmq_tunnel_test.cc:38
  #10 0x000000000042226e in CxxTest::RealTestDescription::run (this=0x3c422634c4265d00) at /usr/include/cxxtest/RealDescriptions.cpp:121
  #11 0x000000000042964d in CxxTest::TestRunner::runTest (this=0x3c422634c4265d00, td=...) at /usr/include/cxxtest/TestRunner.h:103
  #12 0x0000000000429345 in CxxTest::TestRunner::runSuite (this=0x65ad80 <suiteDescription_ZMQTunnelTestSuite>, sd=...) at /usr/include/cxxtest/TestRunner.h:87
  #13 0x0000000000428f95 in CxxTest::TestRunner::runWorld (this=0x65ad80 <suiteDescription_ZMQTunnelTestSuite>) at /usr/include/cxxtest/TestRunner.h:67
  #14 0x0000000000428bc6 in CxxTest::TestRunner::runAllTests (listener=...) at /usr/include/cxxtest/TestRunner.h:40
  #15 0x0000000000429bc0 in CxxTest::ErrorFormatter::run (this=0x7fffffffc860) at /usr/include/cxxtest/ErrorFormatter.h:63
  #16 0x000000000041e672 in main () at build/src/wintermute-transport-zeromq/test/plugin-unit-zmq_tunnel_test.cc:22
@jalcine jalcine added this to the v0.1.x milestone Jan 13, 2015
@jalcine
Copy link
Owner Author

jalcine commented Jan 13, 2015

This relates to blocks issue #38 (implementing the ZeroMQ plugin for Wintermute).

@jalcine
Copy link
Owner Author

jalcine commented Jan 16, 2015

So, I'm not even sure how this happens. But it is related to how libuv is
handling polling (note #5) in the backtrace.

@jalcine jalcine self-assigned this Jan 16, 2015
@jalcine
Copy link
Owner Author

jalcine commented Jan 17, 2015

Finally found the culprit. Somehow during the constructon of the Poller for
the ZMQRecevier, the initialization arguments get garbled up, i.e: set to
maximum values.

(gdb) frame
Stack level 0, frame at 0x7fffffffbd00:
 rip = 0x7ffff7af2c88 in Wintermute::Events::Poller::Poller (src/wintermutecore/event_poller.cpp:66); saved rip = 0x7ffff772b7d4
 called by frame at 0x7fffffffbd80
 source language c++.
 Arglist at 0x7fffffffbcf0, args: this=0x7ffff796e590, aFd=32767, aDirection=4294951312, loopPtr=...
 Locals at 0x7fffffffbcf0, Previous frame's sp is 0x7fffffffbd00
 Saved registers:
  rbx at 0x7fffffffbce0, rbp at 0x7fffffffbcf0, r12 at 0x7fffffffbce8, rip at 0x7fffffffbcf8
this = 0x7ffff796e590
aFd = 32767
aDirection = 4294951312
loopPtr = @0x677850: {
  <__shared_ptr> = {
    _M_ptr = 0x7ffff7d37690 <vtable for Wintermute::Events::Poller+16>,
    _M_refcount = {
      _M_pi = 0x0
    }
  }, <No data fields>}
d = 0x0
d = 0x0
61
62        W_PRV(Poller);
63        d->q_ptr = make_shared<Poller>(*this);
64        d->emitter = make_shared<Emitter>(loopPtr);
65        d->handle.reset(new uv_poll_t);
66        d->fd = aFd;
67        d->loop = loopPtr;
68        d->direction = aDirection;
69        assert(d->handle);
70
(gdb)

Note the values of aFd and aDirection.

@jalcine
Copy link
Owner Author

jalcine commented Jan 17, 2015

Looks like it was a bit more complicated than that. An invalid reference to a
ZMQReceiver is started, thus forming the funky Poller that goes off. This
is magic, man.

(gdb) bt
#0  Wintermute::Events::Poller::start (this=0x100000000) at /home/jalcine/Development/Projects/Wintermute/core/src/wintermutecore/event_poller.cpp:115
#1  0x00007ffff7725142 in Wintermute::ZMQReceiver::attachPoller (this=0x13f7938d98) at /home/jalcine/Development/Projects/Wintermute/core/src/wintermute-transport-zeromq/receiver.cpp:118
#2  0x00007ffff7726610 in Wintermute::ZMQReceiver::start (this=0x6719b0) at /home/jalcine/Development/Projects/Wintermute/core/src/wintermute-transport-zeromq/receiver.cpp:151
#3  0x00007ffff7aa3141 in Wintermute::Tunnel::start()::{lambda(std::pair<std::string const, std::shared_ptr<Wintermute::Tunnel::Receiver> >)#2}::operator()(std::pair<std::string const, std::shared_ptr<Wintermute::Tunnel::Receiver> >) const (__closure=0x7fffffffc300, pair=...) at /home/jalcine/Development/Projects/Wintermute/core/src/wintermutecore/tunnel.cpp:181
#4  0x00007ffff7aa39ef in Wintermute::Tunnel::start()::{lambda(std::pair<std::string const, std::shared_ptr<Wintermute::Tunnel::Receiver> >)#2} std::for_each<std::__detail::_Node_iterator<std::pair<std::string const, std::shared_ptr<Wintermute::Tunnel::Receiver> >, false, true>, Wintermute::Tunnel::start()::{lambda(std::pair<std::string const, std::shared_ptr<Wintermute::Tunnel::Receiver> >)#2}>(std::__detail::_Node_iterator<std::pair<std::string const, std::shared_ptr<Wintermute::Tunnel::Receiver> >, false, true>, Wintermute::Tunnel::start()::{lambda(std::pair<std::string const, std::shared_ptr<Wintermute::Tunnel::Receiver> >)#2}, Wintermute::Tunnel::start()::{lambda(std::pair<std::string const, std::shared_ptr<Wintermute::Tunnel::Receiver> >)#2}) (__first=..., __last=..., __f=...) at /usr/include/c++/4.9/bits/stl_algo.h:3755
#5  0x00007ffff7aa36aa in Wintermute::Tunnel::start () at /home/jalcine/Development/Projects/Wintermute/core/src/wintermutecore/tunnel.cpp:187
#6  0x00000000004341f5 in ZMQTunnelTestSuite::testSendOutMessage (this=0x7ffff7de4991 <_dl_lookup_symbol_x+305 at dl-lookup.c:740>) at /home/jalcine/Development/Projects/Wintermute/core/src/wintermute-transport-zeromq/test/tunnel.hh:177
#7  0x000000000043621e in TestDescription_suite_ZMQTunnelTestSuite_testSendOutMessage::runTest (this=0x65d4a0 <testDescription_suite_ZMQTunnelTestSuite_testSendOutMessage>) at /home/jalcine/Development/Projects/Wintermute/core/build/src/wintermute-transport-zeromq/test/plugin-unit-zmq_tunnel_test.cc:38
#8  0x000000000042333e in CxxTest::RealTestDescription::run (this=0x677c3b2bd02f8e00) at /usr/include/cxxtest/RealDescriptions.cpp:121
#9  0x000000000042a749 in CxxTest::TestRunner::runTest (this=0x677c3b2bd02f8e00, td=...) at /usr/include/cxxtest/TestRunner.h:103
#10 0x000000000042a441 in CxxTest::TestRunner::runSuite (this=0x65d400 <suiteDescription_ZMQTunnelTestSuite>, sd=...) at /usr/include/cxxtest/TestRunner.h:87
#11 0x000000000042a091 in CxxTest::TestRunner::runWorld (this=0x65d400 <suiteDescription_ZMQTunnelTestSuite>) at /usr/include/cxxtest/TestRunner.h:67
#12 0x0000000000429cc2 in CxxTest::TestRunner::runAllTests (listener=...) at /usr/include/cxxtest/TestRunner.h:40
#13 0x000000000042acbc in CxxTest::ErrorFormatter::run (this=0x7fffffffc860) at /usr/include/cxxtest/ErrorFormatter.h:63
#14 0x000000000041f742 in main () at /home/jalcine/Development/Projects/Wintermute/core/build/src/wintermute-transport-zeromq/test/plugin-unit-zmq_tunnel_test.cc:22
(gdb) up 3
#3  0x00007ffff7aa3141 in Wintermute::Tunnel::start()::{lambda(std::pair<std::string const, std::shared_ptr<Wintermute::Tunnel::Receiver> >)#2}::operator()(std::pair<std::string const, std::shared_ptr<Wintermute::Tunnel::Receiver> >) const (__closure=0x7fffffffc300, pair=...) at /home/jalcine/Development/Projects/Wintermute/core/src/wintermutecore/tunnel.cpp:181
181         pair.second->start();
(gdb) print pair
$95 = (pair &) @0x671bb8: {
  first = 0x6719a0,
  second = 0xe912d9786c8ba3b8
}

@jalcine jalcine changed the title SIGABRT when Polling Begins SIGABRT When Polling Begins Jan 18, 2015
@jalcine
Copy link
Owner Author

jalcine commented Jan 18, 2015

Doing some more 'detective' work led to my discovery of this:

Breakpoint 3, Wintermute::ZMQReceiver::connectToClients (this=0x0) at src/wintermute-transport-zeromq/receiver.cpp:65
65        auto socketType = zmqpp::socket_type::subscribe;
(gdb) bt
#0  Wintermute::ZMQReceiver::connectToClients (this=0x0) at src/wintermute-transport-zeromq/receiver.cpp:65
#1  0x00007ffff772518a in Wintermute::ZMQReceiver::start (this=0x671960) at src/wintermute-transport-zeromq/receiver.cpp:159
#2  0x00007ffff7aa52b6 in Wintermute::Tunnel::start()::{lambda(std::pair<std::string const, std::shared_ptr<Wintermute::Tunnel::Receiver> >)#2}::operator()(std::pair<std::string const, std::shared_ptr<Wintermute::Tunnel::Receiver> >) const (__closure=0x7fffffffc280, pair=...) at src/wintermutecore/tunnel.cpp:182
#3  0x00007ffff7aa5b63 in Wintermute::Tunnel::start()::{lambda(std::pair<std::string const, std::shared_ptr<Wintermute::Tunnel::Receiver> >)#2} std::for_each<std::__detail::_Node_iterator<std::pair<std::string const, std::shared_ptr<Wintermute::Tunnel::Receiver> >, false, true>, Wintermute::Tunnel::start()::{lambda(std::pair<std::string const, std::shared_ptr<Wintermute::Tunnel::Receiver> >)#2}>(std::__detail::_Node_iterator<std::pair<std::string const, std::shared_ptr<Wintermute::Tunnel::Receiver> >, false, true>, Wintermute::Tunnel::start()::{lambda(std::pair<std::string const, std::shared_ptr<Wintermute::Tunnel::Receiver> >)#2}, Wintermute::Tunnel::start()::{lambda(std::pair<std::string const, std::shared_ptr<Wintermute::Tunnel::Receiver> >)#2}) (__first=..., __last=..., __f=...) at /usr/include/c++/4.9/bits/stl_algo.h:3755
#4  0x00007ffff7aa581e in Wintermute::Tunnel::start () at src/wintermutecore/tunnel.cpp:188
#5  0x0000000000433ce5 in ZMQTunnelTestSuite::testSendOutMessage (this=0x7ffff7ffe520) at src/wintermute-transport-zeromq/test/tunnel.hh:186
#6  0x0000000000435c84 in TestDescription_suite_ZMQTunnelTestSuite_testSendOutMessage::runTest (this=0x65d520 <testDescription_suite_ZMQTunnelTestSuite_testSendOutMessage>) at build/src/wintermute-transport-zeromq/test/plugin-unit-zmq_tunnel_test.cc:38
#7  0x000000000042333e in CxxTest::RealTestDescription::run (this=0xce0bbb386805ab00) at /usr/include/cxxtest/RealDescriptions.cpp:121
#8  0x000000000042a749 in CxxTest::TestRunner::runTest (this=0xce0bbb386805ab00, td=...) at /usr/include/cxxtest/TestRunner.h:103
#9  0x000000000042a441 in CxxTest::TestRunner::runSuite (this=0x65d480 <suiteDescription_ZMQTunnelTestSuite>, sd=...) at /usr/include/cxxtest/TestRunner.h:87
#10 0x000000000042a091 in CxxTest::TestRunner::runWorld (this=0x65d480 <suiteDescription_ZMQTunnelTestSuite>) at /usr/include/cxxtest/TestRunner.h:67
#11 0x0000000000429cc2 in CxxTest::TestRunner::runAllTests (listener=...) at /usr/include/cxxtest/TestRunner.h:40
#12 0x000000000042acbc in CxxTest::ErrorFormatter::run (this=0x7fffffffc7d0) at /usr/include/cxxtest/ErrorFormatter.h:63
#13 0x000000000041f742 in main () at build/src/wintermute-transport-zeromq/test/plugin-unit-zmq_tunnel_test.cc:22
(gdb)

This is good news, in a while. This means (potentionally) all of my woes
with this was because of the Tunnel's incapability to store shared pointers
(or more likely, my incapability to use pointers properly here). Not at the
drawing board, but I know what's the problem now.

@jalcine
Copy link
Owner Author

jalcine commented Jan 19, 2015

Woke up and conducted a little experiment. Lo and behold, the context that's
being used isn't actually being crafted. I hate C++.

(gdb) run
Starting program: build/bin/plugin-unit-zmq_tunnel
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[wntr] Test suite starting...
Running cxxtest tests (1 test)0 [0x7ffff7fc07c0] INFO root null - Started logging session at Mon Jan 19 05:12:37 2015

0 [0x7ffff7fc07c0] DEBUG root null - 0
0 [0x7ffff7fc07c0] DEBUG root null - Searching for a plugin identified by transport-zmq ...
0 [0x7ffff7fc07c0] DEBUG root null - 0
0 [0x7ffff7fc07c0] DEBUG root null - The plugin is not in the known pool; attempting to load.
1 [0x7ffff7fc07c0] DEBUG root null - Working over 16 paths...
1 [0x7ffff7fc07c0] DEBUG root null - Using path transport-zmq...
1 [0x7ffff7fc07c0] DEBUG root null - Allocated an instance of a library; pre-loading library transport-zmq...
1 [0x7ffff7fc07c0] DEBUG root null - Attempting to load library from the filename 'transport-zmq'...
1 [0x7ffff7fc07c0] DEBUG root null - Claiming handle for library transport-zmq...
1 [0x7ffff7fc07c0] ERROR root null - Failed to load library 'transport-zmq': transport-zmq: cannot open shared object file: No such file or directory
1 [0x7ffff7fc07c0] ERROR root null - Failed to claim handle for library: transport-zmq: cannot open shared object file: No such file or directory
2 [0x7ffff7fc07c0] DEBUG root null - Was library loaded from transport-zmq? 0
2 [0x7ffff7fc07c0] DEBUG root null - Deallocated an instance of a library.
2 [0x7ffff7fc07c0] DEBUG root null - Using path transport-zmq.so...
2 [0x7ffff7fc07c0] DEBUG root null - Allocated an instance of a library; pre-loading library transport-zmq.so...
2 [0x7ffff7fc07c0] DEBUG root null - Attempting to load library from the filename 'transport-zmq.so'...
2 [0x7ffff7fc07c0] DEBUG root null - Claiming handle for library transport-zmq.so...
2 [0x7ffff7fc07c0] ERROR root null - Failed to load library 'transport-zmq.so': transport-zmq.so: cannot open shared object file: No such file or directory
2 [0x7ffff7fc07c0] ERROR root null - Failed to claim handle for library: transport-zmq.so: cannot open shared object file: No such file or directory
2 [0x7ffff7fc07c0] DEBUG root null - Was library loaded from transport-zmq.so? 0
2 [0x7ffff7fc07c0] DEBUG root null - Deallocated an instance of a library.
3 [0x7ffff7fc07c0] DEBUG root null - Using path libtransport-zmq.so...
3 [0x7ffff7fc07c0] DEBUG root null - Allocated an instance of a library; pre-loading library libtransport-zmq.so...
3 [0x7ffff7fc07c0] DEBUG root null - Attempting to load library from the filename 'libtransport-zmq.so'...
3 [0x7ffff7fc07c0] DEBUG root null - Claiming handle for library libtransport-zmq.so...
3 [0x7ffff7fc07c0] INFO root null - Handle loaded successfully.
3 [0x7ffff7fc07c0] DEBUG root null - Was library loaded from libtransport-zmq.so? 1
3 [0x7ffff7fc07c0] DEBUG root null - Resolving function w_plugin_version from libtransport-zmq.so
3 [0x7ffff7fc07c0] DEBUG root null - Raw version string from library: WINTERMUTE_VERSION
3 [0x7ffff7fc07c0] DEBUG root null - System 0.0.1-dev >= library min 0.0.0
3 [0x7ffff7fc07c0] DEBUG root null - Resolving function w_plugin_ctor from libtransport-zmq.so
3 [0x7ffff7fc07c0] DEBUG root null - Defined the internals for the plugin ZMQ.
4 [0x7ffff7fc07c0] DEBUG root null - Allocated a plugin named ZMQ.
4 [0x7ffff7fc07c0] DEBUG root null - Inserted ZMQ into the pool.
4 [0x7ffff7fc07c0] DEBUG root null - Was the plugin ZMQ inserted? 1
4 [0x7ffff7fc07c0] INFO root null - Plugin transport-zmq loaded, starting...
4 [0x7ffff7fc07c0] DEBUG root null - Crafted loop.
5 [0x7ffff7fc07c0] DEBUG root null - Built a new receiver for the tunnel called zmq.
5 [0x7ffff7fc07c0] DEBUG root null - Destroyed the zmq receiver.

In ZMQTunnelTestSuite::testSendOutMessage:
src/wintermute-transport-zeromq/test/tunnel.hh:83: Error: Test failed: ZMQReceiver: Need a pointer to a ZeroMQ context.
src/wintermute-transport-zeromq/test/tunnel.hh:83: Error: Test failed: Exception thrown from setUp()
src/wintermute-transport-zeromq/test/tunnel.hh:83: Error: Expected (suite()->setUp(); ok = true) not to throw, but it did
Failed 1 and Skipped 0 of 1 test
Success rate: 0%
[wntr] Test suite complete.
7 [0x7ffff7fc07c0] DEBUG root null - Loop destroyed.
8 [0x7ffff7fc07c0] DEBUG root null - Unloading plugin's library prior to deallocation...
8 [0x7ffff7fc07c0] DEBUG root null - Deallocated a plugin.
8 [0x7ffff7fc07c0] DEBUG root null - Deallocated an instance of a library.
15 [0x7ffff7fc07c0] DEBUG root null - Terminated logging session at Mon Jan 19 05:12:37 2015

[Inferior 1 (process 5015) exited with code 01]
(gdb)

This is how I start the day. Oi.

@jalcine jalcine closed this as completed Jan 23, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jalcine jalcine

Labels
bug core plugins
Projects
None yet
Milestone
v0.1.x
Development

No branches or pull requests
1 participant
@jalcine