ZeroMQ poll when socket bound to multiple addresses doesn't always work
I am building an application using ZeroMQ as the infrastructure. I'm trying to debug my plumbing and I've run into an unexpected behavior. Examples are in python for simplicity, but my target is C++
This is the world server, written in a polling loop, and bound to two addresses
#!/usr/bin/python
from __future__ import print_function
import zmq
context = zmq.Context()
s = context.socket(zmq.REP)
#s.bind("tcp://*:5555")
s.bind("ipc://world.ipc")
poller = zmq.Poller()
poller.register(s, zmq.POLLIN)
while True:
socks = dict(poller.poll())
print("... poll returned ...", socks)
if socks.get(s) == zmq.POLLIN:
m = s.recv()
print("Received request: [0]".format(m))
s.send(b"World")
This is the hello client, using one of the addresses
#! /usr/bin/python
from __future__ import print_function
import zmq
context = zmq.Context()
# Socket to talk to server
print("Connecting to hello world server...")
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")
socket.connect("ipc://world.ipc")
# Do 10 requests, waiting each time for a response
for request in range(10):
print("Sending request 0 ...".format(request))
socket.send(b"Hello")
# Get the reply
message = socket.recv()
print("Received reply 0 [ 1 ]".format(request, message))
If the hello client connects to two addresses, things go south. If the server is bound (only) to the IPC address, and the client connects to both the IPC and TCP addresses (in that order), exactly one response will be received, then the client hangs. If the TCP address is connected first, no responses are received.
If the world server binds to two addresses, the client seems more robust, regardless of which order the server binds addresses.
However, given a slightly different client (closer to my desired implementation), embedded in a wsgiref web server, I have consistent failure on the IPC address:
#!/usr/bin/python
from __future__ import print_function
import zmq
context = zmq.Context()
import wsgiref
from wsgiref.simple_server import make_server
html_head = """<html><head>meta<title>title</title>script</head>"""
def create_head(title, meta="", script=""):
return html_head.format(title=title, meta=meta, script=script)
def create_foot():
return "</body></html>"
default_resp_headers = [
('Content-Type', 'text/html; charset=utf-8'),
]
import sys
class MyApp(object):
def doIndex(self, environ):
# Handle the default case
response_body = [ create_head("Hello App"),
"""
<body>
<h1>Text goes here</h1>
<a href='/hello'>Hello World</a>
""",
create_foot(),
]
status = '200 OK'
response_headers = default_resp_headers[:]
return status, response_headers, response_body
def do_hello(self, environ):
socket = context.socket(zmq.REQ)
print("DEBUG: connecting to hello socket"); sys.stdout.flush()
socket.connect("ipc://world.ipc")
# socket.connect("tcp://localhost:5555")
print("DEBUG: connected; sending hello"); sys.stdout.flush()
socket.send(b"Hello")
print("DEBUG: sent; receiving reply"); sys.stdout.flush()
response_body = [ create_head("Remote Hello"),
]
if socket.poll(2000) == zmq.POLLIN:
message = socket.recv()
print("DEBUG: received", message); sys.stdout.flush()
response_body.append(
"""<body>
<h1>Hello message</h1>
""".format(message=message)
)
status = '200 OK'
else:
response_body.append(
"""<body><h1>Error</h1><p>Hello Server not responding</p> """
)
status = '504 Gateway Timeout'
response_body.append(create_foot())
response_headers = default_resp_headers[:]
return status, response_headers, response_body
def __call__(self, environ, start_response):
wsgiref.util.shift_path_info(environ)
path = environ.get('SCRIPT_NAME', '/')[1:] # strip the leading /
method_name = "do_"+path
print("DEBUG: finding", method_name)
if not hasattr(self, method_name): method_name = "doIndex"
print("DEBUG: calling", method_name)
method = getattr(self, method_name)
print("DEBUG: method", repr(method))
status, response_headers, response_body = method(environ)
# the content-length is the sum of all string's lengths
content_length = sum([len(s) for s in response_body])
response_headers.append( ('Content-Length', str(content_length)) )
start_response(status, response_headers)
return response_body
def event_loop(interface, port):
httpd = make_server(interface, port, MyApp())
dispatch =
poller = zmq.Poller()
# For each socket to be watched, register with the poller and set a dispatch function.
poller.register(httpd, zmq.POLLIN)
dispatch[httpd.fileno()] = httpd.handle_request # note this is the bound function, not the invocation
while True:
for s,f in poller.poll():
if f == zmq.POLLIN: dispatch[s]()
if __name__ == "__main__":
event_loop("", 8051)
There's a lot going on there that isn't necessarily germane to the problem. The key part is the do_hello method. This gets invoked when a request like curl http://localhost:8051/hello is made. If the socket connects to the IPC address, the request times out. If it connects to the TCP address, it always works.
I hope that's clear. I'm still learning ZMQ, so if I missed something obvious, I welcome feedback.
zeromq
asked Nov 13 '18 at 21:57
jwmjwm
462113
add a comment |
I am building an application using ZeroMQ as the infrastructure. I'm trying to debug my plumbing and I've run into an unexpected behavior. Examples are in python for simplicity, but my target is C++
This is the world server, written in a polling loop, and bound to two addresses
#!/usr/bin/python
from __future__ import print_function
import zmq
context = zmq.Context()
s = context.socket(zmq.REP)
#s.bind("tcp://*:5555")
s.bind("ipc://world.ipc")
poller = zmq.Poller()
poller.register(s, zmq.POLLIN)
while True:
socks = dict(poller.poll())
print("... poll returned ...", socks)
if socks.get(s) == zmq.POLLIN:
m = s.recv()
print("Received request: [0]".format(m))
s.send(b"World")
This is the hello client, using one of the addresses
#! /usr/bin/python
from __future__ import print_function
import zmq
context = zmq.Context()
# Socket to talk to server
print("Connecting to hello world server...")
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")
socket.connect("ipc://world.ipc")
# Do 10 requests, waiting each time for a response
for request in range(10):
print("Sending request 0 ...".format(request))
socket.send(b"Hello")
# Get the reply
message = socket.recv()
print("Received reply 0 [ 1 ]".format(request, message))
If the hello client connects to two addresses, things go south. If the server is bound (only) to the IPC address, and the client connects to both the IPC and TCP addresses (in that order), exactly one response will be received, then the client hangs. If the TCP address is connected first, no responses are received.
If the world server binds to two addresses, the client seems more robust, regardless of which order the server binds addresses.
However, given a slightly different client (closer to my desired implementation), embedded in a wsgiref web server, I have consistent failure on the IPC address:
#!/usr/bin/python
from __future__ import print_function
import zmq
context = zmq.Context()
import wsgiref
from wsgiref.simple_server import make_server
html_head = """<html><head>meta<title>title</title>script</head>"""
def create_head(title, meta="", script=""):
return html_head.format(title=title, meta=meta, script=script)
def create_foot():
return "</body></html>"
default_resp_headers = [
('Content-Type', 'text/html; charset=utf-8'),
]
import sys
class MyApp(object):
def doIndex(self, environ):
# Handle the default case
response_body = [ create_head("Hello App"),
"""
<body>
<h1>Text goes here</h1>
<a href='/hello'>Hello World</a>
""",
create_foot(),
]
status = '200 OK'
response_headers = default_resp_headers[:]
return status, response_headers, response_body
def do_hello(self, environ):
socket = context.socket(zmq.REQ)
print("DEBUG: connecting to hello socket"); sys.stdout.flush()
socket.connect("ipc://world.ipc")
# socket.connect("tcp://localhost:5555")
print("DEBUG: connected; sending hello"); sys.stdout.flush()
socket.send(b"Hello")
print("DEBUG: sent; receiving reply"); sys.stdout.flush()
response_body = [ create_head("Remote Hello"),
]
if socket.poll(2000) == zmq.POLLIN:
message = socket.recv()
print("DEBUG: received", message); sys.stdout.flush()
response_body.append(
"""<body>
<h1>Hello message</h1>
""".format(message=message)
)
status = '200 OK'
else:
response_body.append(
"""<body><h1>Error</h1><p>Hello Server not responding</p> """
)
status = '504 Gateway Timeout'
response_body.append(create_foot())
response_headers = default_resp_headers[:]
return status, response_headers, response_body
def __call__(self, environ, start_response):
wsgiref.util.shift_path_info(environ)
path = environ.get('SCRIPT_NAME', '/')[1:] # strip the leading /
method_name = "do_"+path
print("DEBUG: finding", method_name)
if not hasattr(self, method_name): method_name = "doIndex"
print("DEBUG: calling", method_name)
method = getattr(self, method_name)
print("DEBUG: method", repr(method))
status, response_headers, response_body = method(environ)
# the content-length is the sum of all string's lengths
content_length = sum([len(s) for s in response_body])
response_headers.append( ('Content-Length', str(content_length)) )
start_response(status, response_headers)
return response_body
def event_loop(interface, port):
httpd = make_server(interface, port, MyApp())
dispatch =
poller = zmq.Poller()
# For each socket to be watched, register with the poller and set a dispatch function.
poller.register(httpd, zmq.POLLIN)
dispatch[httpd.fileno()] = httpd.handle_request # note this is the bound function, not the invocation
while True:
for s,f in poller.poll():
if f == zmq.POLLIN: dispatch[s]()
if __name__ == "__main__":
event_loop("", 8051)
There's a lot going on there that isn't necessarily germane to the problem. The key part is the do_hello method. This gets invoked when a request like curl http://localhost:8051/hello is made. If the socket connects to the IPC address, the request times out. If it connects to the TCP address, it always works.
I hope that's clear. I'm still learning ZMQ, so if I missed something obvious, I welcome feedback.
zeromq
asked Nov 13 '18 at 21:57
jwmjwm
462113
add a comment |
I am building an application using ZeroMQ as the infrastructure. I'm trying to debug my plumbing and I've run into an unexpected behavior. Examples are in python for simplicity, but my target is C++
This is the world server, written in a polling loop, and bound to two addresses
#!/usr/bin/python
from __future__ import print_function
import zmq
context = zmq.Context()
s = context.socket(zmq.REP)
#s.bind("tcp://*:5555")
s.bind("ipc://world.ipc")
poller = zmq.Poller()
poller.register(s, zmq.POLLIN)
while True:
socks = dict(poller.poll())
print("... poll returned ...", socks)
if socks.get(s) == zmq.POLLIN:
m = s.recv()
print("Received request: [0]".format(m))
s.send(b"World")
This is the hello client, using one of the addresses
#! /usr/bin/python
from __future__ import print_function
import zmq
context = zmq.Context()
# Socket to talk to server
print("Connecting to hello world server...")
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")
socket.connect("ipc://world.ipc")
# Do 10 requests, waiting each time for a response
for request in range(10):
print("Sending request 0 ...".format(request))
socket.send(b"Hello")
# Get the reply
message = socket.recv()
print("Received reply 0 [ 1 ]".format(request, message))
If the hello client connects to two addresses, things go south. If the server is bound (only) to the IPC address, and the client connects to both the IPC and TCP addresses (in that order), exactly one response will be received, then the client hangs. If the TCP address is connected first, no responses are received.
If the world server binds to two addresses, the client seems more robust, regardless of which order the server binds addresses.
However, given a slightly different client (closer to my desired implementation), embedded in a wsgiref web server, I have consistent failure on the IPC address:
#!/usr/bin/python
from __future__ import print_function
import zmq
context = zmq.Context()
import wsgiref
from wsgiref.simple_server import make_server
html_head = """<html><head>meta<title>title</title>script</head>"""
def create_head(title, meta="", script=""):
return html_head.format(title=title, meta=meta, script=script)
def create_foot():
return "</body></html>"
default_resp_headers = [
('Content-Type', 'text/html; charset=utf-8'),
]
import sys
class MyApp(object):
def doIndex(self, environ):
# Handle the default case
response_body = [ create_head("Hello App"),
"""
<body>
<h1>Text goes here</h1>
<a href='/hello'>Hello World</a>
""",
create_foot(),
]
status = '200 OK'
response_headers = default_resp_headers[:]
return status, response_headers, response_body
def do_hello(self, environ):
socket = context.socket(zmq.REQ)
print("DEBUG: connecting to hello socket"); sys.stdout.flush()
socket.connect("ipc://world.ipc")
# socket.connect("tcp://localhost:5555")
print("DEBUG: connected; sending hello"); sys.stdout.flush()
socket.send(b"Hello")
print("DEBUG: sent; receiving reply"); sys.stdout.flush()
response_body = [ create_head("Remote Hello"),
]
if socket.poll(2000) == zmq.POLLIN:
message = socket.recv()
print("DEBUG: received", message); sys.stdout.flush()
response_body.append(
"""<body>
<h1>Hello message</h1>
""".format(message=message)
)
status = '200 OK'
else:
response_body.append(
"""<body><h1>Error</h1><p>Hello Server not responding</p> """
)
status = '504 Gateway Timeout'
response_body.append(create_foot())
response_headers = default_resp_headers[:]
return status, response_headers, response_body
def __call__(self, environ, start_response):
wsgiref.util.shift_path_info(environ)
path = environ.get('SCRIPT_NAME', '/')[1:] # strip the leading /
method_name = "do_"+path
print("DEBUG: finding", method_name)
if not hasattr(self, method_name): method_name = "doIndex"
print("DEBUG: calling", method_name)
method = getattr(self, method_name)
print("DEBUG: method", repr(method))
status, response_headers, response_body = method(environ)
# the content-length is the sum of all string's lengths
content_length = sum([len(s) for s in response_body])
response_headers.append( ('Content-Length', str(content_length)) )
start_response(status, response_headers)
return response_body
def event_loop(interface, port):
httpd = make_server(interface, port, MyApp())
dispatch =
poller = zmq.Poller()
# For each socket to be watched, register with the poller and set a dispatch function.
poller.register(httpd, zmq.POLLIN)
dispatch[httpd.fileno()] = httpd.handle_request # note this is the bound function, not the invocation
while True:
for s,f in poller.poll():
if f == zmq.POLLIN: dispatch[s]()
if __name__ == "__main__":
event_loop("", 8051)
There's a lot going on there that isn't necessarily germane to the problem. The key part is the do_hello method. This gets invoked when a request like curl http://localhost:8051/hello is made. If the socket connects to the IPC address, the request times out. If it connects to the TCP address, it always works.
I hope that's clear. I'm still learning ZMQ, so if I missed something obvious, I welcome feedback.
zeromq
asked Nov 13 '18 at 21:57
jwmjwm
462113
I am building an application using ZeroMQ as the infrastructure. I'm trying to debug my plumbing and I've run into an unexpected behavior. Examples are in python for simplicity, but my target is C++
This is the world server, written in a polling loop, and bound to two addresses
#!/usr/bin/python
from __future__ import print_function
import zmq
context = zmq.Context()
s = context.socket(zmq.REP)
#s.bind("tcp://*:5555")
s.bind("ipc://world.ipc")
poller = zmq.Poller()
poller.register(s, zmq.POLLIN)
while True:
socks = dict(poller.poll())
print("... poll returned ...", socks)
if socks.get(s) == zmq.POLLIN:
m = s.recv()
print("Received request: [0]".format(m))
s.send(b"World")
This is the hello client, using one of the addresses
#! /usr/bin/python
from __future__ import print_function
import zmq
context = zmq.Context()
# Socket to talk to server
print("Connecting to hello world server...")
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")
socket.connect("ipc://world.ipc")
# Do 10 requests, waiting each time for a response
for request in range(10):
print("Sending request 0 ...".format(request))
socket.send(b"Hello")
# Get the reply
message = socket.recv()
print("Received reply 0 [ 1 ]".format(request, message))
If the hello client connects to two addresses, things go south. If the server is bound (only) to the IPC address, and the client connects to both the IPC and TCP addresses (in that order), exactly one response will be received, then the client hangs. If the TCP address is connected first, no responses are received.
If the world server binds to two addresses, the client seems more robust, regardless of which order the server binds addresses.
However, given a slightly different client (closer to my desired implementation), embedded in a wsgiref web server, I have consistent failure on the IPC address:
#!/usr/bin/python
from __future__ import print_function
import zmq
context = zmq.Context()
import wsgiref
from wsgiref.simple_server import make_server
html_head = """<html><head>meta<title>title</title>script</head>"""
def create_head(title, meta="", script=""):
return html_head.format(title=title, meta=meta, script=script)
def create_foot():
return "</body></html>"
default_resp_headers = [
('Content-Type', 'text/html; charset=utf-8'),
]
import sys
class MyApp(object):
def doIndex(self, environ):
# Handle the default case
response_body = [ create_head("Hello App"),
"""
<body>
<h1>Text goes here</h1>
<a href='/hello'>Hello World</a>
""",
create_foot(),
]
status = '200 OK'
response_headers = default_resp_headers[:]
return status, response_headers, response_body
def do_hello(self, environ):
socket = context.socket(zmq.REQ)
print("DEBUG: connecting to hello socket"); sys.stdout.flush()
socket.connect("ipc://world.ipc")
# socket.connect("tcp://localhost:5555")
print("DEBUG: connected; sending hello"); sys.stdout.flush()
socket.send(b"Hello")
print("DEBUG: sent; receiving reply"); sys.stdout.flush()
response_body = [ create_head("Remote Hello"),
]
if socket.poll(2000) == zmq.POLLIN:
message = socket.recv()
print("DEBUG: received", message); sys.stdout.flush()
response_body.append(
"""<body>
<h1>Hello message</h1>
""".format(message=message)
)
status = '200 OK'
else:
response_body.append(
"""<body><h1>Error</h1><p>Hello Server not responding</p> """
)
status = '504 Gateway Timeout'
response_body.append(create_foot())
response_headers = default_resp_headers[:]
return status, response_headers, response_body
def __call__(self, environ, start_response):
wsgiref.util.shift_path_info(environ)
path = environ.get('SCRIPT_NAME', '/')[1:] # strip the leading /
method_name = "do_"+path
print("DEBUG: finding", method_name)
if not hasattr(self, method_name): method_name = "doIndex"
print("DEBUG: calling", method_name)
method = getattr(self, method_name)
print("DEBUG: method", repr(method))
status, response_headers, response_body = method(environ)
# the content-length is the sum of all string's lengths
content_length = sum([len(s) for s in response_body])
response_headers.append( ('Content-Length', str(content_length)) )
start_response(status, response_headers)
return response_body
def event_loop(interface, port):
httpd = make_server(interface, port, MyApp())
dispatch =
poller = zmq.Poller()
# For each socket to be watched, register with the poller and set a dispatch function.
poller.register(httpd, zmq.POLLIN)
dispatch[httpd.fileno()] = httpd.handle_request # note this is the bound function, not the invocation
while True:
for s,f in poller.poll():
if f == zmq.POLLIN: dispatch[s]()
if __name__ == "__main__":
event_loop("", 8051)
There's a lot going on there that isn't necessarily germane to the problem. The key part is the do_hello method. This gets invoked when a request like curl http://localhost:8051/hello is made. If the socket connects to the IPC address, the request times out. If it connects to the TCP address, it always works.
I hope that's clear. I'm still learning ZMQ, so if I missed something obvious, I welcome feedback.
zeromq
zeromq
asked Nov 13 '18 at 21:57
jwmjwm
462113
asked Nov 13 '18 at 21:57
jwmjwm
462113
asked Nov 13 '18 at 21:57
jwmjwm
462113
asked Nov 13 '18 at 21:57
jwmjwm
462113
asked Nov 13 '18 at 21:57
jwmjwm
462113
462113
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53290128%2fzeromq-poll-when-socket-bound-to-multiple-addresses-doesnt-always-work%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53290128%2fzeromq-poll-when-socket-bound-to-multiple-addresses-doesnt-always-work%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
