My application needs to pull data from a PUSH socket (I do not have control of the server side). The data is processed and written to file, which is single-threaded and I/O bound with variable performance.
I would like the application to do the minimum work to pull the messages into process memory so that a) the server is not blocked and won't drop any messages (which is already the case with just a normal PULL socket) and b) I can monitor the backlog of messages caused by slow processing.
I can achieve this using a wrapper socket with three sockets internally:
- One for the actual connection to the server
- A pair of sockets with an
inproc://connection where I can keep count of messages as they are sent and received. So the data flows is:
Thread 1
rx.recv()inproc_tx.send()queued_messages++
Thread 2
inproc_rx.recv()queued_messages--do_processing()
This seemed like a sensible way to buffer the messages, as it already has zmq::message_ts from recv() and I am sure it will be more efficiently implemented than rolling my own queuing logic. It works, but it seems like a lot of overhead and extra code when the underlying code already knows how many messages are queued to decide when to drop new messages.
Is there an API to query the queue usage on a ZeroMQ PULL socket? Or a better way to implement the above?