I am using simple one to one PUSH/PULL worker/server python codes to send and receive messages.
The worker uses the PUSH socket to send messages to the PULL server. The server processing unit is not as strong as the worker, therefore when sending too many messages, the server's RAM starts to grow until the system kills everything.
I tried setting the receiver high water mark as follows:
worker_sock_in = ZMQ_CTXT.socket(zmq.PULL)worker_sock_in.setsockopt(zmq.LINGER, 1000))worker_sock_in.setsockopt(zmq.RCVTIMEO, 1000)) # detects if the link is brokenworker_sock_in.setsockopt(zmq.RCVHWM, 1000)worker_sock_in_port = worker_sock_in.bind_to_random_port(listen_addr, port_start, port_end)The code below used for the worker to create and send messages:
sock_dest = ZMQ_CTXT.socket(zmq.PUSH)sock_dest.setsockopt(zmq.LINGER, 1000))sock_dest.setsockopt(zmq.SNDTIMEO, 1000)) # detects if the link is brokensock_dest.setsockopt(zmq.SNDHWM, 0) # never block on sending msgsock_dest.connect(sock_dest_address)# sends a msgself.sock_dest.send(msg, zmq.NOBLOCK)And it seems to correct the problem but my guess is overflow messages are just dropped by the server which is not acceptable in my situation.
I have based my development using this thread but I am not sure to understand the Additional Info part of the answer.
So questions are, what are the real behavior of HWM reached on noblock push/pull zeromq sockets and is there a way to have a push pull infrastructure that guaranties all sent messages will be received by the pull socket without inflating its memory or blocking the sender?