Connection manager hang issue
Posted in 2023
Topics: Clustering, Grid & MACH11, Jobs, Consulting & Announcements
Hi, Tom and Gurus, We have a similar issue with CM hang as it described in the following message threads. Our CSDK is 4.10.FC1. What are the tips, workaround or solutions to this issue? Thanks Frank Connection Manager Issue (Rampant Speculation Invited!) Follow TOM GIRSCH Thu September 15, 2022 04:48 PM In which I describe a problem and invite rampant speculation. We have a CM (Connection Manager) with ... · 1. Connection Manager Issue (Rampant Speculation Invited!) 0 Like IBM Champion Actions Options Dropdown Posted Thu September 15, 2022 04:48 PM Mark As Inappropriate In which I describe a problem and invite rampant speculation. We have a CM (Connection Manager) with multiple redirect mode TLS SLAs where one of those SLAs would routinely stop responding. Sessions would connect and redirect as expected for a brief period of time, then all subsequent connection attempts would hang. The other SLAs would continue responding as normal. Upon restarting the CM process, the port would briefly resume function but quickly start hanging again. I spent the better part of a day troubleshooting this, and thanks to some selective wireshark-ing, I found one client IP that seemed to have been responsible for the issue. Every time that particular client tried to connect to that port, the hang would recur. And it looks to have been some sort of auto-spawned process, because it would pop up in bursts no more than about four minutes apart. I blocked that specific IP at the CM machine's firewall, and the problem ceased. [The owner of the client machine claims ignorance, but that's another story.] I tell you all of that to tell you this: I have no idea how I'd reproduce that behavior even if I were _trying_ to. In all of our direct-to-that-port testing, there were three basic outcomes: the connection was rejected because of a protocol mismatch; the client rejects the connection (because of a TLS certificate mismatch); or the connection succeeds and is immediately redirected, at which point the connection is closed. In any case, it really seems to me that a DoS against a connection manager shouldn't be _that_ easy to pull off. So, two questions: Does this sound like a CM bug to any of you? Because it sort of seems that way to me. And second, can you think of a way to make a program connect to that port and hold onto it in a way that would block any new connections from coming in? Thanks in advance. Note: Before I blocked the IP, I was able to see this behavior with both 4.50.FC8 and 4.50.FC7. ------------------------------ TOM GIRSCH ------------------------------ #informix · 2. RE: Connection Manager Issue (Rampant Speculation Invited!) 0 Like IBM Champion Actions Options Dropdown Posted Thu September 15, 2022 04:56 PM Mark As Inappropriate This was the basis of the program I wrote back in the dark ages to repo' the xctl functionality of XPS within a IDS environment. Might help Cheers Paul
HI, When this happens get a pstack of the connection manager process a lsof of the connection manager process netstat -an on the connection manager machine and get the state of the socket between the machines If on Linux also use the ss -n command With -e to get the inode and match that to lsof to find the socket With -T to show the thread for the socket match that to the thread from the pstack WIth -i to get more information If you can let us know the stack for the connection manager thread and the state of the socket that will help. Also if you can do the same on the client side (if known) that would also help. Regards, David. ------------------------------ David Williams ------------------------------