Informix JDBC driver cleanup bug - memory leak
Kristian Arlt saw Tomcat warn that the Informix JDBC IfxSqliConnectCleaner thread was not stopped on redeploy (JDBC 4.50.10.1). Markus Holzbauer suggested IFMXCONNECTION_CLEANER_THREADS=0 in the URL; Brian Hughes agreed with disabling it under frameworks. That removed the warning and memory stayed steady. Xavier Escote argued the driver still queues connections forever when no cleaner runs, a leak.
Auto-generated by Claude from the posts below — may be imperfect; read the full thread.
I have a problem when hot deploying our Java service to Tomcat, when the app is restarted getting the following WARNING: 07-Jun-2024 14:18:12.815 WARNING [Catalina-utility-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [dashboard] appears to have started a thread named [ IfxSqliConnectCleaner -Thread-1] but has failed to stop it. This is very likely to create a memory leak . Stack trace of thread: java.base/jdk.internal.misc.Unsafe.park(Native Method) java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:269) java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1758) java.base/java.util.concurrent.DelayQueue.poll(DelayQueue.java:305) java.base/java.util.concurrent.DelayQueue.poll(DelayQueue.java:100) com.informix.jdbc.IfxSqliConnect$IfxSqliConnectCleaner.run(IfxSqliConnect.java:645) java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) java.base/java.lang.Thread.run(Thread.java:1583) This can be reproduced by consecutive WAR deployments, so that the memory usage increases after each time the service is restarted. I was wondering if the JDBC driver could be enhanced to help close all resources. ------------------------------ Kristian Arlt ------------------------------
What version of the JDBC driver? ------------------------------ Øyvind Gjerstad Developer/Architect PostNord AS ------------------------------
The latest found on mvnrepository: <dependency> <groupId>com.ibm.informix</groupId> <artifactId>jdbc</artifactId> <version>4.50.10.1</version> </dependency> ------------------------------ Kristian Arlt ------------------------------
Ok, we are using 4.50.9. Not exactly same environment, though, we are using jboss, not tomcat. Haven't noticed this problem. ------------------------------ Øyvind Gjerstad Developer/Architect PostNord AS ------------------------------
Hello Kristian, The connection cleaner thread has following properties: IFMXCONNECTION_CLEANER_THREADS (defaults to 0) IFMXCONNECTION_CLEANER_DELAY_MAXIMUM (defaults to 15000 msec/15 sec) You can try to disalbe the connection cleaner thread with following Parameter in the connection string/URL: ;IFMXCONNECTION_CLEANER_THREADS=0 HTH, Cheers, Markus ------------------------------ Markus Holzbauer ------------------------------
From which version of the JDBC driver are these connection cleaner threads used? ------------------------------ Øyvind Gjerstad Developer/Architect PostNord AS ------------------------------
I think the feature is available since JDBC 4.10.JC12. Maybe someone from IBM can correct me... Cheers, Markus ------------------------------ Markus Holzbauer ------------------------------
The cleaner thread was introduced way back, before 4.10.JC6 even if I recall. But the ability to turn it off was introduced much later. Currently the default value is 1 thread. The point of this thread was to help with developers who didn't understand they needed to close connections after they used them (think back before try-with-resources feature or web frameworks like Spring were built in java). My opinion the feature should not ever be enabled if you are using a framework and web servers like Tomcat or Spring to manage connections for you. I would just shut it off since those frameworks ensure connections are correctly closed when needed and pooled when wanted. The lingering thread is a defect/bug but most of the time you don't even want it running in the first place so that's the easier solution for you I think. Should the default be 0? Probably yes, but unsure if someone will change it. Changing default behavior is always a tricky subject. Hope this helps! ------------------------------ Brian Hughes ------------------------------
Thank you for the quick reply, I added the IFMXCONNECTION_CLEANER_THREADS parameter to our connection string and the Warning is gone. After also analyzing the memory usage, Tomcat memory usage stays steady. ------------------------------ Kristian Arlt ------------------------------
I think there could be a possible memory leak. Even the threads are not started witch I agree is the best, I see lot of weak references. Looking at code: Even cleaner will not be created, queueOfConnection gets a connection on every creation. On a long living application that creates/closes many connections this queue grows to infinite. I think. /* 588 */ protected static final BlockingQueue < DelayedWeakReference < IfxSqliConnect >> queueOfConnections = new DelayQueue <> () ; static synchronized < T extends IfxSqliConnect > void ensureConnectionMonitorThreadIsRunning ( T connection ) { /* 597 */ long delay = IfxConnectionProperty . IFMXCONNECTION_CLEANER_DELAY_MAXIMUM . getLongValue (( IfxSqliConnect ) connection ) ; /* 598 */ queueOfConnections . offer ( new DelayedWeakReference ( connection , delay )) ; /* */ /* 600 */ if ( cleaner == null ) { /* 601 */ int nThreads = IfxConnectionProperty . IFMXCONNECTION_CLEANER_THREADS . getIntValue (( IfxSqliConnect ) connection ) ; /* 602 */ if ( nThreads > 0 ) { /* 603 */ cleaner = Executors . newFixedThreadPool ( nThreads , new IfxSqliConnect CleanerThreadFactory ()) ; /* 604 */ for ( int i = 0 ; i < nThreads ; i ++ ) /* 605 */ cleaner . execute ( new IfxSqliConnect Cleaner ()) ; /* */ } /* */ } /* */ } Class Name | Shallow Heap | Retained Heap ------------------------------ com.informix.jdbc.IfxSqli @ 0x6e0f7b7d0 | 312 | 2,101,048 '- prot com.informix.jdbc.IfxStatement @ 0x6e0f7b748 | 136 | 376 '- referent java.lang.ref.WeakReference @ 0x6e0f7b728 | 32 | 32 '- weakReferenceStmt com.informix.jdbc.IfxSqliConnect$StmtList @ 0x6e0f7ba68 | 24 | 56 '- [31] java.lang.Object[160] @ 0x69f9c9820 | 656 | 3,008 '- elementData java.util.Vector @ 0x68de64be0 | 32 | 3,040 '- stmtList com.informix.jdbc.IfxSqliConnect @ 0x68de601c8 | 376 | 14,208 '- referent com.informix.util.DelayedWeakReference @ 0x71c13de88 | 48 | 48 '- [56] java.lang.Object[343] @ 0x6cbe0def8 | 1,392 | 14,352 '- queue java.util.PriorityQueue @ 0x684c184a0 | 32 | 14,384 '- q java.util.concurrent.DelayQueue @ 0x6849d7c00 | 32 | 14,520 '- queueOfConnections class com.informix.jdbc.IfxSqliConnect @ 0x6849e5dc8 | 336 | 37,944 |- <class> com.informix.jdbc.IfxSqliConnect @ 0x6b53cc3b0 | 376 | 10,792 | |- <Java Local> java.lang.Thread @ 0x6849d7b00 IfxSqliConnectCleaner-Thread-1 Thread| 104 | 4,784 | |- jconn com.informix.jdbc.IfxStatement @ 0x71dfa3078 | 136 | 376 | |- conn com.informix.jdbc.IfxSqli @ 0x71dfa31b0 | 312 | 2,504 | |- conn com.informix.jdbc.IfxResultSet @ 0x71dfa5300 | 104 | 104 | |- conn com.informix.jdbc.IfxRowColumn @ 0x71dfaa008 | 48 | 48 | '- Total: 5 entries | | |- <class> com.informix.jdbc.IfxSqliConnect @ 0x714f85588 | 376 | 8,832 |- <class> com.informix.jdbc.IfxSqliConnect @ 0x6950c8ec0 | 376 | 16,040 |- <class> com.informix.jdbc.IfxSqliConnect @ 0x6a09d5bb8 | 376 | 13,168 |- <class> com.informix.jdbc.IfxSqliConnect @ 0x6c3283988 | 376 | 15,832 |- <class> com.informix.jdbc.IfxSqliConnect @ 0x6eb3e70f8 | 376 | 7,392 |- <class> com.informix.jdbc.IfxSqliConnect @ 0x709f0db10 | 376 | 10,688 |- <class> com.informix.jdbc.IfxSqliConnect @ 0x70c3c3250 | 376 | 10,664 '- <class> com.informix.jdbc.IfxSqliConnect @ 0x70cd302f8 | 376 | 9,064 ------------------------------ ------------------------------ Xavier Escote ------------------------------
| view attached The driver has a memory leak. When cleaner is not started it's internal queue stores connections (should not). As the cleaner will never run, connections will be keep in the queue forever. See test case. Output: Queue size:1 Queue size:2 Queue size:3 Queue size:4 Queue size:5 Queue size:6 Queue size:7 Queue size:8 Queue size:9 Queue size:10 ------------------------------ Xavier Escote ------------------------------