I-Net protocol stupid design: a performance issue for networking.

I have been requested by one of the customers of the company for which I work about networking performance that I did not detect before during development until I put our software in the production process. After some searches, I have found that the most important problems lays in the I-Net protocol used, from version 5.0 to the newest 7.22 I-Net implementation I use since a few months: INFORMIX-NET DOES NOT COMPRESS TRANSMITTED DATA TO THEIR ACTUAL LENGTH, BUT SEND THEM WITH THEIR MAXIMUM LENGTH, PADDING DATA WITH UNNECESSARY NULL BYTES, EVEN THOUGH THE PROTOCOL HAS A PREFIX BYTE FOR THE ACTUAL LENGTH. This means: if you design a procedure or queried table to return VARCHAR(255) values, Informix sends 256 bytes in the TCP data stream, even though actual mean size is much smaller. Because Informix uses 512-bytes long packet size by default, this means that even selecting only two VARCHAR(255) packets requires at least 2 packets, due to the small override required for data identification in the protocol. Why 256 bytes ? There is 1 leading byte for the actual length, plus the actual characters plus NUL padding characters. Selecting long lists of VARCHAR(255) values, even though the mean actual length is much smaller generates an unnecessary traffic on the network and this causes problem when your Informix server is running on a machine using a 10-Base-T based network interface: this generates unwanted contention on the network. Why network buffering does not solve the problem ? Because there is always an extra header before the transmitted data in the Informix protocol (about 14 bytes for each row), and all further columns in that row may be splitted into multiple TCP data packets (transmitting two VARCHAR(255) requires this 14 bytes header, and then twice 256 bytes, and this gives a total of more than the default 512 bytes packet size. You may solve a part of the problem by enlarging the packet size taking this 14 bytes offset into account (for example choosing a 530 bytes maximum packet size), however most low-cost network interfaces used by clients will manage them as large packets, a sparce resource on Win3.1x TCP/IP implementations, due to memory constraints, so clients are making blocking send/receive calls too often, due to the lack of large buffers for TCP. You may argue that a server should have a faster network interface, but the majority of networks only support 10MB/s Ethernet Interface. There is effectiveley the possibility to connect via IPC, but Informix does not guarantee the synchronization of non-blocking events via IPC, so the connection via a socket is always much more robust. Imagine also the case when you must give access to your Informix server via low-rated transmission lines, like ISDN or even modem PPP. For ISDN, the ressources required to support it nicely is to use an external router, connected to the Ethernet network, and most of them use 10-base-T interfaces. The same applies to modem pools, which are nearly always on a separate dedicated router or host, with the required hardware to support multiple serail interfaces (most servers don't have enough hardawre extensibility to provide enough serial lines). The net effect of the network traffic is that the server responses are much slower than expected when you have to connect many interactive users, even though your server theorically has enough resources can manage it softly. So you have to build either complicated network structures involving multiple Ethernet Interfaces, or intelligent network switches, and this adds a lot ot the cost of installation, management, and use (or renting) of the network. Very bad, because this is simply due to a stupid design of the I-Net protocol. Compared to Sybase or Oracle protocols, Informix then generates a traffic about 10 times (often more) important ! And the solutions proposed byu Informix to use delayed traffic options, packets grouping, extended packet sizes does not change the problem by a significant amount, and cause many design issue for the developper. I think that Informix should think of revamping completely its protocol and use a more compact encoding. Simply sniff your network (on AiX use the "iptrace", and "ipreport" buil-in tools on the server port), and you'll understand the problem when you'll large bunches of packets filled with many NUL bytes for just a simple query ! To provide compatibility, this new protocol should be a negotiable option at connect time, so that 5.x or 7.x I-Net clients can still connect to your server, until they upgrade their Ifx-Net (Connect or ESQL/C) libraries. In addition, when rewriting this protocol, other options like implicit description of dynamic requests, without actually requesting them explicitly, should be added, to reduce the requests/answers loops between the server and the client. For now I do not see any solution other than installing on the Informix server (and on the client host !) a local compression proxy to fix the problem, or to choose to implement your PPP servers directly on the same host than your Informix server and allow for data compression (in PPP options, and/or in modem settings). I have tried to implement the compression proxy, this was simple to do on the server, but harder on Win95 clients. For Win3.1x clients, the problem is much more difficult, because socket implementation is much less efficient, and so this adds extra time for events detection, even though it reduces network traffic dramatically, and gives a much better throughput on the Informix server running under Unix. I think that networking performance is a major issue, because it completely restrains the real performance of the server. All performance tests conducted by Informix via IPC or local IP are in fact biased by this issue, and all consumers will see in fact less positive results. In fact, the fact that Informix processes much more queries than Sybase or Oracle in TPS tests does not signify anything: real environments use networking, and network congestion is a real problem with Informix. If Informix does not solve this problem fast (just stripping unnecessary zeroes on the network should be easy and could dramatically improve performance), concurrents like sybase and Oracle will take the markets because Informix will not fit in customers environments.