Re: high performance loader

Posted in 2004

Pravin Kedia — Fri, 2 Apr 2004 16:13:14 -0600 — source: Usenet: comp.databases.informix

--0__=08BBE4F9DFEA0D928f9e8a93df938690918c08BBE4F9DFEA0D92 Content-type: text/plain; charset=US-ASCII Hi, Onpload has a highly parallel Client-Server Architecture. It is like a mini-server. These onpload's that you are seeing are for each of the vp's. (Just like IDS server) If you configure the job properly (and also the plconfig.std or you own one using $PLCONFIG variable) you can maximise the throughput further more. If only 1 unload/load file was specified then all pload threads will write into the same file in a highly parallel fashion. You can ask the rows to be unloaded into or loaded from multiple files simultaneously in parallel. Though it will never merge those files. There is no need to do that as you can always load back using multiple files. Thanks, Pravin. owner-informix-list@iiug.org wrote on 04/02/2004 12:30:15 PM: > I had to download 13 million rows from a table and I decided to use > onpload for that. so I used myonpload to download. > I noticed that it spawned off many onpload process to download > the table:- > > dba 16892 14234 0 13:15:40 pts/10 0:00 myonpload -d big_bro > -t search_request2003 -f > search2003.unl -U D > dba 16904 16892 0 13:15:42 pts/10 0:00 sh -c onpload -p > PR16892 -j JB16892 -fu > dba 16954 16905 0 13:15:47 pts/10 0:01 onpload -p PR16892 -j > JB16892 -fu > dba 16952 16905 0 13:15:46 pts/10 0:00 onpload -p PR16892 -j > JB16892 -fu > dba 16915 16905 0 13:15:43 pts/10 0:00 onpload -p PR16892 -j > JB16892 -fu > dba 16955 16905 4 13:15:47 pts/10 0:17 onpload -p PR16892 -j > JB16892 -fu > dba 16916 16905 1 13:15:43 pts/10 0:02 onpload -p PR16892 -j > JB16892 -fu > dba 16905 16904 41 13:15:42 pts/10 5:08 onpload -p PR16892 -j > JB16892 -fu > dba 16934 16905 0 13:15:44 pts/10 0:00 onpload -p PR16892 -j > JB16892 -fu > dba 16953 16905 0 13:15:46 pts/10 0:00 onpload -p PR16892 -j > JB16892 -fu > dba 16914 16905 0 13:15:43 pts/10 0:00 onpload -p PR16892 -j > JB16892 -fu > > Just out of curiosity, how does it work. If there are many processesparallely > downloading the table, how do they write to the same unload file > simultaneously. > I was expecting each process to write to its own unload file and > merge all files > into one at the end. It doesn't seem so. > > BTW onpload is so DAMN fast. Just 3 seconds to download 100000 rows. > > > > --0__=08BBE4F9DFEA0D928f9e8a93df938690918c08BBE4F9DFEA0D92 Content-type: text/html; charset=US-ASCII Content-Disposition: inline <html><body> Hi, Onpload has a highly parallel Client-Server Architecture. It is like a mini-server. These onpload's that you are seeing are for each of the vp's. (Just like IDS server) If you configure the job properly (and also the plconfig.std or you own one using $PLCONFIG variable) you can maximise the throughput further more. If only 1 unload/load file was specified then all pload threads will write into the same file in a highly parallel fashion. You can ask the rows to be unloaded into or loaded from multiple files simultaneously in parallel. Though it will never merge those files. There is no need to do that as you can always load back using multiple files. Thanks, Pravin. <tt>owner-informix-list@iiug.org wrote on 04/02/2004 12:30:15 PM: > I had to download 13 million rows from a table and I decided to use > onpload for that. so I used myonpload to download. > I noticed that it spawned off many onpload process to download > the table:- > >      dba 16892 14234  0 13:15:40 pts/10   0:00 myonpload -d big_bro > -t search_request2003 -f > search2003.unl -U D >      dba 16904 16892  0 13:15:42 pts/10   0:00 sh -c onpload -p > PR16892 -j JB16892 -fu >      dba 16954 16905  0 13:15:47 pts/10   0:01 onpload -p PR16892 -j > JB16892 -fu >      dba 16952 16905  0 13:15:46 pts/10   0:00 onpload -p PR16892 -j > JB16892 -fu >      dba 16915 16905  0 13:15:43 pts/10   0:00 onpload -p PR16892 -j > JB16892 -fu >      dba 16955 16905  4 13:15:47 pts/10   0:17 onpload -p PR16892 -j > JB16892 -fu >      dba 16916 16905  1 13:15:43 pts/10   0:02 onpload -p PR16892 -j > JB16892 -fu >      dba 16905 16904 41 13:15:42 pts/10   5:08 onpload -p PR16892 -j > JB16892 -fu >      dba 16934 16905  0 13:15:44 pts/10   0:00 onpload -p PR16892 -j > JB16892 -fu >      dba 16953 16905  0 13:15:46 pts/10   0:00 onpload -p PR16892 -j > JB16892 -fu >      dba 16914 16905  0 13:15:43 pts/10   0:00 onpload -p PR16892 -j > JB16892 -fu > > Just out of curiosity, how does it work. If there are many processesparallely > downloading the table, how do they write to the same unload file > simultaneously. > I was expecting each process to write to its own unload file and > merge all files > into one at the end. It doesn't seem so. > > BTW onpload is so DAMN fast. Just 3 seconds to download 100000 rows. > > > > </tt></body></html> --0__=08BBE4F9DFEA0D928f9e8a93df938690918c08BBE4F9DFEA0D92-- sending to informix-list