Re: Problem loading data - 9.21.FC4

> dd if=$UNLOADS/$TABNAME.unl.Z | uncompress | dd of=$TEMPUNL & I don't quite understand this line. Well I sort of do, but what special thing is it accomplishing that uncompress -c $UNLOADS/$TABNAME.unl.Z > $TEMPUNL. Other than putting it in the background. Could this be a syncronization problem? If this table loads faster than other tables or causes a check point, could the load statement and the dd statement get crosswised with each other? Have you ever used named pipes? that way you never have a copy of the uncompressed file on the disk and Unix handles the syncronization(spelling?). I have had great sucees with them when space is limited. Here is a sample of how it works. mknod /tmp/unload.pipe p -- May be different on some machines uncompress -c $UNLOADS/$TABNAME >/tmp/unload.pipe & dbaccess <database> EOF load from /tmp/unload.pipe insert into $TABNAME ; EOF rm /tmp/unload.pipe /tmp/unload.pipe will not actually get bigger than the size of the pipe buffer which is probably adjustable but I have never done that. Mine always end up about 2K. Disclaimer: I haven't tested this answer and I haven't done it in awhile and I almost always get it sort of right which doesn't do you any good if you aren't famillar with named pipes. If you do want to do it this way and this doesn't actually work for you and you don't know how to fix it then and only then I can actually generate the proper commands for you, but I don't want to waste the time if you have no interest in this method. I think this method can be made to work with dbload and hpload, also. Also if you are cleaning up table extents you might want to make the load table raw while you are loading it. And then add the indexes and constraints after you load it. I have also heard that some people don't like named pipes, but I haven't ever had any trouble with them that I haven't caused myself. If anyone else knows some pros or cons on named pipes I wouldn't mind hearing about it.

Curtis Crowson wrote: > > > dd if=$UNLOADS/$TABNAME.unl.Z | uncompress | dd of=$TEMPUNL & > > I don't quite understand this line. Well I sort of do, but what > special thing is it accomplishing that > uncompress -c $UNLOADS/$TABNAME.unl.Z > $TEMPUNL. Other than putting > it in the background. Could this be a syncronization problem? If this > table loads faster than other tables or causes a check point, could > the load statement and the dd statement get crosswised with each > other? > > Have you ever used named pipes? that way you never have a copy of the [ ... example snipped ... ] > If you do want to do it this way and this doesn't actually work for > you and you don't know how to fix it then and only then I can actually > generate the proper commands for you, but I don't want to waste the > time if you have no interest in this method. I think this method can > be made to work with dbload and hpload, also. HPL has it all builtin. Look in the HPL users guide onpladm crate job ...... -fpl -d <name_of_pipecommand> actually does meta: <name_of_pipecommand> | onpload-reading-from-stdin when executing the job. onpladm create job ..... -fl -d <name_of_fifo> works with names pipes as expected (V9.21,V9.3,V9.40.UC2) dbload works fine using named pipes and for a very long time (> 11 years) was the solution for load-/unload files > 2GB which never ever was a show stopper - at least not for me. One needs to be on UNIX or Linux, though. > > Also if you are cleaning up table extents you might want to make the > load table raw while you are loading it. And then add the indexes and > constraints after you load it. Way to go, if you use a PDQ & parallel sort setup for the index rebuild. > > I have also heard that some people don't like named pipes, but I > haven't ever had any trouble with them that I haven't caused myself. > If anyone else knows some pros or cons on named pipes I wouldn't mind > hearing about it. Depending of OS version there are techniques around to increase the named pipe buffer size, but I think this could only have been as far back as the BSD days, when it was a problem to keep a slow streamer tape in streaming mode - gee I'm getting old it seems dic_k -- Richard Kofler SOLID STATE EDV Dienstleistungen GmbH Vienna/Austria/Europe

"Richard Kofler" <richard.kofler@chello.at> wrote in message news:401B707D.7AD75F18@chello.at... > Curtis Crowson wrote: > > > > > dd if=$UNLOADS/$TABNAME.unl.Z | uncompress | dd of=$TEMPUNL & > > > > I don't quite understand this line. Well I sort of do, but what > > special thing is it accomplishing that > > uncompress -c $UNLOADS/$TABNAME.unl.Z > $TEMPUNL. Other than putting > > it in the background. Could this be a syncronization problem? If this > > table loads faster than other tables or causes a check point, could > > the load statement and the dd statement get crosswised with each > > other? > > > > Have you ever used named pipes? that way you never have a copy of the > > [ ... example snipped ... ] > > > If you do want to do it this way and this doesn't actually work for > > you and you don't know how to fix it then and only then I can actually > > generate the proper commands for you, but I don't want to waste the > > time if you have no interest in this method. I think this method can > > be made to work with dbload and hpload, also. > > HPL has it all builtin. Look in the HPL users guide > onpladm crate job ...... -fpl -d <name_of_pipecommand> > actually does > meta: <name_of_pipecommand> | onpload-reading-from-stdin > when executing the job. Just to repeat that, due to a bug in his release of IDS, HPL unloads are very sloooowwwww.

Neil Truby wrote: > > "Richard Kofler" <richard.kofler@chello.at> wrote in message > news:401B707D.7AD75F18@chello.at... > > Curtis Crowson wrote: > > > > > > > dd if=$UNLOADS/$TABNAME.unl.Z | uncompress | dd of=$TEMPUNL & > > > > > > I don't quite understand this line. Well I sort of do, but what > > > special thing is it accomplishing that > > > uncompress -c $UNLOADS/$TABNAME.unl.Z > $TEMPUNL. Other than putting > > > it in the background. Could this be a syncronization problem? If this > > > table loads faster than other tables or causes a check point, could > > > the load statement and the dd statement get crosswised with each > > > other? > > > > > > Have you ever used named pipes? that way you never have a copy of the > > > > [ ... example snipped ... ] > > > > > If you do want to do it this way and this doesn't actually work for > > > you and you don't know how to fix it then and only then I can actually > > > generate the proper commands for you, but I don't want to waste the > > > time if you have no interest in this method. I think this method can > > > be made to work with dbload and hpload, also. > > > > HPL has it all builtin. Look in the HPL users guide > > onpladm crate job ...... -fpl -d <name_of_pipecommand> > > actually does > > meta: <name_of_pipecommand> | onpload-reading-from-stdin > > when executing the job. > > Just to repeat that, due to a bug in his release of IDS, HPL unloads are > very sloooowwwww. hmm maybe I am wrong, but the OP showed a load not an unload. I am not aware that the bug in question does slow down a load, but I may be wrong, or missing something. And for all folks out there having slow HPL unloads IMHO it is always worth a try to drop any constraints (drop! not only disable) and then drop all indexes on the table one wants to unload using HPL. Then try again to unlod and see what happens. It is also a *very* good idea to have no query running under grant manager control at the time when you unload...... dic_k -- Richard Kofler SOLID STATE EDV Dienstleistungen GmbH Vienna/Austria/Europe

"Richard Kofler" <richard.kofler@chello.at> wrote in message news:401B8F15.B8FC55BD@chello.at... > Neil Truby wrote: > > > > "Richard Kofler" <richard.kofler@chello.at> wrote in message > > news:401B707D.7AD75F18@chello.at... > > > Curtis Crowson wrote: > hmm > > maybe I am wrong, but the OP showed a load > not an unload. > > I am not aware that the bug in question does slow down > a load, but I may be wrong, or missing something. No, you're correct, it only affects unloads, not loads.