Re: dbaccess unload to a pipe
Here is my code. I should clarify that I don't close
the pipe. I close the file that I'm writing to. Thank
everyone for your input.
#!/opt/perl5/bin/perl
use strict;
use MyModule;
use Getopt::Long;
use vars qw($fifo $file @table $filesize $filenum $key
$line);
# Set my FIFO device to whatever name is given me
$fifo = shift;
$key = shift;
# Check to make sure that I received a name
unless(-p "$fifo"){
die "I cannot start unless you give me a fifo device
to listen to.\\n";
}
# Open my FIFO device
open(FIFO, "<$fifo");
# Until the world ends do this stuff
while (1){
# If this is my code to die then die
if ($line =~ /REST_NOW_$key/o){
close(FIFO);
exit;
}
# If I recieve a line from my FIFO device
if ($line = <FIFO>){
# If this is my code to die then die
if ($line =~ /REST_NOW_$key/o){
close(FIFO);
exit;
} else {
# Until the current line tells me to close the
unload file do this stuff
until ($line =~ /END_TABLE_$key/o || $line =~
/REST_NOW_$key/o){
# If the current line is the start of a table
unload then set up my little world
if($line =~ /START_TABLE_$key/o){
# Trim and split the current line
chomp($line);
@table = split(/=/o,$line);
# Set the file name = to the current table name
$file = $table[1];
# Set the filesize to 0 and the file number to 1
$filesize = 0;
$filenum = 1;
# Open the output file handle with the current
table name
open(FILE,">$file.unl");
# Write this filename to the .ctrl file
open(CTRL,">$file.ctrl");
print CTRL "$file.unl\\n";
close(CTRL);
# select the output file handle and change it to
unbuffered
select((select(FILE), $| = 1)[0]);
} else {
# Print the current line to the unload file
print FILE "$line";
# Add the current line length to the filesize
$filesize += length($line);
# If the file size is larger than 1900000000
bytes
if ($filesize > 1900000000){
# Close my unload file
close(FILE);
# Open a new unload file with the file number
appended to the table name
open(FILE,">${file}_${filenum}.unl");
# Write this new file name to the .ctrl file
open(CTRL,">>$file.ctrl");
print CTRL "${file}_${filenum}.unl\\n";
close(CTRL);
# select the output file handle and change it to
unbuffered
select((select(FILE), $| = 1)[0]);
# Add 1 to the file number in case we need
another file later
$filenum++;
# Reset the file size to 0
$filesize = 0;
}
}
# Read the next line
$line = <FIFO>;
}
}
# Close my unload file
close(FILE);
}
}
--- Jonathan Leffler <jleffler@earthlink.net> wrote:
> DL Redden wrote:
>
> > We have several tables that if unloaded to a file
> > would be larger than 2GB. To manage this I have
> > written a pipe in perl. I use dbaccess' unload to
> to
> > unload to the pipe file that my perl program is> > sitting behind. My program will catch rows and
> slam
> > them into a file until the file gets to around 1.7
> GB
> > (just in case my calculations are wrong). I then
> close
> > that file and open a new one and continue throwing
> > rows into the new file.
> >
> > In testing this I have found that I am dropping
> rows.
> > I have poured through my code and cannot find any
> > reason for this.
>
> If you want, you can consider posting your Perl code
> so we can pore
> over it too, before pouring
> scorn^H^H^H^H^H^H^H^H^H^H^H^H^H we provide
> a diagnosis. :-))
>
> > The question:
> >
> > I was wondering if maybe when I stop reading the
> pipe
> > so that I can switch output files that the unload
> > doesn't block and it just keeps dumping rows that
> my
> > pipe isn't catching?
>
> I guess you mean that you are using a named pipe,
> rather than just
> piping the output of dbaccess to perl (which will
> likely work; send
> the output to /dev/stdout or /dev/fd/1). Actually,
> it doesn't matter;
> both named and anonymous pipes are blocking devices;
> when they reach
> capacity, the writers are held up pending a reader
> being available to
> read it. So, I don't think rows are being lost
> because the records
> are being slammed onto the pipe and overwritten
> before they are read.
>
>
> A couple of people mentioned SQLCMD. It should
> configure with large
> file support automatically - so it should be able to
> write direct to
> large files. The SQLUNLOAD (aka sqlcmd -U) command
> is very succinct
> for full table dumps. SQLCMD also has the ability
> to do:
>
> UNLOAD TO PIPE "program of your choosing" SELECT> ...;
>
> However - you are being WARNED - it does not handle
> SIGPIPE well if
> there's a problem while the output is being
> unloaded. It is not
> recommended to try UNLOAD TO PIPE "sed 1q >
> /tmp/first.line", for
> example. That requires a bit of work in the
> output.c file code which
> hasn't occurred yet - due to a severe lack of round
> tuits.
>
> (It also has skeletal support for LOAD FROM PIPE and
> RELOAD FROM PIPE.
> Faintly similar comments apply - but the problem is
> a lot less severe.)
>
> --
> Jonathan Leffler #include
> <disclaimer.h>
> Email: jleffler@earthlink.net, jleffler@us.ibm.com
> Guardian of DBD::Informix v2003.04 --
> http://dbi.perl.org/
>
__________________________________
Do you Yahoo!?
Yahoo! Mail - More reliable, more storage, less spam
http://mail.yahoo.com
sending to informix-list