HPL Unloading error
A user on IDS 7.31.UC2 (Solaris 2.7) got intermittent "out of virtual shared memory" / "Cannot allocate shared memory" errors when using HPL to unload a ~33-million-row table to multiple files, even though the machine and the engine's virtual segment had free memory. Suggestions: raise SHMADD/SHMVIRTSIZE; check SHMTOTAL, since a non-zero limit causes this regardless of free RAM; note that onpload allocates its own shared memory segments (check with onstat -j <plod log> then 'seg', and reserve ~100MB for HPL); also try a single device and upgrade from the old 7.31.UC2. No confirmation back from the poster, so no resolution is recorded.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Dear all,
OS: Solaris 2.7
Memory : 2048MB
CPU : 4
Engine Version : IDS 7.31.UC2
I'm using HPL to unload my database into the flat
files format. I encountered the problem when the HPL
unloading a table that having ~33 mil rows into
multiple files. However, this problem only happen
intermittently. The error that I got shown as below:
Fri Jun 27 09:16:02 2003
09:16:02 dynamically allocated new shared memory
segment (size 4194304)
09:16:02 dynamically allocated new shared memory
segment (size 4194304)
09:16:02 out of virtual shared memory
09:16:02 dynamically allocated new shared memory
segment (size 4194304)
09:16:02 dynamically allocated new shared memory
segment (size 4194304)
09:16:02 dynamically allocated new shared memory
segment (size 4194304)
09:16:02 dynamically allocated new shared memory
segment (size 4194304)
and the log file on that particular table shown as
below:
Fri Jun 27 09:16:00 2003
SHMBASE 0x50000000CLIENTNUM 0x49010000
Session ID 2
Unload Database -> prodmac
Query Name -> AUTO.2
Device Array -> c_obs
Query Mapping -> AUTO.2
Query -> select * from c_obs where obsdate >
today - 365 for read only
Convert Reject -> /tmp/c_obs.rej
Cannot allocate shared memory
Database Unload Completed -- Unloaded 0 Records
Detected 0 Errors
Fri Jun 27 09:16:02 2003
The above error log was telling me the shared memory
was not enough. But, the system still have plenty of
memory while the HPL is running. I even tried to
increase the shminfo_shmmax from 256MB to 512MB, but
it doesn't help at all. Does anyone out there know how
to resolve this problem?
Here's my current plconfig and /etc/system files
# System Configuration
CONVERTTHREADS 1 # Number of conversion threads
per device
CONVERTVPS 4 # Max number of vps for
converters (total)
# Buffer Configuration
STRMBUFFSIZE 1024 # Buffer size for server
stream buffer (kbytes)
STRMBUFFERS 5 # Number of server stream
buffers per device
AIOBUFSIZE 256 # Buffer size for tape/file
I/O (kbytes)
AIOBUFFERS 5 # Number of buffers for I/O
per device
/etc/system file:
=================
set shmsys:shminfo_shmmax=536870912
set semsys:seminfo_semmap=256
set semsys:seminfo_semmni=4096
set semsys:seminfo_semmns=4096
set semsys:seminfo_semmnu=4096
set semsys:seminfo_semume=64
set semsys:seminfo_semmsl=100
set shmsys:shminfo_shmmin=100
set shmsys:shminfo_shmmni=500
set shmsys:shminfo_shmseg=100
Thanks & regards,
Jason
It relates to SHMVIRTSIZE and SHMADD in your $ONCONFIG.
Your SHMADD is unusually small.
Monitor the results with onstat -g seg
> -----Original Message-----
> From: forum.subscriber@iiug.org [mailto:forum.subscriber@iiug.org]On
> Behalf Of JASON PANG
> Sent: Wednesday, 2 July 2003 3:26 p.m.
> To: ids@iiug.org
> Subject: HPL Unloading error [1466]
>
>
> Dear all,
>
> OS: Solaris 2.7
> Memory : 2048MB
> CPU : 4
> Engine Version : IDS 7.31.UC2
>
> I'm using HPL to unload my database into the flat
> files format. I encountered the problem when the HPL
> unloading a table that having ~33 mil rows into
> multiple files. However, this problem only happen
> intermittently. The error that I got shown as below:
>
> Fri Jun 27 09:16:02 2003
>
> 09:16:02 dynamically allocated new shared memory
> segment (size 4194304)
>
> 09:16:02 dynamically allocated new shared memory
> segment (size 4194304)
>
> 09:16:02 out of virtual shared memory
>
> 09:16:02 dynamically allocated new shared memory
> segment (size 4194304)
>
> 09:16:02 dynamically allocated new shared memory
> segment (size 4194304)
>
> 09:16:02 dynamically allocated new shared memory
> segment (size 4194304)
>
> 09:16:02 dynamically allocated new shared memory
> segment (size 4194304)
>
> and the log file on that particular table shown as
> below:
>
> Fri Jun 27 09:16:00 2003
>
> SHMBASE 0x50000000> CLIENTNUM 0x49010000
> Session ID 2
>
> Unload Database -> prodmac
> Query Name -> AUTO.2
> Device Array -> c_obs
> Query Mapping -> AUTO.2
> Query -> select * from c_obs where obsdate >
> today - 365 for read only
> Convert Reject -> /tmp/c_obs.rej
> Cannot allocate shared memory
>
> Database Unload Completed -- Unloaded 0 Records
> Detected 0 Errors
> Fri Jun 27 09:16:02 2003
>
>
> The above error log was telling me the shared memory
> was not enough. But, the system still have plenty of
> memory while the HPL is running. I even tried to
> increase the shminfo_shmmax from 256MB to 512MB, but
> it doesn't help at all. Does anyone out there know how
> to resolve this problem?
>
> Here's my current plconfig and /etc/system files
>
> # System Configuration
>
> CONVERTTHREADS 1 # Number of conversion threads
> per device
> CONVERTVPS 4 # Max number of vps for
> converters (total)
>
> # Buffer Configuration
>
> STRMBUFFSIZE 1024 # Buffer size for server
> stream buffer (kbytes)
> STRMBUFFERS 5 # Number of server stream
> buffers per device
> AIOBUFSIZE 256 # Buffer size for tape/file
> I/O (kbytes)
> AIOBUFFERS 5 # Number of buffers for I/O
> per device
>
> /etc/system file:
> =================
> set shmsys:shminfo_shmmax=536870912
> set semsys:seminfo_semmap=256
> set semsys:seminfo_semmni=4096
> set semsys:seminfo_semmns=4096
> set semsys:seminfo_semmnu=4096
> set semsys:seminfo_semume=64
> set semsys:seminfo_semmsl=100
> set shmsys:shminfo_shmmin=100
> set shmsys:shminfo_shmmni=500
> set shmsys:shminfo_shmseg=100
>
>
> Thanks & regards,
> Jason
>
Here's my onconfig file
SHMVIRTSIZE 147456
SHMADD 8192
SHMTOTAL 0
From the onstat -g seg, it doesn't shown any
additional virtual segment. In fact, the current
virtual segment still has plenty of memory.
Informix Dynamic Server Version 7.31.UC2 -- On-Line
-- Up 1 days 17:21:18 -- 316208 Kbytes
Segment Summary:
id key addr size ovhd class
blkused blkfree
3500 1381386241 50000000 172802048 3496 R
21087 7
3501 1381386242 5a4cc000 150994944 2900 V
2899 15533
3502 1381386243 634cc000 606208 608 M
67 7
Total: - - 324403200 - -
24053 15547
(* segment locked in memory)
--- "Murray Wood...." <ifxmaillist@quanta.co.nz>
wrote:
> It relates to SHMVIRTSIZE and SHMADD in your
> $ONCONFIG.
> Your SHMADD is unusually small.
>
> Monitor the results with onstat -g seg
>
>
> > -----Original Message-----
> > From: forum.subscriber@iiug.org
> [mailto:forum.subscriber@iiug.org]On
> > Behalf Of JASON PANG
> > Sent: Wednesday, 2 July 2003 3:26 p.m.
> > To: ids@iiug.org
> > Subject: HPL Unloading error [1466]
> >
> >
> > Dear all,
> >
> > OS: Solaris 2.7
> > Memory : 2048MB
> > CPU : 4
> > Engine Version : IDS 7.31.UC2
> >
> > I'm using HPL to unload my database into the flat
> > files format. I encountered the problem when the
> HPL
> > unloading a table that having ~33 mil rows into
> > multiple files. However, this problem only happen
> > intermittently. The error that I got shown as
> below:
> >
> > Fri Jun 27 09:16:02 2003
> >
> > 09:16:02 dynamically allocated new shared memory
> > segment (size 4194304)
> >
> > 09:16:02 dynamically allocated new shared memory
> > segment (size 4194304)
> >
> > 09:16:02 out of virtual shared memory
> >
> > 09:16:02 dynamically allocated new shared memory
> > segment (size 4194304)
> >
> > 09:16:02 dynamically allocated new shared memory
> > segment (size 4194304)
> >
> > 09:16:02 dynamically allocated new shared memory
> > segment (size 4194304)
> >
> > 09:16:02 dynamically allocated new shared memory
> > segment (size 4194304)
> >
> > and the log file on that particular table shown as
> > below:
> >
> > Fri Jun 27 09:16:00 2003
> >
> > SHMBASE 0x50000000> > CLIENTNUM 0x49010000
> > Session ID 2
> >
> > Unload Database -> prodmac
> > Query Name -> AUTO.2
> > Device Array -> c_obs
> > Query Mapping -> AUTO.2
> > Query -> select * from c_obs where
> obsdate >
> > today - 365 for read only
> > Convert Reject -> /tmp/c_obs.rej
> > Cannot allocate shared memory
> >
> > Database Unload Completed -- Unloaded 0 Records
> > Detected 0 Errors
> > Fri Jun 27 09:16:02 2003
> >
> >
> > The above error log was telling me the shared
> memory
> > was not enough. But, the system still have plenty
> of
> > memory while the HPL is running. I even tried to
> > increase the shminfo_shmmax from 256MB to 512MB,
> but
> > it doesn't help at all. Does anyone out there know
> how
> > to resolve this problem?
> >
> > Here's my current plconfig and /etc/system files
> >
> > # System Configuration
> >
> > CONVERTTHREADS 1 # Number of conversion
> threads
> > per device
> > CONVERTVPS 4 # Max number of vps for
> > converters (total)
> >
> > # Buffer Configuration
> >
> > STRMBUFFSIZE 1024 # Buffer size for server
> > stream buffer (kbytes)
> > STRMBUFFERS 5 # Number of server stream
> > buffers per device
> > AIOBUFSIZE 256 # Buffer size for
> tape/file
> > I/O (kbytes)
> > AIOBUFFERS 5 # Number of buffers for
> I/O
> > per device
> >
> > /etc/system file:
> > =================
> > set shmsys:shminfo_shmmax=536870912
> > set semsys:seminfo_semmap=256
> > set semsys:seminfo_semmni=4096
> > set semsys:seminfo_semmns=4096
> > set semsys:seminfo_semmnu=4096
> > set semsys:seminfo_semume=64
> > set semsys:seminfo_semmsl=100
> > set shmsys:shminfo_shmmin=100
> > set shmsys:shminfo_shmmni=500
> > set shmsys:shminfo_shmseg=100
> >
> >
> > Thanks & regards,
> > Jason
> >
>
__________________________________
Do you Yahoo!?
SBC Yahoo! DSL - Now only $29.95 per month!
http://sbc.yahoo.com
Hi Miyaki,
Problem is that shared memory in pload process space has run out of
segments.
You can check the pload segments using the following command.
onstat -j <Complete path of pload log file>then do seg at the promt
onstat> seg
This will show pload process segment list.
What you are seeing is the server process segment list.
Also why are you on such a old 7.31.UC2 release.
Please upgrade to the latest 7.31 or 9.3 release.
Let me also know how many files you are trying to unload to.
Also try unloading to just single device and see if it still hangs.
Also let me know if you are unloading to a delimited or fixed format.
Thanks,
Pravin.
|---------+---------------------------->
| | miyaki |
| | <lcib@yahoo.com> |
| | Sent by: |
| | owner-informix-li|
| | st@iiug.org |
| | |
| | |
| | 07/02/2003 12:08 |
| | AM |
| | |
|---------+---------------------------->
>-------------------------------------------------------------------------------
------------------------------------------------|
| |
| To: ids@iiug.org, informix-list@iiug.org |
| cc: |
| Subject: RE: HPL Unloading error [1467] |
| |
>-------------------------------------------------------------------------------
------------------------------------------------|
Here's my onconfig file
SHMVIRTSIZE 147456
SHMADD 8192
SHMTOTAL 0
From the onstat -g seg, it doesn't shown any
additional virtual segment. In fact, the current
virtual segment still has plenty of memory.
Informix Dynamic Server Version 7.31.UC2 -- On-Line
-- Up 1 days 17:21:18 -- 316208 Kbytes
Segment Summary:
id key addr size ovhd class
blkused blkfree
3500 1381386241 50000000 172802048 3496 R
21087 7
3501 1381386242 5a4cc000 150994944 2900 V
2899 15533
3502 1381386243 634cc000 606208 608 M
67 7
Total: - - 324403200 - -
24053 15547
(* segment locked in memory)
--- "Murray Wood...." <ifxmaillist@quanta.co.nz>
wrote:
> It relates to SHMVIRTSIZE and SHMADD in your
> $ONCONFIG.
> Your SHMADD is unusually small.
>
> Monitor the results with onstat -g seg
>
>
> > -----Original Message-----
> > From: forum.subscriber@iiug.org
> [mailto:forum.subscriber@iiug.org]On
> > Behalf Of JASON PANG
> > Sent: Wednesday, 2 July 2003 3:26 p.m.
> > To: ids@iiug.org
> > Subject: HPL Unloading error [1466]
> >
> >
> > Dear all,
> >
> > OS: Solaris 2.7
> > Memory : 2048MB
> > CPU : 4
> > Engine Version : IDS 7.31.UC2
> >
> > I'm using HPL to unload my database into the flat
> > files format. I encountered the problem when the
> HPL
> > unloading a table that having ~33 mil rows into
> > multiple files. However, this problem only happen
> > intermittently. The error that I got shown as
> below:
> >
> > Fri Jun 27 09:16:02 2003
> >
> > 09:16:02 dynamically allocated new shared memory
> > segment (size 4194304)
> >
> > 09:16:02 dynamically allocated new shared memory
> > segment (size 4194304)
> >
> > 09:16:02 out of virtual shared memory
> >
> > 09:16:02 dynamically allocated new shared memory
> > segment (size 4194304)
> >
> > 09:16:02 dynamically allocated new shared memory
> > segment (size 4194304)
> >
> > 09:16:02 dynamically allocated new shared memory
> > segment (size 4194304)
> >
> > 09:16:02 dynamically allocated new shared memory
> > segment (size 4194304)
> >
> > and the log file on that particular table shown as
> > below:
> >
> > Fri Jun 27 09:16:00 2003
> >
> > SHMBASE 0x50000000> > CLIENTNUM 0x49010000
> > Session ID 2
> >
> > Unload Database -> prodmac
> > Query Name -> AUTO.2
> > Device Array -> c_obs
> > Query Mapping -> AUTO.2
> > Query -> select * from c_obs where
> obsdate >
> > today - 365 for read only
> > Convert Reject -> /tmp/c_obs.rej
> > Cannot allocate shared memory
> >
> > Database Unload Completed -- Unloaded 0 Records
> > Detected 0 Errors
> > Fri Jun 27 09:16:02 2003
> >
> >
> > The above error log was telling me the shared
> memory
> > was not enough. But, the system still have plenty
> of
> > memory while the HPL is running. I even tried to
> > increase the shminfo_shmmax from 256MB to 512MB,
> but
> > it doesn't help at all. Does anyone out there know
> how
> > to resolve this problem?
> >
> > Here's my current plconfig and /etc/system files
> >
> > # System Configuration
> >
> > CONVERTTHREADS 1 # Number of conversion
> threads
> > per device
> > CONVERTVPS 4 # Max number of vps for
> > converters (total)
> >
> > # Buffer Configuration
> >
> > STRMBUFFSIZE 1024 # Buffer size for server
> > stream buffer (kbytes)
> > STRMBUFFERS 5 # Number of server stream
> > buffers per device
> > AIOBUFSIZE 256 # Buffer size for
> tape/file
> > I/O (kbytes)
> > AIOBUFFERS 5 # Number of buffers for
> I/O
> > per device
> >
> > /etc/system file:
> > =================
> > set shmsys:shminfo_shmmax=536870912
> > set semsys:seminfo_semmap=256
> > set semsys:seminfo_semmni=4096
> > set semsys:seminfo_semmns=4096
> > set semsys:seminfo_semmnu=4096
> > set semsys:seminfo_semume=64
> > set semsys:seminfo_semmsl=100
> > set shmsys:shminfo_shmmin=100
> > set shmsys:shminfo_shmmni=500
> > set shmsys:shminfo_shmseg=100
> >
> >
> > Thanks & regards,
> > Jason
> >
>
__________________________________
Do you Yahoo!?
SBC Yahoo! DSL - Now only $29.95 per month!
http://sbc.yahoo.com
sending to informix-list
Hi, Jason,
AFAIK HPL allocates it's own shared memory segments
not related to IDS segments.
These segments are used by 'onpload' procesess
for IPC (to coordinate load/unload jobs into multiple streams)
Usually You need to reserve proper amount of RAM for
HPL processes (I usually reserve 100MB)
------------------------------------------
Alexey Sonkin
Senior Database Administrator
> -----Original Message-----
> From: JASON PANG [mailto:jason.yl.pang@intel.com]
> Sent: Tuesday, July 01, 2003 11:26 PM
> To: ids@iiug.org
> Subject: HPL Unloading error [1466]
>
>
> Dear all,
>
> OS: Solaris 2.7
> Memory : 2048MB
> CPU : 4
> Engine Version : IDS 7.31.UC2
>
> I'm using HPL to unload my database into the flat
> files format. I encountered the problem when the HPL
> unloading a table that having ~33 mil rows into
> multiple files. However, this problem only happen
> intermittently. The error that I got shown as below:
>
> Fri Jun 27 09:16:02 2003
>
> 09:16:02 dynamically allocated new shared memory
> segment (size 4194304)
>
> 09:16:02 dynamically allocated new shared memory
> segment (size 4194304)
>
> 09:16:02 out of virtual shared memory
>
> 09:16:02 dynamically allocated new shared memory
> segment (size 4194304)
>
> 09:16:02 dynamically allocated new shared memory
> segment (size 4194304)
>
> 09:16:02 dynamically allocated new shared memory
> segment (size 4194304)
>
> 09:16:02 dynamically allocated new shared memory
> segment (size 4194304)
>
> and the log file on that particular table shown as
> below:
>
> Fri Jun 27 09:16:00 2003
>
> SHMBASE 0x50000000> CLIENTNUM 0x49010000
> Session ID 2
>
> Unload Database -> prodmac
> Query Name -> AUTO.2
> Device Array -> c_obs
> Query Mapping -> AUTO.2
> Query -> select * from c_obs where obsdate >
> today - 365 for read only
> Convert Reject -> /tmp/c_obs.rej
> Cannot allocate shared memory
>
> Database Unload Completed -- Unloaded 0 Records
> Detected 0 Errors
> Fri Jun 27 09:16:02 2003
>
>
> The above error log was telling me the shared memory
> was not enough. But, the system still have plenty of
> memory while the HPL is running. I even tried to
> increase the shminfo_shmmax from 256MB to 512MB, but
> it doesn't help at all. Does anyone out there know how
> to resolve this problem?
>
> Here's my current plconfig and /etc/system files
>
> # System Configuration
>
> CONVERTTHREADS 1 # Number of conversion threads
> per device
> CONVERTVPS 4 # Max number of vps for
> converters (total)
>
> # Buffer Configuration
>
> STRMBUFFSIZE 1024 # Buffer size for server
> stream buffer (kbytes)
> STRMBUFFERS 5 # Number of server stream
> buffers per device
> AIOBUFSIZE 256 # Buffer size for tape/file
> I/O (kbytes)
> AIOBUFFERS 5 # Number of buffers for I/O
> per device
>
> /etc/system file:
> =================
> set shmsys:shminfo_shmmax=536870912
> set semsys:seminfo_semmap=256
> set semsys:seminfo_semmni=4096
> set semsys:seminfo_semmns=4096
> set semsys:seminfo_semmnu=4096
> set semsys:seminfo_semume=64
> set semsys:seminfo_semmsl=100
> set shmsys:shminfo_shmmin=100
> set shmsys:shminfo_shmmni=500
> set shmsys:shminfo_shmseg=100
>
>
> Thanks & regards,
> Jason
>
You did
not show the Value of SHMTOTAL. This may be the limit you are
hitting.
Informix will report out of shared memory ( regardless of what is available
on the machine ) if you have SHMTOTAL set, and reach that size. This
parameter is most likely the limit you are hitting. If you have the memory
set it to 0. Remember with light scans you are bypassing the buffers and
will require much Virtual memory ( also with light appends.)
George Palmer
-----Original Message-----
From: JASON PANG [mailto:jason.yl.pang@intel.com]
Sent: Tuesday, July 01, 2003 8:26 PM
To: ids@iiug.org
Subject: HPL Unloading error [1466]
Dear all,
OS: Solaris 2.7
Memory : 2048MB
CPU : 4
Engine Version : IDS 7.31.UC2
I'm using HPL to unload my database into the flat
files format. I encountered the problem when the HPL
unloading a table that having ~33 mil rows into
multiple files. However, this problem only happen
intermittently. The error that I got shown as below:
Fri Jun 27 09:16:02 2003
09:16:02 dynamically allocated new shared memory
segment (size 4194304)
09:16:02 dynamically allocated new shared memory
segment (size 4194304)
09:16:02 out of virtual shared memory
09:16:02 dynamically allocated new shared memory
segment (size 4194304)
09:16:02 dynamically allocated new shared memory
segment (size 4194304)
09:16:02 dynamically allocated new shared memory
segment (size 4194304)
09:16:02 dynamically allocated new shared memory
segment (size 4194304)
and the log file on that particular table shown as
below:
Fri Jun 27 09:16:00 2003
SHMBASE 0x50000000CLIENTNUM 0x49010000
Session ID 2
Unload Database -> prodmac
Query Name -> AUTO.2
Device Array -> c_obs
Query Mapping -> AUTO.2
Query -> select * from c_obs where obsdate >
today - 365 for read only
Convert Reject -> /tmp/c_obs.rej
Cannot allocate shared memory
Database Unload Completed -- Unloaded 0 Records
Detected 0 Errors
Fri Jun 27 09:16:02 2003
The above error log was telling me the shared memory
was not enough. But, the system still have plenty of
memory while the HPL is running. I even tried to
increase the shminfo_shmmax from 256MB to 512MB, but
it doesn't help at all. Does anyone out there know how
to resolve this problem?
Here's my current plconfig and /etc/system files
# System Configuration
CONVERTTHREADS 1 # Number of conversion threads
per device
CONVERTVPS 4 # Max number of vps for
converters (total)
# Buffer Configuration
STRMBUFFSIZE 1024 # Buffer size for server
stream buffer (kbytes)
STRMBUFFERS 5 # Number of server stream
buffers per device
AIOBUFSIZE 256 # Buffer size for tape/file
I/O (kbytes)
AIOBUFFERS 5 # Number of buffers for I/O
per device
/etc/system file:
=================
set shmsys:shminfo_shmmax=536870912
set semsys:seminfo_semmap=256
set semsys:seminfo_semmni=4096
set semsys:seminfo_semmns=4096
set semsys:seminfo_semmnu=4096
set semsys:seminfo_semume=64
set semsys:seminfo_semmsl=100
set shmsys:shminfo_shmmin=100
set shmsys:shminfo_shmmni=500
set shmsys:shminfo_shmseg=100
Thanks & regards,
Jason
Related threads
- Posting from the Informix-list
- Migrating from IDS 9.40.UC6 to 11.50.UC3
- Ip for a network session
- questions onstat -g