Informix DataBlade API Programmer's Manual Writing a User-Defined Routine

A C user-defined routine has access to the following types of memory:

• User memory for dynamic allocations
• Stack memory for routine arguments (including the MI_FPARAM structure), local stack variables, return values

The DataBlade API provides functions to manage these types of memory.

The DataBlade API provides memory-management functions to dynamically allocate memory in a C UDR. These functions allocate user memory, which is allocated from database server shared memory. Figure 12-14 shows the memory-management functions that the DataBlade API provides for memory operations on user memory.

Tip: The DataBlade API memory-management functions execute in client LIBMI applications as well as C UDRs. For DataBlade API modules that you design to run in both client LIBMI applications and UDRs, use these memory-management functions. For information on the behavior of these functions in a client LIBMI application, see Appendix A, Writing a Client LIBMI Application.

Allocating User Memory

To handle dynamic memory allocation of user memory, use one of the following DataBlade API memory-management functions.

Tip: The DataBlade API library also provides several private memory-management functions. These private functions are only available to DataBlade partners for special DataBlade functionality. For more information on private memory-management durations, see your Informix sales representative.

These DataBlade API memory-management functions work correctly with the transaction management and garbage collection of the database server. In particular, they provide the following advantages:

• These functions allocate the user memory from shared memory so that all VPs can access it.
• The database server automatically performs garbage collection on memory allocated with these functions.

These functions establish a memory duration for the memory they allocate. When this memory duration expires, the database server automatically marks the memory for garbage collection.

While a C UDR executes on a VP (VP #1), it can access memory that is part of that virtual process. Figure 12-15 shows a schematic representation of what a virtual processor that loads a shared-object file looks like internally.

Figure 12-15 Memory Allocated for a C UDR

However, if the UDR migrates to another VP, it no longer has access to any information in the memory space of VP #1. It can now only access the memory in the new VP (VP #2). The only memory that the UDR can access from both VP #1 and VP #2 is the database server shared memory. This restriction leads to the following guidelines for the dynamic memory allocation in a C UDR:

• The C UDR must be thread safe.

It must not assume that it can access information that is stored in VP memory. This guideline is part of the requirements for a well-behaved UDR. For more information, see Creating a Well-Behaved Routine.

• To dynamically allocate memory, the UDR must use the DataBlade API memory-management functions to allocate user memory.

Thee DataBlade API memory-management functions (Figure 12-14 on page 12-54) allocate user memory from the database server shared memory, not from the memory of a virtual process.

The DataBlade API memory-management functions allocate user memory from the shared memory of the database server. The database server organizes this shared memory into memory pools. Keeping related memory allocations in one pool helps to reduce memory fragmentation.

Tip: For more information about the use and structure of database server memory pools, see your Administrator's Guide.

As Figure 12-16 shows, the DataBlade API memory-management functions (such as mi_alloc() and mi_dalloc()) allocate memory from shared memory, which remains accessible if a thread migrates to another virtual processor. The system memory-management functions (such as malloc() and free()) allocate memory in the heap space of the VP. If a thread migrates to another VP, it no longer has access to the heap space of the previous VP. Therefore, the address to dynamic memory in some variable is not valid while the UDR executes in the new VP.

Figure 12-16 Location of Dynamically Allocated Memory for a C UDR

A C UDR must allocate user memory from the shared memory of the database server, not from the memory of the virtual processor that runs the C UDR. Therefore, a C UDR must use the DataBlade API memory-management functions to manage user memory.

A memory duration specifies the lifetime of the user memory. The DataBlade API memory-management functions allocate user memory from database server shared memory, which is organized into memory pools. Each memory duration has a separate memory pool.

When the database server calls a UDR, it creates a memory context. This context records all of the allocations that the UDR makes before the routine returns. The UDR might run for some time, calling other user-defined routines or DataBlade API functions. The database server automatically performs garbage collection on user memory based on its memory duration. When a particular memory duration expires, the database server marks the associated memory pool for deallocation.

In a C UDR, you can allocate shared memory with any of the public memory durations in Figure 12-17.

Tip: The DataBlade API memory-management functions also support several private memory durations. These private memory durations are only available to DataBlade module partners for special DataBlade module functionality. For more information on private memory durations, see your Informix sales representative.

These memory-duration constants are of type MI_MEMORY_DURATION. The memdur.h header file declares these memory-duration constants.

Make sure you choose a memory duration that is appropriate to the use of the user memory. An inappropriate memory duration can cause the following problems:

• If you allocate memory with a duration that is too small, expect to see assertion failures in the memory log file.

For example, if you allocate PER_ROUTINE memory and store its pointer in the MI_FPARAM structure (which as a PER_COMMAND duration), the memory might be freed after one invocation of the UDR and the pointer in the MI_FPARAM is no longer valid.

• If you allocate memory with a duration that is too large, you might see memory leaks as your UDR executes.

Memory leakage can occur when you assign user memory that has a higher duration than the structure that holds its address. For more information, see Monitoring User Memory.

Important: It is important to keep track of the scope and memory duration of the memory you allocate with the DataBlade API memory-management functions. Incorrect memory duration can create serious memory corruption.

The following sections provide additional information about each of the valid memory durations in Figure 12-17 on page 12-58.

The current memory duration is the memory duration that applies when the mi_alloc() or mi_zalloc() function allocates memory. These functions do not specify a memory duration for their allocation. The default memory duration is PER_ROUTINE; that is, the database server marks the memory for garbage collection when the UDR completes. By default, the current memory duration is default memory duration. Therefore, the mi_alloc() and mi_zalloc() functions allocate memory with a duration of PER_ROUTINE by default.

Figure 12-18 shows the DataBlade API data structures that have a memory duration of the current memory duration.

You can change the current memory duration that these functions use with the mi_switch_mem_duration() function.

To increase the memory duration of a DataBlade API data structure, call the mi_switch_mem_duration() function with the desired duration before the DataBlade API function call that allocates the object. For more information, see Changing the Memory Duration.

Important: All the DataBlade API functions in Figure 12-18 allocate structures with the current memory duration. If you switch the current memory duration, you affect not only explicit allocations you make with mi_alloc() or mi_zalloc() but the memory allocations that these DataBlade API constructor functions do as well. PER_ROUTINE Memory Duration

Memory that is allocated with the PER_ROUTINE memory duration has a duration of a single invocation of the UDR.

Tip: The two memory-duration constants PER_ROUTINE and PER_FUNCTION are synonyms for the same memory duration. PER_ROUTINE is the more current name.

The database server performs garbage collection on PER_ROUTINE memory when a single invocation of a UDR completes. At this time, the database server reclaims any PER_ROUTINE memory in the memory context that is marked with the PER_ROUTINE duration. This memory is actually freed on entry to the next routine invocation. The database server does not reclaim any memory in the memory context with a higher duration than PER_ROUTINE. Therefore, the PER_ROUTINE memory duration causes the database server to free memory automatically when a UDR invocation completes, if the UDR did not already free the memory.

The PER_ROUTINE memory duration is useful for user memory required for a single UDR invocation. A UDR cannot allocate memory, save a pointer to this memory in static space, and expect the pointer to be valid for the next routine invocation. To save information across invocations, use the function-state pointer of the MI_FPARAM structure. For more information, see Saving a User State.

The routine manager allocates memory with a PER_ROUTINE duration for UDR pass-by-reference arguments that are pass-by-reference and for the UDR return value.

The default memory duration is PER_ROUTINE. The current memory duration is initialized to this default memory duration. For more information, see Current Memory Duration.

Memory that is allocated with the PER_COMMAND memory duration has a duration of the current SQL command. An SQL command is an SQL statement (such as a subquery) that was initiated as part of the current client SQL statement. Examples of SQL commands include:

• A PREPARE statement
• A subquery (a SELECT statement)
• The SQL statement that a DataBlade API statement-execution function (mi_exec(), mi_exec_prepared_statement(), or mi_open_prepared_statement()) executes
• Implicit UDR calls

An implicit UDR is a UDR that the database server calls automatically in response to some SQL task. For example, in the following SELECT statement, the database server calls the a_to_int() cast function when it executes the SELECT:

CREATE IMPLICIT CAST (a AS INTEGER WITH a_to_int);

...

SELECT a:int FROM tab1 WHERE b > 6

Memory allocations within the a_to_int() function that have a PER_COMMAND duration are available

One client SQL statement might initiate several SQL commands, which the PREPARE statement or one of the DataBlade API execution functions executes. The database server marks any PER_COMMAND memory for garbage collection when the current SQL command completes.

For a query, an SQL command completes when the subquery finishes execution for one outer row of the outer query. The only exception to this rule is if this statement is a cursor statement (DECLARE, OPEN, FETCH, UPDATE...WHERE CURRENT OF or DELETE...WHERE CURRENT OF, CLOSE), in which case the database server frees the memory when the cursor closes. For a routine sequence, an SQL command completes when the routine instance is complete; that is, when every routine invocation in the routine sequence is complete.

The PER_COMMAND memory duration is useful for accumulating calculations, in iterator functions, and for initialization of expensive resources. To retain user state across a routine instance, a user-defined routine can allocate PER_COMMAND memory and store it in the MI_FPARAM structure. For example, if a UDR calculates a total, PER_ROUTINE memory would be available only for a single routine invocation; PER_COMMAND memory would be available for the entire routine instance, regardless of the number of invocations involved. For more information on the use of MI_FPARAM, see Saving a User State.

Figure 12-19 shows the DataBlade API data structures that have a memory duration of PER_COMMAND.

Switching the current memory duration before one of these constructor functions does not have an effect on the memory duration of the allocated DataBlade API structure. To retain access to some of these DataBlade API data structures after the command completes, you must save them at the per-session level.

Tip: The DataBlade API supports the ability to save information at a per-session level. However, this ability is a controlled feature of the DataBlade API. For information on how to obtain more information on this feature, contact your Informix sales representative. PER_STATEMENT Memory Duration

Memory that is allocated with the PER_STATEMENT memory duration has a duration of the current SQL statement. A statement is the entire SQL statement plus any SQL commands that the SQL statement initiates. Examples of SQL statements include:

• An SQL statement that the client application invokes
• An SPL statement that an SPL routine invokes
• The duration of a statement descriptor (from the mi_prepare() that creates it up to the mi_drop_prepared_statement() that frees it)

One SQL statement might contain several SQL commands. If the statement does not contain any commands, the database server deallocates PER_STATEMENT memory at the same time it deallocates PER_COMMAND memory.

The database server marks any PER_STATEMENT shared memory for garbage collection when the current client SQL statement completes or when every routine sequence in the SQL statement is complete.

The PER_STATEMENT memory duration is useful for communication between UDRs, parallel execution, user-defined aggregates, named memory, and for memory allocations within an end-of-statement callback (if you have information to pass to the callback).

Figure 12-20 shows that the DataBlade API data structures have a memory duration of PER_STATEMENT.

Switching the current memory duration before one of these constructor functions does not have an effect on the memory duration of the allocated DataBlade API structure. To retain access to some of these DataBlade API data structures after the command completes, you must save them at the per-session level.

Tip: The DataBlade API supports the ability to save information at a per-session level. However, this ability is a controlled feature of the DataBlade API. For information on how to obtain more information on this feature, contact your Informix sales representative. Changing the Memory Duration

The PER_ROUTINE memory duration is the default protection on a region of database server shared memory. You can change the memory duration of user memory to another duration in either of the following ways:

• Use mi_dalloc() instead of mi_alloc() to allocate memory.

The mi_dalloc() function works in the same way as mi_alloc() but provides the option of specifying the memory duration. This function does not switch the current memory duration.

• Call mi_switch_mem_duration() before you call mi_alloc().

The mi_switch_mem_duration() function switches the current memory duration. All user-memory allocations by subsequent calls to mi_alloc() or mi_zalloc() have the new current memory duration.

Changing the current memory duration with mi_switch_mem_duration() has an effect on the memory durations of all DataBlade API data structures that Figure 12-18 on page 12-60 lists. It does not have an effect on the memory duration of DataBlade API data structures allocated at the PER_COMMAND (Figure 12-19 on page 12-63) and PER_STATEMENT (Figure 12-20 on page 12-64) durations.

The mi_switch_mem_duration() function returns the previous memory duration. You can use this return value to restore memory duration after performing some allocations at a different duration.

The following code fragment temporarily changes the current memory duration from PER_ROUTINE (the default) to PER_COMMAND:

/* Switch current memory duration to PER_COMMAND and save
  * old current memory duration in 'old_mem_dur' */
old_mem_dur = mi_switch_mem_duration(PER_COMMAND);

/* Perform allocations a new current memory duration */
buffer = (char *)mi_alloc(BUFF_SIZE);
new_lvarch = mi_new_var(BUFF_SIZE-1);
save_set = mi_save_set_create(conn);

/* Restore old current memory duration */
(void)mi_switch_mem_duration(old_mem_dur);

In the preceding code fragment, the PER_COMMAND memory duration is in effect for the allocation of user memory that the call to mi_alloc() makes. Because the mi_new_var() function allocates a new varying-length structure in the current memory duration, this call to mi_new_var() allocates the varying-length structure with a PER_COMMAND duration. However, the mi_save_set_create() function does not allocate its save-set structure at the current memory duration. Therefore, the call to mi_save_set_create() still allocates its save-set structure with the PER_STATEMENT duration. The second call to mi_switch_mem_duration() restores the current memory duration to PER_ROUTINE.

The database server automatically performs garbage collection on the user memory that mi_alloc(), mi_dalloc(), and mi_zalloc() allocate. The memory duration of the user memory determines when the database server marks the memory for deallocation.

User memory remains valid until whichever of the following events occurs first:

• The mi_free() function frees the memory.
• The memory duration expires.
• The mi_close() function closes the current connection.
• A database server exception is raised.

A C UDR is not allowed to cache information from the database across transaction boundaries. Because the state of the database might change entirely when the current transaction commits, any cached information might be invalid. Therefore, UDRs must reinitialize any database state that they require when the next transaction begins. To enforce the policy of no caching across transactions, the database server automatically performs garbage collection of memory at transaction boundaries. In addition, the database server performs garbage collection when specified memory durations expire, usually when a UDR allocates and returns a value.

To conserve resources, use the mi_free() function to explicitly deallocate the user memory once your DataBlade API module no longer needs it. The mi_free() function is the destructor function for user memory.

Keep the following restrictions in mind about memory deallocation:

• Do not free user memory that you allocate for the return value of a UDR.
• Do not free memory until you are finished accessing the memory.
• Do not use mi_free() to deallocate memory that you have not explicitly allocated.
• Do not use mi_free() for data structures that other DataBlade API constructor functions allocate.
• Do not attempt to free user memory after its memory duration has expired.
• Reuse memory whenever possible. Do not repeat calls to allocation functions if you can reuse the memory for another task.

Memory leakage occurs when memory remains allocated after the structure that holds its address was deallocated. There is no way to access the memory once its address is gone. Therefore, the memory remains with no way to remove it. You can monitor use of memory that your UDR allocates (explicitly or implicitly) with the following options of the onstat utility:

• The -g ses option outputs session information.

You can specify a particular session identifier after the ses keyword. If you do not include a session identifier, onstat -g ses outputs a one-line summary for each active session. Look for any session whose memory allocation or usage steadily increases.

• The -g mem option outputs statistics for a memory pool.

You can specify a particular pool name after the mem keyword. If you do not include a pool name, onstat -g mem outputs a one-line summary for each memory pool. Look for any pool whose memory allocation or usage steadily increases.

Tip: You can use the -r option of onstat in conjunction with any of the options listed above to have onstat repeat the command every specified number of seconds.

Suppose that the onstat -g ses command produces the following sample output:

session #RSAM total used

id user tty pid hostname threads memory memory

24 informix - 0 - 0 8192 5880

23 informix - 13453 bison 1 6701056 6608512

8 informix - 0 - 0 8192 5880

7 informix - 0 - 0 16384 14344

6 informix - 0 - 0 8192 5880

4 informix - 0 - 0 16384 14344

3 informix - 0 - 0 8192 5880

2 informix - 0 - 0 8192 5880

Suppose also that your DB-Access session is hooked to session 23 (the italicized line in the preceding sample output).

You can determine the session identifier of the DB-Access session on a UNIX system with the following command:

ps auxw | grep 13453

You can now obtain information about memory-pool usage with the
-g mem option of onstat:

onstat -g mem

Suppose that the preceding onstat command generated the following sample output (some output lines are omitted for brevity):

Pool Summary:
name			class		addr
resident			R		a002018
res-buff			R		a118018
global			V		a18a018
mt			V		a18e018
smartblob			V		a192018
...
23			V		a3e0018
24			V		a3e2018
23.RTN.SAPI			V		a40c018
23.CMD.SAPI			V		a3ee018
...

Figure 12-21 shows the lines of the onstat -g mem output that indicate user memory allocations.

Figure 12-21 Memory Pools in onstat -g mem Output

Each memory duration has a separate memory pool. The three letters before "SAPI" identify each memory pool. The following table shows the memory-pool names for the public memory durations.

Tip: The onstat -g mem output also shows memory pools for the private memory durations. These private memory durations are only available to DataBlade partners for special DataBlade functionality. For more information on private memory durations, see your Informix sales representative.

Once you have determined a specific session identifier or memory-pool name that exhibits a problem, you can find out which specific kind of memory is affected with the -g ufr option of onstat. The -g ufr option of onstat shows memory fragments by usage. For example, the following onstat command captures a snapshot every 30 seconds of memory pools that session 23 uses:

onstat -g ufr 23 -r 30

Sample output for the preceding command follows:

Memory usage for pool name 23:
size		memid
2152		log
2016		ostcb
2600		sqtcb
8472		gentcb
1664		osenv
6392648		sqscb
792		filetable
112		rdahead
120		overhead
96		scb
416		sapi
3296		fragman
18808		opentable
280		hashfiletab
10056		temprec
592		GenPg
224		ru
56		sort
94848		ralloc

In the preceding sample output, the main memory consumer is sqscb.

Any memory leakage from a DataBlade API function would show up in the memid column entry labelled sapi. For more information on the onstat command, see your Administrator's Reference.

Session threads execute C UDRs and their thread-stack space is allocated from a common region in shared memory. The thread stack stores nonshared data for the UDRs and system routines that the thread executes. This stack contains everything that would normally be on the execution stack, including the following:

• Routine arguments, including the MI_FPARAM structure
• Local (stack) variables
• Function return values

Like all memory that UDRs use, stack segments can be overrun. The database server can only check for stack violations when the user-defined routine yields. Therefore, you must ensure that you perform the following tasks within a UDR:

• Efficiently manage stack space usage in your UDR

Limit usage of stack space in your UDR and ensure sufficient stack-space allocation when you register the user-defined routine.

• Use the mi_call() function for user-defined routines that potentially use unlimited recursion.

To avoid stack overflow, follow these design restrictions in your UDR:

• Do not use large automatic arrays.
• Avoid excessively deep calling sequences.
• Use mi_call() within a UDR to manage recursive calls.

By default, when a thread of a virtual processor executes a UDR, the database server uses a thread-stack size that the STACKSIZE configuration parameter specifies (32 kilobytes, if STACKSIZE is not set). To determine how much stack space a UDR requires, monitor the UDR from the system prompt with the following onstat command:

onstat -g sts

The -g sts option prints the maximum and current stack usage per thread. For more information on the onstat utility and its -g sts option, see your Administrator's Reference.

You must ensure that your UDR has sufficient stack space for its execution. That is, the UDR must have enough stack space to hold all local variables of the routine. If you see errors in the message-log file of the following format when you try to allocate a large block of memory, your stack space is being overrun:

Assert Failed: Condition Failed (Bad pool pointer 0xe2fe018),
in (mt_shm_free)

Assert Failed: Memory block header corruption detected in 
mt_shm_free

To override the stack size for a particular UDR, use the STACK routine modifier of the CREATE FUNCTION or CREATE PROCEDURE statement when you register your user-defined routine. For example, the following CREATE FUNCTION statement specifies a stack size of 64 kilobytes for the func1() UDR:

CREATE FUNCTION func1(INTEGER)

	RETURNS INTEGER

	WITH (STACK=64,000)

	EXTERNAL NAME '/usr/srv_routs/funcs.so'

	LANGUAGE C

When the UDR completes, the database server allocates thread stacks for subsequent UDRs based on the STACKSIZE parameter (unless these subsequent UDRs have also specified the STACK routine modifier).

The DataBlade API provides the mi_call() function to dynamically manage stack space. This function performs the following tasks:

• It checks the amount of unused stack space and allocates additional stack segments if necessary.
• It executes the specified UDR.

Use the mi_call() function to increase stack space for recursive UDRs.

Keep in mind that mi_call() does not know the size of the routine arguments. When mi_call() creates a new stack and copies the arguments onto this new stack, it assumes that each argument is the size of an MI_DATUM. If the data type of the routine argument is larger than an MI_DATUM, mi_call() does not copy all the argument bytes.

For example, consider a UDR that includes an mi_double argument.

On a Sun Sparc system, an mi_double_precision argument takes the space of two long int values. Therefore, the mi_call() function pushes only half of the argument onto the new stack. Any arguments after the mi_double_precision might get garbled, and the last one might be truncated.

When you design UDRs that require use of mi_call(), make sure you use the correct passing mechanism for the argument data type. Pass all data types larger than MI_DATUM by reference. Examples of large data types include floating-point types (such as mi_real, mi_double_precision) and data structures.

The following example illustrates stack-space allocation with mi_call():

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include "mi.h"

mi_integer factorial(mi_integer value)
{
	mi_integer			callstatus=0,
				retval=0;

	if ( value < 0 )
		return -1;
	else if ( value == 1 || value == 0 )
		return 1;
	else if ( value > 30 )
		mi_db_error_raise(NULL, MI_EXCEPTION,
	"factorial: input value too big for a long int result.");

	callstatus = mi_call(&retval, factorial, 1, value-1);
	switch ( callstatus )
		{
		case MI_TOOMANY:
			mi_db_error_raise(NULL, MI_EXCEPTION,
				"factorial: too many parameters.");

		case MI_CONTINUE:
			return (value * factorial(value-1));

		case MI_NOMEM:
			mi_db_error_raise(NULL, MI_EXCEPTION,
				"factorial: not enough memory");

		case MI_DONE:
			/* At the end of the factorial recursion, the
			** function still needs to calculate:
			**      value * factorial(value-1)
			*/
			retval *= value;
			break;
		}

	return retval;
}

This code sample implements a factorial function. If the mi_call() function determines that there is sufficient stack space, the code recursively calls the handle_row() function to process the row value. The return value of the mi_call() function indicates whether mi_call() has allocated additional thread-stack memory, as follows.

factorial(value-1)

value * factorial(value-1)

The other mi_call() return values (MI_NOMEM and MI_TOOMANY) indicate error conditions. For these return values, the function uses the mi_db_error_raise() function to raise a database server exception and provide an error message.

The following CREATE FUNCTION registers the factorial() function:

CREATE FUNCTION factorial (INTEGER)
	RETURNS INTEGER
	EXTERNAL NAME
		"$INFORMIXDIR/extend/misc/fact_ius.bld"
	LANGUAGE C

The following EXECUTE FUNCTION invokes the factorial() function:

EXECUTE FUNCTION factorial(5);