Slow indexing with BTS

Hello list, We have an application in which we use a free text search engine as an integrated part. The application hold mebership information for a large (>7 million members) organisation. The search engine supports free text searches for names, addresses and a lot more. Basically we have five tables each holding a lvarchar column containing a xml-record. Each xml-record conintains 12-20 xml-fields, all togetgher 250-500 characters long. We are using a search engine which is integrated with IDS as a data blade, the index and the engine are external. We are now considering to change this old search engine for BTS because the old product does not have a proper support any longer. The reason for this post is that our first try with a full scale BTS index is not very successfull. Whith the old search engine it take 3,5 minutes to create an index for a search table with 320 000 records (2 minutes for the index, 1,5 minute for a index reorg which is mandatory after index creation). To search the BTS index seems to be resonably fast (I have so far only made very few searches) but: The BTS index took a little more than 9 hours to create! It would be a nigthmare to create an index on the largest table with more that 8 millikon records... Does anybody have experiance using BTS? Is it possible to speed up the index creation in some way? (IDS 11.50UC7, BTS 2.00, BTS index in sbspace, all the database on raw disk. OS: Solaris 10 on Sparc) Best regards, Rolf Wasteson

Hi Rolf, BTS does not have the fastest index build and I do keep working on that, but I have not seen performance like this. A couple of things to check, a) Do you have SBSPACETEMP set to a temporary sbspace (ie an sbspace created with the -t option). This space should be twice the size of the text being index. b) With an index placed in an sbspace, you will see much more activity in the buffer pool. Have you looked at how the buffer pool is performing. Consider tuning for BUFFERPOOL. You can also look tuning RA_PAGES and RA_THRESHOLD c) XML based indexes can use virtual memory. Is may help to set the onconfig variable RESIDENT on? Outgoing email check by Norton AntiVirus -----Original Message----- From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of ROLF WASTESON Sent: June 18, 2010 5:20 AM To: ids@iiug.org Subject: Slow indexing with BTS [20427] Hello list, We have an application in which we use a free text search engine as an integrated part. The application hold mebership information for a large (>7 million members) organisation. The search engine supports free text searches for names, addresses and a lot more. Basically we have five tables each holding a lvarchar column containing a xml-record. Each xml-record conintains 12-20 xml-fields, all togetgher 250-500 characters long. We are using a search engine which is integrated with IDS as a data blade, the index and the engine are external. We are now considering to change this old search engine for BTS because the old product does not have a proper support any longer. The reason for this post is that our first try with a full scale BTS index is not very successfull. Whith the old search engine it take 3,5 minutes to create an index for a search table with 320 000 records (2 minutes for the index, 1,5 minute for a index reorg which is mandatory after index creation). To search the BTS index seems to be resonably fast (I have so far only made very few searches) but: The BTS index took a little more than 9 hours to create! It would be a nigthmare to create an index on the largest table with more that 8 millikon records... Does anybody have experiance using BTS? Is it possible to speed up the index creation in some way? (IDS 11.50UC7, BTS 2.00, BTS index in sbspace, all the database on raw disk. OS: Solaris 10 on Sparc) Best regards, Rolf Wasteson **************************************************************************** *** Forum Note: Use "Reply" to post a response in the discussion forum.