Concurrency in BTS
Hi, This mail should maybe have been sent to the datablade list but it seems to be almost dead. I hope this is OK. We have a system using a 3rd party Datablade which is a text (indexing and) search engine. The indexes are updated in real time as the text to be indexed is updated by an online application with a lot of concurrent users. For different reasons we now consider to use Informix Basic Text Search (BTS) as a substitute for the search engine used today. Performance is something to be tested, we still remember testing the Excalibur Text Datablade when the system was built 8 years ago. That was not fun... Now, my question to all of you is if you have any experience using BTS. In the BTS manual you can read: "Transactions with Basic Text Search The bts index is located in an sbspace. INSERT, DELETE, and UPDATE operations lock the index during modifications, which prevents any other transaction from changing the index during the change. Therefore, each modification is done in a series. The operations make one attempt at a modification. If the index is locked, the operation fails." The fact that update of each index is serialized is probably not a mayor problem but the last sentence makes me nervous. If modifications are done in a serie then the index should not be locked when a modification is to be done. So the last sentence must have another meaning. If the index is locked during index compacting or reorganization, that is OK but can it be locked during normal use (update/search). Does anyone know what? Regards Rolf Wasteson PS Our platform today is Sun Sparc with Solaris 10, Informix IDS 11.50UC2, to be upgraded to 11.50FC6 before BTS 2.00 will be used. -------------------------------------------- Rolf Wasteson Tel: 08 738 47 68 | Mobil: 0708 38 40 56 E-post: rolf.wasteson@applicate.se<mailto:rolf.wasteson@applicate.se> [cid:image001.jpg@01CAA4D8.145BEB60] Infodata Applicate AB Besöksadress: Primusgatan 18, Stockholm Postadress: Box 34101 | 100 26 Stockholm Tel växel: 08 738 54 00 | Fax: 08 738 50 50 www.applicate.se Applicate stärker uppdragsgivarnas konkurrenskraft genom kundunika, säkra och effektiva IT-lösningar.
Hi Rolf Wasteson,
I am the project leader on BTS for IBM Informix.
The sentences:
"Therefore, each modification is done in a series.
The operations make one attempt at a modification.
If the index is locked, the operation fails."
Are not worded very well. Thank you for bringing
it to my attention and I will work with the team
on improving it.
Let me expand a little on how bts behaves. It does depend a little
on the version of IDS:
In 11.50.xC5 and earlier, the BTS index operations execute in
The a single bts VP and each operation executed sequentially.
Multiple readers and writers can interleave their operations.
There is an internal lock to ensure no concurrent (threaded)
operations execute on the VP and multiple VPs were not allowed. There
is a slim possibility that the lock would timeout and the SQL
statement would fail but I have not had a report of this happening.
That was essentially the behaviour for bts.1.00, bts.1.10 and
bts.2.00 up to and including 11.50.xC5.
Starting with 11.50.xC6, we enabled bts to work with multiple
bts VPs. There is still a restriction that each VP only executes
one operation at one time, however multiple bts VPs
may be create with the VPCLASS onconfig variable
or more bts VPs can be added with the onmode -p command.
(also, the noyield flag is no longer required when defining a bts
VPCLASS).
There may be multiple readers and writers of any bts index
across several bts VPs. There is a Critical Section in
the transaction commit with put an exclusive
(write) lock on the BTS index during this phase. The lock
will wait a significantly long time before it times out.
The compact operation still has an exclusive lock. The
purpose of compact is to free up space back to the file
system in an index built in an extent space or if the index
is built in an sbspace (a bts.2.00 feature) will free
up pages back to the sbspace.
But unlike the first release of BTS, BTS in 11.50
will eventually reuse pages in the index when new rows
are inserted after rows have been deleted.
This greatly reduces the need to compact.
I hope this helps explain the behaviour of BTS a
little better.
Thank you.
Fair winds,
Mark
mailto:mark.ashworth@acm.org
> -----Original Message-----
> From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of
> rolf.wasteson@applicate.se
> Sent: February 3, 2010 8:24 AM
> To: ids@iiug.org
> Subject: Concurrency in BTS [18877]
>
> Hi,
>
> This mail should maybe have been sent to the datablade list but it
> seems to be
> almost dead. I hope this is OK.
>
> We have a system using a 3rd party Datablade which is a text (indexing
> and)
> search engine. The indexes are updated in real time as the text to be
> indexed
> is updated by an online application with a lot of concurrent users. For
> different reasons we now consider to use Informix Basic Text Search
> (BTS) as a
> substitute for the search engine used today. Performance is something
> to be
> tested, we still remember testing the Excalibur Text Datablade when the
> system
> was built 8 years ago. That was not fun...
>
> Now, my question to all of you is if you have any experience using BTS.
> In the
> BTS manual you can read:
>
> "Transactions with Basic Text Search
> The bts index is located in an sbspace. INSERT, DELETE, and UPDATE
> operations
> lock the index during modifications, which prevents any other
> transaction from
> changing the index during the change. Therefore, each modification is
> done in
> a series. The operations make one attempt at a modification. If the
> index is
> locked, the operation fails."
>
> The fact that update of each index is serialized is probably not a
> mayor
> problem but the last sentence makes me nervous. If modifications are
> done in a
> serie then the index should not be locked when a modification is to be
> done.
> So the last sentence must have another meaning. If the index is locked
> during
> index compacting or reorganization, that is OK but can it be locked
> during
> normal use (update/search). Does anyone know what?
>
> Regards
> Rolf Wasteson
>
> PS Our platform today is Sun Sparc with Solaris 10, Informix IDS
> 11.50UC2, to
> be upgraded to 11.50FC6 before BTS 2.00 will be used.
>
> --------------------------------------------
> Rolf Wasteson
> Tel: 08 738 47 68 | Mobil: 0708 38 40 56
> E-post: rolf.wasteson@applicate.se<mailto:rolf.wasteson@applicate.se>
>
> [cid:image001.jpg@01CAA4D8.145BEB60]
>
> Infodata Applicate AB
> Besöksadress: Primusgatan 18, Stockholm
> Postadress: Box 34101 | 100 26 Stockholm
> Tel växel: 08 738 54 00 | Fax: 08 738 50 50
> www.applicate.se
>
> Applicate stärker uppdragsgivarnas konkurrenskraft
> genom kundunika, säkra och effektiva IT-lösningar.
>
>
> ***********************************************************************
> ********
> Forum Note: Use "Reply" to post a response in the discussion forum.
Thank you Mark,
That was good to hear. Looks as if are getting closer to use BTS...
We have good experience from using CLucene in other projects so I am not so
afraid of performance issues, tests will show.
Regards,
Rolf
-----Ursprungligt meddelande-----
Från: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] För Mark Ashworth
Skickat: den 3 februari 2010 21:16
Till: ids@iiug.org
Ämne: RE: Concurrency in BTS [18893]
Hi Rolf Wasteson,
I am the project leader on BTS for IBM Informix.
The sentences:
"Therefore, each modification is done in a series.
The operations make one attempt at a modification.
If the index is locked, the operation fails."
Are not worded very well. Thank you for bringing
it to my attention and I will work with the team
on improving it.
Let me expand a little on how bts behaves. It does depend a little
on the version of IDS:
In 11.50.xC5 and earlier, the BTS index operations execute in
The a single bts VP and each operation executed sequentially.
Multiple readers and writers can interleave their operations.
There is an internal lock to ensure no concurrent (threaded)
operations execute on the VP and multiple VPs were not allowed. There
is a slim possibility that the lock would timeout and the SQL
statement would fail but I have not had a report of this happening.
That was essentially the behaviour for bts.1.00, bts.1.10 and
bts.2.00 up to and including 11.50.xC5.
Starting with 11.50.xC6, we enabled bts to work with multiple
bts VPs. There is still a restriction that each VP only executes
one operation at one time, however multiple bts VPs
may be create with the VPCLASS onconfig variable
or more bts VPs can be added with the onmode -p command.
(also, the noyield flag is no longer required when defining a bts
VPCLASS).
There may be multiple readers and writers of any bts index
across several bts VPs. There is a Critical Section in
the transaction commit with put an exclusive
(write) lock on the BTS index during this phase. The lock
will wait a significantly long time before it times out.
The compact operation still has an exclusive lock. The
purpose of compact is to free up space back to the file
system in an index built in an extent space or if the index
is built in an sbspace (a bts.2.00 feature) will free
up pages back to the sbspace.
But unlike the first release of BTS, BTS in 11.50
will eventually reuse pages in the index when new rows
are inserted after rows have been deleted.
This greatly reduces the need to compact.
I hope this helps explain the behaviour of BTS a
little better.
Thank you.
Fair winds,
Mark
mailto:mark.ashworth@acm.org
> -----Original Message-----
> From: ids-bounces@iiug.org [mailto:ids-bounces@iiug.org] On Behalf Of
> rolf.wasteson@applicate.se
> Sent: February 3, 2010 8:24 AM
> To: ids@iiug.org
> Subject: Concurrency in BTS [18877]
>
> Hi,
>
> This mail should maybe have been sent to the datablade list but it
> seems to be
> almost dead. I hope this is OK.
>
> We have a system using a 3rd party Datablade which is a text (indexing
> and)
> search engine. The indexes are updated in real time as the text to be
> indexed
> is updated by an online application with a lot of concurrent users. For
> different reasons we now consider to use Informix Basic Text Search
> (BTS) as a
> substitute for the search engine used today. Performance is something
> to be
> tested, we still remember testing the Excalibur Text Datablade when the
> system
> was built 8 years ago. That was not fun...
>
> Now, my question to all of you is if you have any experience using BTS.
> In the
> BTS manual you can read:
>
> "Transactions with Basic Text Search
> The bts index is located in an sbspace. INSERT, DELETE, and UPDATE
> operations
> lock the index during modifications, which prevents any other
> transaction from
> changing the index during the change. Therefore, each modification is
> done in
> a series. The operations make one attempt at a modification. If the
> index is
> locked, the operation fails."
>
> The fact that update of each index is serialized is probably not a
> mayor
> problem but the last sentence makes me nervous. If modifications are
> done in a
> serie then the index should not be locked when a modification is to be
> done.
> So the last sentence must have another meaning. If the index is locked
> during
> index compacting or reorganization, that is OK but can it be locked
> during
> normal use (update/search). Does anyone know what?
>
> Regards
> Rolf Wasteson
>
> PS Our platform today is Sun Sparc with Solaris 10, Informix IDS
> 11.50UC2, to
> be upgraded to 11.50FC6 before BTS 2.00 will be used.
>
> --------------------------------------------
> Rolf Wasteson
> Tel: 08 738 47 68 | Mobil: 0708 38 40 56
> E-post: rolf.wasteson@applicate.se<mailto:rolf.wasteson@applicate.se>
>
> [cid:image001.jpg@01CAA4D8.145BEB60]
>
> Infodata Applicate AB
> Besöksadress: Primusgatan 18, Stockholm
> Postadress: Box 34101 | 100 26 Stockholm
> Tel växel: 08 738 54 00 | Fax: 08 738 50 50
> www.applicate.se
>
> Applicate stärker uppdragsgivarnas konkurrenskraft
> genom kundunika, säkra och effektiva IT-lösningar.
>
>
> ***********************************************************************
> ********
> Forum Note: Use "Reply" to post a response in the discussion forum.
*******************************************************************************
Forum Note: Use "Reply" to post a response in the discussion forum.