Mach11 questions
A poster asked for in-depth MACH11 details: how UPDATEs issued on an SDS node are handled, how SDS avoids serving stale buffer data, and what shared-storage hardware is supported. Replies pointed to the Information Center and 'Informix on Campus' presentations and explained: any storage visible to all nodes (I/O fencing noted as a likely requirement) works, no vendor-specific list; the primary streams log position (LSN) and SDS reads the shared logs to refresh its buffer pages; and since 11.50 secondaries can be updatable, with writes forwarded to the primary and optimistic concurrency/VERCOLS shadow columns detecting stale rows (error -7350). The questioner was still clarifying details at thread's end; no single formal resolution.
Auto-generated by DrWatson from the posts below — may be imperfect; read the full thread.
Does anyone have a good source of in-depth information on Mach11? I googled, perused info center and watched some audio conferences but have not found the info I am looking for so was hoping there was a better place to look. Here are a few things I am looking for : -How UPDATE commands are routed if coming through an SDS node -How does the SDS handle stale data in its buffer. Since things are run asynchronously, if a read command comes in on the SDS but the Primary has just been updated and the SDS hasnt yet rolled through that log how does the SDS know the data has changed? -What are the shared disk requirements/restrictions
I think you should deeper navigate in the documentation (Information Center): for example, take a look at the following page.... http://publib.boulder.ibm.com/infocenter/idshelp/v115/topic/com.ibm.admin.doc/ids_admin_1008.htm There you can find (in the "Hardware and software requirements for SD secondary servers" page) some rules regarding how to configure the disks to use SDS in the correct way. Then, you can simply find some tips on the documentation provided by the "Informix on Campus" portal, such as.... Shared Disk Secondary (SDS) Next evolutionary step of HDR -SDS nodes share disks with the primary -Can have 0 to N SDS nodes How does it work? -Primary transmits the current Log Sequence Number (LSN) as it is flushing logs -SDS instance(s) receives the LSN from the primary and reads the logs from the shared disks -SDS instance(s) applies log changes to its buffer cache - SDS instance(s) re-sync processed LSN to primary Uses -Adjust capacity online as demand changes -Lower data storage costs Similarities with HDR secondary node -Dirty reads allowed on SDS nodes -The primary can failover to any SDS node Differences with HDR secondary node -Only manual failover of primary supported Updates on Secondary -Allows updating activity to be performed from the secondary node -Allows the customers to take better advantage of their investment -Supports a DML operation (insert, update, and delete) on the secondary node. -Uses optimistic concurrency to avoid updating a stale copy of the row. -Works on HDR secondary, RSS nodes, and SDS nodes. - Works with the basic data types, UDTs (those that store data in the server), logged smart BLOBs, and partition BLOBs. -Supports temp tables- both explicit and implicit. -Works with ER. Optimistic Concurrency and Writes on the Secondary -We did not implement a distributed lock manager -Added support for row versioning CREATE TABLE …. WITH VERCOLS ALTER TABLE …. [ADD]/[DROP] VERCOLS -Creates a shadow column consisting of an insert checksum value (ifx_insert_checksum) and update version column (ifx_row_version). -If it is determined that the before image on the secondary is different than the current image on the primary, then the write operation is not allowed and an EVERCONFLICT (-7350) error is returned. -If table does not use VERCOLS, then a before image of the row is used to verify that the update is not being attempted on a stale row. Committed Reads on the Secondary -The secondary node supports ‘Committed Read’ and ‘Last Committed Read’ isolation levels. -This is implemented as a locally committed read, not a globally committed read. -The Read on the secondary node will not return an uncommitted read -However the row could be in the process of being updated The above notices come from the PPT presentation called "IDS_Rep_Avail_Overview", you can find it there: https://www.ibm.com/developerworks/wikis/display/IDS101/Home Hope this helps, Francesco
On Sep 7, 4:25 pm, shorti <lbrya...@juno.com> wrote: > Does anyone have a good source of in-depth information on Mach11? I > googled, perused info center and watched some audio conferences but > have not found the info I am looking for so was hoping there was a > better place to look. Here are a few things I am looking for : > > -How UPDATE commands are routed if coming through an SDS node > -How does the SDS handle stale data in its buffer. Since things are > run asynchronously, if a read command comes in on the SDS but the > Primary has just been updated and the SDS hasnt yet rolled through > that log how does the SDS know the data has changed? > -What are the shared disk requirements/restrictions Thanks for your response...I will look at the info you sent but the first link took me to the place I already looked, which doesnt tell me what the shared disk requirements are.... "Hardware and software requirements for SD secondary servers Except for disk requirements (which are shared with the primary server), hardware and software requirements are generally the same as for HDR secondary servers (Please refer to the Machine Notes for specific supported platforms). In addition, the primary disk system must be shared across the computers that are hosting the database servers. This means that the path to the dbspaces from the SD secondary is the same dbspace path as the primary server. Refer to Configuring a System for HDR."
On 08/09/2010 17:59, shorti wrote: > On Sep 7, 4:25 pm, shorti<lbrya...@juno.com> wrote: >> Does anyone have a good source of in-depth information on Mach11? I >> googled, perused info center and watched some audio conferences but >> have not found the info I am looking for so was hoping there was a >> better place to look. Here are a few things I am looking for : >> >> -How UPDATE commands are routed if coming through an SDS node >> -How does the SDS handle stale data in its buffer. Since things are >> run asynchronously, if a read command comes in on the SDS but the >> Primary has just been updated and the SDS hasnt yet rolled through >> that log how does the SDS know the data has changed? >> -What are the shared disk requirements/restrictions > > Thanks for your response...I will look at the info you sent but the > first link took me to the place I already looked, which doesnt tell me > what the shared disk requirements are.... Without trying to be sarcastic at all - you need disks that can be shared by a number of hosts. With the small caveat that each sds secondary need two private files as well - from memory they are both <1M Why don't you download the vmware image that has running examples of hdr and rss and sds nodes already configured and have a play? -- This message has been scanned for viruses and dangerous content by OpenProtect(http://www.openprotect.com), and is believed to be clean.
"shorti" <lbryan21@juno.com> wrote in message news:ee945faf-b47a-4009-90e8-ee48724848c8@g17g2000yqe.googlegroups.com... On Sep 7, 4:25 pm, shorti <lbrya...@juno.com> wrote: > Does anyone have a good source of in-depth information on Mach11? I > googled, perused info center and watched some audio conferences but > have not found the info I am looking for so was hoping there was a > better place to look. Here are a few things I am looking for : > > -How UPDATE commands are routed if coming through an SDS node > -How does the SDS handle stale data in its buffer. Since things are > run asynchronously, if a read command comes in on the SDS but the > Primary has just been updated and the SDS hasnt yet rolled through > that log how does the SDS know the data has changed? > -What are the shared disk requirements/restrictions >> Thanks for your response...I will look at the info you sent but the first link took me to the place I already looked, which doesnt tell me what the shared disk requirements are.... What *do* you want to know??
On Sep 8, 2:31 pm, "Neil Truby" <neil.tr...@ardenta.com> wrote: > "shorti" <lbrya...@juno.com> wrote in message > > news:ee945faf-b47a-4009-90e8-ee48724848c8@g17g2000yqe.googlegroups.com... > On Sep 7, 4:25 pm, shorti <lbrya...@juno.com> wrote: > > > Does anyone have a good source of in-depth information on Mach11? I > > googled, perused info center and watched some audio conferences but > > have not found the info I am looking for so was hoping there was a > > better place to look. Here are a few things I am looking for : > > > -How UPDATE commands are routed if coming through an SDS node > > -How does the SDS handle stale data in its buffer. Since things are > > run asynchronously, if a read command comes in on the SDS but the > > Primary has just been updated and the SDS hasnt yet rolled through > > that log how does the SDS know the data has changed? > > -What are the shared disk requirements/restrictions > >> Thanks for your response...I will look at the info you sent but the > > first link took me to the place I already looked, which doesnt tell me > what the shared disk requirements are.... > > What *do* you want to know?? hmm..this shouldnt be that hard of a question. What is supported...most products have a limit of what it has been tested on so I doubt just any shared disk storage system is supported? Does it support all IBM DSxxxx, XIV, HP StorageWorks, EMC Sym, Sun 9xxxV, etc, etc. I did read that GFS and GPFS filesystems are supported. Anything like this would be helpful. Also, what I have read so far doesnt tell me how updates or reads work on the SDS. I did however learn that there is an update option for SDS...why optional? Are there limitations or performance differences. I have only gone through almost all the presentations on the Informix on Campus site so still working though them and plus the last link.
Just ANY shared disk subsystem is supported. This is Informix not Oracle. Art Art S. Kagel Advanced DataTools (www.advancedatatools.com) IIUG Board of Directors (art@iiug.org) Disclaimer: Please keep in mind that my own opinions are my own opinions and do not reflect on my employer, Advanced DataTools, the IIUG, nor any other organization with which I am associated either explicitly, implicitly, or by inference. Neither do those opinions reflect those of other individuals affiliated with any entity with which I am affiliated nor those of the entities themselves. On Wed, Sep 8, 2010 at 6:39 PM, shorti <lbryan21@juno.com> wrote: > On Sep 8, 2:31 pm, "Neil Truby" <neil.tr...@ardenta.com> wrote: > > "shorti" <lbrya...@juno.com> wrote in message > > > > news:ee945faf-b47a-4009-90e8-ee48724848c8@g17g2000yqe.googlegroups.com. > .. > > On Sep 7, 4:25 pm, shorti <lbrya...@juno.com> wrote: > > > > > Does anyone have a good source of in-depth information on Mach11? I > > > googled, perused info center and watched some audio conferences but > > > have not found the info I am looking for so was hoping there was a > > > better place to look. Here are a few things I am looking for : > > > > > -How UPDATE commands are routed if coming through an SDS node > > > -How does the SDS handle stale data in its buffer. Since things are > > > run asynchronously, if a read command comes in on the SDS but the > > > Primary has just been updated and the SDS hasnt yet rolled through > > > that log how does the SDS know the data has changed? > > > -What are the shared disk requirements/restrictions > > >> Thanks for your response...I will look at the info you sent but the > > > > first link took me to the place I already looked, which doesnt tell me > > what the shared disk requirements are.... > > > > What *do* you want to know?? > > hmm..this shouldnt be that hard of a question. What is > supported...most products have a limit of what it has been tested on > so I doubt just any shared disk storage system is supported? > > Does it support all IBM DSxxxx, XIV, HP StorageWorks, EMC Sym, Sun > 9xxxV, etc, etc. I did read that GFS and GPFS filesystems are > supported. Anything like this would be helpful. > > Also, what I have read so far doesnt tell me how updates or reads work > on the SDS. I did however learn that there is an update option for > SDS...why optional? Are there limitations or performance > differences. I have only gone through almost all the presentations on > the Informix on Campus site so still working though them and plus the > last link. > _______________________________________________ > Informix-list mailing list > Informix-list@iiug.org > http://www.iiug.org/mailman/listinfo/informix-list >
"shorti" <lbryan21@juno.com> wrote in message news:3dcfd7b6-dd4e-4c78-99b4-085997005631@t11g2000vbc.googlegroups.com... On Sep 8, 2:31 pm, "Neil Truby" <neil.tr...@ardenta.com> wrote: > "shorti" <lbrya...@juno.com> wrote in message > > news:ee945faf-b47a-4009-90e8-ee48724848c8@g17g2000yqe.googlegroups.com... > On Sep 7, 4:25 pm, shorti <lbrya...@juno.com> wrote: > >> hmm..this shouldnt be that hard of a question. What is supported...most products have a limit of what it has been tested on so I doubt just any shared disk storage system is supported? >> Does it support all IBM DSxxxx, XIV, HP StorageWorks, EMC Sym, Sun 9xxxV, etc, etc. I did read that GFS and GPFS filesystems are supported. Anything like this would be helpful. Hmm .. this shouldn't be that hard of an answer. ANY disk system that allows two servers access to the same LUN/volume. I'm am sure all you mention would be fine. I have tested it successfully with NAS. >> Also, what I have read so far doesnt tell me how updates or reads work on the SDS. I did however learn that there is an update option for SDS...why optional? Because you may specifically want to configure OLTP and read-only nodes. >> Are there limitations or performance differences. Well now I start to agree with you. I too would like to read some real-world experiences. In test systems I have set up on HP EVA and NAS, with clustered Connection Manager, it's worked brilliantly - 2-3s to fail over if a node is pulled, compared with 30-40s with RHEL Cluster or Sun Cluster. But, not used it in production yet. SDS has only recently been aletered from a chargeable extra to being integral within the base product, so I doubt if more than a handful of companies have been able to afford it up until now, so this experience may be hard to find.
On Wed, Sep 8, 2010 at 12:25 AM, shorti <lbryan21@juno.com> wrote: > -How UPDATE commands are routed if coming through an SDS node > Not sure about what you need to know... But there are specific threads on the primary to deal with that type of requests. It's important to check (and if you browsed through the several presentations you've seen references to it) the VERCOLS feature. If your tables don't have VERCOLS it will work, but with some performance impact. VERCOLS are a very quick way for the primary to check if the secondary is trying to update and "up to date" row. If it's not it will raise an error. If you don't use the VERCOLS the system will work transparently but it will have to check the whole row (data comparison). The logic question is: Is it worth? Wouldn't it be better to direct updatable sessions to the primary? I don't know if anyone will be able to give you a definitive answer on this, because the issue is complex. Let's see. An update requires the system to locate the row. This workload will move out of the primary. Here you gain (possibly a lot). Than it will have to check the "up to date" status of the image the secondary found. Here you will loose (more or less depending if you use VERCOLS). In any case, it was a very common situation in bigger environments for applications to run DML on several instances (either explicitly or by the use of synonyms. These way of doing things was more expensive that what updatable secondaries provide you). In the last IIUG conference there were some presentations and corridor talks about customers using MACH 11 transparently for horizontal scalability and with good results. In any case, my personal feeling would be to direct "write most" applications to the primary and "read most" applications to the secondaries. And this is easy to do (as long as you know your applications) with the connection manager. Two opposite examples, and yes, typically life will not be as simple as this: 1- A batch that loads data into the database essentially with a "load from /som/path/file.unl insert into table" would be going against the primary 2- A report that does a huge query, sorts, groups etc and at the end updates one row marking that user XPTO run the report would be going against one of the secondary servers. But again, apparently customers are running unchanged applications against MACH 11 clusters without these concerns with success. > -How does the SDS handle stale data in its buffer. Since things are > run asynchronously, if a read command comes in on the SDS but the > Primary has just been updated and the SDS hasnt yet rolled through > that log how does the SDS know the data has changed? > The primary keeps sending the current log position to the SDS secondary(ies). Note that this is just a log position, not the log itself. And this will not wait for a logical log to complete. It's a continuous stream... SDS will read the logical logs following that logical log position (LSN - Logical Log Serial Number) and will check the operations in it. If the operation "says" that page N was changed it will change that page if it's in that server memory. -What are the shared disk requirements/restrictions > I asked that exactly same question to our MACH 11 guru and the answer was pretty straight forward: The storage system must be visible in the several nodes and must support I/O Fencing. I never heard about I/O Fencing before, but a Google search will do for you what is has for me. Then I asked if that requirement was explicitly and cleared stated in the docs and the answer was not straight forward :). He ended up with something like "we must verify that". Your doubt makes me think we're still in the same situation. Hopefully he's reading this and will take note :) (and correct me if I wrote something wrong) Regards. -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently...
On Sep 8, 12:44 am, fv_it <f.ver...@strhold.it> wrote: > I think you should deeper navigate in the documentation (Information > Center): for example, take a look at the following page....http://publib.boulder.ibm.com/infocenter/idshelp/v115/topic/com.ibm.a... > There you can find (in the "Hardware and software requirements for SD > secondary servers" page) some rules regarding how to configure the > disks to use SDS in the correct way. > > Then, you can simply find some tips on the documentation provided by > the "Informix on Campus" portal, such as.... > > Shared Disk Secondary (SDS) > > Next evolutionary step of HDR > -SDS nodes share disks with the primary > -Can have 0 to N SDS nodes > How does it work? > -Primary transmits the current Log Sequence Number (LSN) as it is > flushing logs > -SDS instance(s) receives the LSN from the primary and reads the logs > from the shared disks > -SDS instance(s) applies log changes to its buffer cache > - SDS instance(s) re-sync processed LSN to primary > Uses > -Adjust capacity online as demand changes > -Lower data storage costs > Similarities with HDR secondary node > -Dirty reads allowed on SDS nodes > -The primary can failover to any SDS node > Differences with HDR secondary node > -Only manual failover of primary supported > > Updates on Secondary > -Allows updating activity to be performed from the secondary node > -Allows the customers to take better advantage of their investment > -Supports a DML operation (insert, update, and delete) on the > secondary node. > -Uses optimistic concurrency to avoid updating a stale copy of the > row. > -Works on HDR secondary, RSS nodes, and SDS nodes. > - Works with the basic data types, UDTs (those that store data in the > server), logged smart BLOBs, and partition BLOBs. > -Supports temp tables- both explicit and implicit. > -Works with ER. > > Optimistic Concurrency and Writes on the Secondary > -We did not implement a distributed lock manager > -Added support for row versioning > CREATE TABLE …. WITH VERCOLS > ALTER TABLE …. [ADD]/[DROP] VERCOLS > -Creates a shadow column consisting of an insert checksum value > (ifx_insert_checksum) and update version column (ifx_row_version). > -If it is determined that the before image on the secondary is > different than the current image on the primary, then the write > operation is not allowed and an EVERCONFLICT (-7350) error is > returned. > -If table does not use VERCOLS, then a before image of the row is > used to verify that the update is not being attempted on a stale row. > > Committed Reads on the Secondary > -The secondary node supports ‘Committed Read’ and ‘Last Committed > Read’ isolation levels. > -This is implemented as a locally committed read, not a globally > committed read. > -The Read on the secondary node will not return an uncommitted read > -However the row could be in the process of being updated > > The above notices come from the PPT presentation called > "IDS_Rep_Avail_Overview", you can find it there:https://www.ibm.com/developerworks/wikis/display/IDS101/Home > > Hope this helps, > Francesco This is really good information. However, its very confusing since information on sites indicated all secondary solutions for Mach11 are read-only. So, now it seems you can do writes on the SDS as well. Now I have more questions! Before I was thinking that the application program would be doing reads on the buffer in the SDS node. Are the updates are being done on the actual database on the shared storage? If so, what is the purpose of the buffer on the SDS nodes.
On Thu, Sep 9, 2010 at 1:20 AM, shorti <lbryan21@juno.com> wrote: > On Sep 8, 12:44 am, fv_it <f.ver...@strhold.it> wrote: > > I think you should deeper navigate in the documentation (Information > > Center): for example, take a look at the following page.... > http://publib.boulder.ibm.com/infocenter/idshelp/v115/topic/com.ibm.a... > > There you can find (in the "Hardware and software requirements for SD > > secondary servers" page) some rules regarding how to configure the > > disks to use SDS in the correct way. > > > > Then, you can simply find some tips on the documentation provided by > > the "Informix on Campus" portal, such as.... > > > > Shared Disk Secondary (SDS) > > > > Next evolutionary step of HDR > > -SDS nodes share disks with the primary > > -Can have 0 to N SDS nodes > > How does it work? > > -Primary transmits the current Log Sequence Number (LSN) as it is > > flushing logs > > -SDS instance(s) receives the LSN from the primary and reads the logs > > from the shared disks > > -SDS instance(s) applies log changes to its buffer cache > > - SDS instance(s) re-sync processed LSN to primary > > Uses > > -Adjust capacity online as demand changes > > -Lower data storage costs > > Similarities with HDR secondary node > > -Dirty reads allowed on SDS nodes > > -The primary can failover to any SDS node > > Differences with HDR secondary node > > -Only manual failover of primary supported > > > > Updates on Secondary > > -Allows updating activity to be performed from the secondary node > > -Allows the customers to take better advantage of their investment > > -Supports a DML operation (insert, update, and delete) on the > > secondary node. > > -Uses optimistic concurrency to avoid updating a stale copy of the > > row. > > -Works on HDR secondary, RSS nodes, and SDS nodes. > > - Works with the basic data types, UDTs (those that store data in the > > server), logged smart BLOBs, and partition BLOBs. > > -Supports temp tables- both explicit and implicit. > > -Works with ER. > > > > Optimistic Concurrency and Writes on the Secondary > > -We did not implement a distributed lock manager > > -Added support for row versioning > > CREATE TABLE …. WITH VERCOLS > > ALTER TABLE …. [ADD]/[DROP] VERCOLS > > -Creates a shadow column consisting of an insert checksum value > > (ifx_insert_checksum) and update version column (ifx_row_version). > > -If it is determined that the before image on the secondary is > > different than the current image on the primary, then the write > > operation is not allowed and an EVERCONFLICT (-7350) error is > > returned. > > -If table does not use VERCOLS, then a before image of the row is > > used to verify that the update is not being attempted on a stale row. > > > > Committed Reads on the Secondary > > -The secondary node supports ‘Committed Read’ and ‘Last Committed > > Read’ isolation levels. > > -This is implemented as a locally committed read, not a globally > > committed read. > > -The Read on the secondary node will not return an uncommitted read > > -However the row could be in the process of being updated > > > > The above notices come from the PPT presentation called > > "IDS_Rep_Avail_Overview", you can find it there: > https://www.ibm.com/developerworks/wikis/display/IDS101/Home > > > > Hope this helps, > > Francesco > > This is really good information. However, its very confusing since > information on sites indicated all secondary solutions for Mach11 are > read-only. > Weird, unless you're reading v11.10 stuff. Updatable secondaries were introduced in v11.50. You can configure any secondary (HDR, RSS or SDS) to allow updates. And your question: "-How UPDATE commands are routed if coming through an SDS node" made me think you understood that. So, now it seems you can do writes on the SDS as well. Now I have more > questions! Before I was thinking that the application program would > be doing reads on the buffer in the SDS node. Are the updates are > being done on the actual database on the shared storage? If so, what > is the purpose of the buffer on the SDS nodes. > The updates are transparently sent to the primary node. Only the primary node will effectively write to the disk. The buffer on the secondary nodes has exactly the same role an in the primary. It's used for caching disk pages for speeding up access times (just like in any RDBMS system) Regards. -- Fernando Nunes Portugal http://informix-technology.blogspot.com My email works... but I don't check it frequently...
> And your question: "-How UPDATE commands are routed if coming through an SDS > node" made me think you understood that. > The updates are transparently sent to the primary node. Only the primary > node will effectively write to the disk. > The buffer on the secondary nodes has exactly the same role an in the > primary. It's used for caching disk pages for speeding up access times (just > like in any RDBMS system) > Regards. > -- > Fernando Nunes > Portugal No..I was thinking of the topology where you have two nodes (primary and SDS) where there is an application processes running on each that wants to read/write to the database. If the reads on the SDS retrieves data from its own bufferpool and updates were not allowed (or so I thought at the time) I was wondering how updates were routed because I assumed they had to go to the primary. Before V11.50 this was done manually but the user some how? Now, you state that UPDATEs are allowed on the SDS but are actually rerouted through the Primary, so it must really be the same as before except there is some sort of automatic rerouting. One thing that is confusing with this reroute to the primary is one of the slides stated the following: "Uses optimistic concurrency to avoid updating a stale copy of the row" "If it is determined that the before image on the secondary is different than the current image on the primary, then the write operation is not allowed and an EVERCONFLICT (-7350) error is returned" This leads me to believe that updates are not just rerouted to the primary...its saying there is a possibility of a "stale" copy on the SDS. If the update is merely rerouted to the primary for the update wouldnt it just look like any other update to a server where if another process/thread is updating the row, the update request coming from the SDS would just have wait until that update finishes? This concept of versioning of each row is interesting...and is centered around my original question on how READs work on the SDS. But I see in the slides that the reads on the SDS only supports committed reads so this must mot apply to read operations. I really appreciate your help on this...these concepts are important for us to determine if Mach11 is right for the new product we want to design.