Shared Memory overflow

[Surya Singh Ghale]

Given the size of your databases I assume you are using Enterprise RDBMS licenses, not Workgroup.

Yes, you should always run a watchdog but its function is to clean up after dead shared-memory clients, not to prevent a database hang. Also, you should always run a BIW, AIW, and at least one APW.

Since you mention potential data loss: are you using after imaging?

Yes, We are running after imaging after one of 24 hour downtime due to database corruption due to -Mxs issue. We are running BIW, AIW and APW but not POW

We are running Enterprise License.

Thanks a lot....

 

[Surya Singh Ghale]

Run watchdog for sure. Once upon a time we were trying to recreate a database crash issue to prove that the Unix admins were causing the database shutdowns by killing processes.

In the test database the watchdog was not running (by accident) and the database never crashed but we did get it to hang for new connections and do crazy things for existing connections. Once we started watchdog and repeated the tests the database shut down like we would expect when a process was killed in an ugly way.

Thanks a lot. Will do....

 

[Surya Singh Ghale]

As Rob says -- if you are worried about database integrity you need to have after-imaging enabled. Actually if you are not worried about it you should have ai enabled. Running without after-imaging is irresponsible.

We have enabled AI...

 

[Surya Singh Ghale]

When the database is supposedly "hung" is there always a message regarding -Mxs in the .lg file?

Hi Tom,

Yes. -Mxs is always there for every instance we have faced. I have also one more point to say is that, the issue is happening when MFG/PRO batch processes are runaway... Same issue in 3 different databases when batch process is runaway. I tell application team that it is application issue but in vain as they say see log tells to increase -Mxs...

Thanks a lot....

 

[Rob Fitzpatrick]

BTW, Any idea of MFG/PRO programs need to be recompiled when we upgrade from 10.2A ro 10.2B?

I don't know MFG/PRO, so I can't comment on QAD's code. But the compiler version numbers for 32-bit and 64-bit didn't change between 10.2A and 10.2B08. So your existing r-code should just work.

 

[Surya Singh Ghale]

What is this monitoring tool that fails to tell you that the database is "hung"?

We are using simple tool called Centreon and it polls if the main broker is running and as well as proutil -C dbipcs is returing a value.

Thanks a lot

 

[Surya Singh Ghale]

I don't know MFG/PRO, so I can't comment on QAD's code. But the compiler version numbers for 32-bit and 64-bit didn't change between 10.2A and 10.2B08. So your existing r-code should just work.

Thanks a lot Rob

 

[Rob Fitzpatrick]

Same issue in 3 different databases when batch process is runaway.

How do you define a "runaway" process? What are its characteristics?

 

[TheMadDBA]

For now I would increase the values of -L for every database that is getting a lock table overflow. Monitor which databases are still getting lock table overflows and decide if the code is the problem or the settings are too low. What are the values for -L now?

You have to check the SQL isolation level on the clients that are connecting to the DBs as far as I know. Unless somebody else knows how to tell for active connections (other than looking for lots of locks).

 

[Surya Singh Ghale]

R-code is upwardly compatible. So, strictly speaking, you do not need to recompile going from 10.2a to 10.2b.

But if you have the ability to compile code then there is no reason to compile anything less than everything. What possible reason would you have for picking and choosing?

Yup... Thanks a lot.

 

[Surya Singh Ghale]

Thanks a lot to everybody....

Good Night... I am in ASIA...

Have anyone heard about progress DBA training in Singapore or ASIAN nations?

 

[Surya Singh Ghale]

How do you define a "runaway" process? What are its characteristics?

Usually, there is maximum of 2 MFG/PRO batch processes running concurrently. But during the issue we faced, we had hunderds of these processes.

Thanks a lot...

 

[TomBascom]

This really just sounds like a classic out of control mfg/pro shop being run by cowboys.

Training sounds like a great idea. Unfortunately organized classroom style training has gone out of style. Your best bet is mentoring from a good consultant. Or you could form a habit of attending user conferences -- Progress Exchange is in 3 weeks in Orlando. But it isn't very technical and is mostly focused on futures and marketing fluff. PUG Challenge is much, much more technical -- the next one is in Dusseldorf in mid-November. Or the America's will be in June. QAD has an annual conference that, as I recall, is in April or May but they mostly focus on their product's functionality not so much the DBA side of things.

 

[tamhas]

And there is a conference in Australia, but I can't speak to its coverage. But, mentoring is a good option. Get a good consultant out for a visit, handle a bunch of issues, provide some monitoring tools, and then much beyond that can be handled remotely as needed. You not only get your system sharpened up, if you pay attention, you learn a lot in the process.

 

[Rob Fitzpatrick]

For some basic training, there is Progress' eLearning site (https://wbt.progress.com). If you have a PSDN subscription it entitles you to access. There are some DBA courses in the curriculum.

A really good (and free!) online resource is the material from past Progress conferences. You can download slide decks from the conference sessions and in some cases there are also audio or video recordings available. This is valuable stuff that in many cases contains more detailed info than you'll get in the documentation or even the KB articles. Older stuff should be viewed in context (details may be out of date) but is still valuable for learning fundamentals.

PUG Challenge Americas: http://pugchallenge.org/downloads.html
PUG Challenge EMEA: http://www.pugchallenge.eu/sessions_2013.html
Progress Exchange: https://community.progress.com/technicalusers/w/exchange/default.aspx

 

[Surya Singh Ghale]

And there is a conference in Australia, but I can't speak to its coverage. But, mentoring is a good option. Get a good consultant out for a visit, handle a bunch of issues, provide some monitoring tools, and then much beyond that can be handled remotely as needed. You not only get your system sharpened up, if you pay attention, you learn a lot in the process.

Thanks a lot...

 

[Surya Singh Ghale]

Hi All,

Finally, We are applying patch 10.2A03 to 10.2A followed by 10.2B.

BTW, I found that 10.2A also has the shared memory leak issue and it is resolved in 10.2A01 and above. Is it true?

Thanks to all.

 

[TomBascom]

1) After you upgrade to 10.2b you need to also apply service pack 8. That will get you up to date. (Vanilla 10.2b was released in 2009...) There have been many, many bug fixes and important enhancements in the intervening years.

2) How did you find that an issue has been resolved? Do you have a kbase entry to reference? A bug #? A forum posting elsewhere? Or is this a rumor picked up on the street somewhere? It matters because you have asked if it is true. On the face of it it is false. The -Mxs exceeded error still occurs in every release of Progress. And, so far, I personally am not convinced that it has anything at all to do with your reported database hangs -- which seem like the real problem that you are trying to solve. So if there is a reference of some sort to the source of the information that some problem has been resolved in some release we might be able to tell you whether or not that particular resolution has any relevance to the particular issues that you are reporting.

 

[Surya Singh Ghale]

1) After you upgrade to 10.2b you need to also apply service pack 8. That will get you up to date. (Vanilla 10.2b was released in 2009...) There have been many, many bug fixes and important enhancements in the intervening years.

2) How did you find that an issue has been resolved? Do you have a kbase entry to reference? A bug #? A forum posting elsewhere? Or is this a rumor picked up on the street somewhere? It matters because you have asked if it is true. On the face of it it is false. The -Mxs exceeded error still occurs in every release of Progress. And, so far, I personally am not convinced that it has anything at all to do with your reported database hangs -- which seem like the real problem that you are trying to solve. So if there is a reference of some sort to the source of the information that some problem has been resolved in some release we might be able to tell you whether or not that particular resolution has any relevance to the particular issues that you are reporting.

Hi Tom,

Thanks a lot for your suggestion...

Application Team provided Article Number 000012752 (KBASE12752) and pointed out that there is a bug in 10.2A which is the shared memory leak.

The issue is not resolved yet. We are going to apply patch 10.2A03 and later upgrade to 10.2B with service pack 8.

I am unable to prove that the issue was there in 10.1 and it is resolved in 10.1C and 10.2A. Any KBASE you know of which states that shared memory leak is resolved in 10.2A?

Thanks a lot again...

 

[Rob Fitzpatrick]

The article you cite references a bug in 10.2A FCS, fixed in 10.2A01, where 168 bytes of memory is leaked with every client disconnect/reconnect. This is a small amount of memory, and it is not clear whether it is the cause of your (6495) errors, especially as you are restarting your databases at least monthly. Regardless, once you upgrade to 10.2B08 you won't be faced with this bug.