Swap space?

ron

Member
Linux 3.10.0-1160.2.2.el7.x86_64 x86_64
OE 11.7.4 Enterprise

I have a Development server with 82 GB RAM and 21 GB swap.

There are two copies of Production - each with 7.5 GB of buffers -- and another 23 copies that are also copies but have very small buffer allocations (2 MB each).

Most of the small copies exist but brokers only get loaded occasionally -- only about 5 are running most of the time.

On rare occasions the kernel panics and kills-off the two "large" brokers -- because of a lack of memory. By "memory", of course, it means "virtual memory".

I asked my System Admin to bump-up swap -- preferably by about 10 GB.

He's not happy about it and suggests that I reduce the memory buffers.

I am not a System Admin and I don't profess to be any sort of expert about memory, but I don't see a problem. Adding another 10 GB of swap just means creating an extra swap partition on disc. Is there any downside to this?

Any thoughts??

Ron.
 

TomBascom

Curmudgeon
Back in the good old days the rule of thumb was to allocate 2x memory for swap. By that adage your SysAdmin is getting off lucky.

Modern Linux systems don’t need that, they allocate swap “optimistically” and rarely run short anymore. If they do run short then the out-of-memory daemon picks something and kills it. As I recall the algorithm likes to start where it can get the most bang for the buck There are ways to influence this behavior but it’s been a while and I’m not in my office (it’s late here).

I doubt that your real problem is -B. I have seen cases where memory leaks will exhaust memory. Some client parameters can also be surprisingly memory intense (like -Bt as an example) and sometimes get set way too high for everyone, to fix a problem that only impacts one session.

You might also find this useful: https://pugchallenge.org/downloads2017/Dude wheres my memory.pdf
 

TomBascom

Curmudgeon
One more thing... if you are getting the “out of swap, cannot fork” message you should be aware that the message really ought to read: “cannot fork, I might be out of swap (but it’s probably something else)”.
 

ron

Member
Yes, Rob, I honestly have encountered that sort of logic!

---------------

To be clear -- the question I'm asking is like this. We sometimes run out of swap. I am confident that if we added an extra swap partition of (say) 10 GB that would have zero impact on real memory and stop us running out of swap and having the kernel kill-off a couple of processes.

But is there any down-side to this? Is it as simple as it seems to me -- or would doing this cause some other problem that I am not aware of?

Ron.
 
Hi Ron,

If you :
- set your startup parameters "-baseindex and -basetableas" to -1,610 and -16,384 (in your post "Question about VST Table/Index I/O Statistics" )
- do not set -baseuserindex and -baseusertable,
- have a big "-n"
You could have a huge number of record in the VST _UserTableStat and _UserIndexStat hold in memory . Thoses record are created when the Db start, even if no user are connected to the DB.

If you have "not enough" memory on your machine , you must take care about this.

Patrice
 

TomBascom

Curmudgeon
Making the swap partition larger has no direct downside. Indirectly I suppose you might argue that you are (potentially) covering up “the real problem” with a band-aid.
 

ron

Member
Thank you Patrice, but two different servers are involved. VST statistics collection is on a Production system where there is no current issue with RAM or swap. It is a development server where I have the problem and VST statistics are only collected in rare cases where I'm investigating a problem. However -- thank you for highlighting this, because I can reduce the tablerange and indexrange to "nil" on the Development system, which I have not done.
 

TomBascom

Curmudgeon
What is the actual error that you are getting that indicates that you are out of swap space?

Another diagnostic thing to do is to run "ipcs -m" and find out if there are any orphaned shared memory segments laying around.
 

ron

Member
The errors are kernel messages "out of memeory". I have a monitor process that records many metrics including RAM and virtual memory every minute. Available virtual memory is always quite small and in the minute before an OOM event it gets really small and the next minute it jumps up by about 18 GB because two broker processes have been killed-off.
 

TomBascom

Curmudgeon
I agree that adding swap space will very likely give you at least short term relief.

Your system description, however, does not sound like one where you should be seeing this kind of problem. You have 82GB of RAM and only a small fraction of that is going to db buffers (less than 16GB from what you said). I am very suspicious that something very unusual is happening.

I can think of two similar cases that I've seen in the last couple of years:

1) The application uses temp-tables very heavily and the -Bt parameter was set very high in order to avoid growth of DBI* temp files. The idea was to keep the big temp tables in memory to keep client performance fast. That worked until the user count grew and the amount of memory being allocated for -Bt exceeded the amount of RAM on the system (if I recall *each* user had something like 512MB being allocated and there were about 500 users when they ran out of RAM). Only a few of those users actually needed more than 20MB so the RAM was mostly being wasted. Setting -Bt smaller by default and only increasing it for selected users cured the problem.

2) Alluded to in Nectar's presentation above - a large file (the nightly backup if I remember) was being written to a memory backed filesystem. As I recall the problem occurred because a tape device that used to be in /dev was no longer there. The backup user was "root" so it happily created the missing device as an ordinary, but quite large, file. And since /dev is a "tmpfs" filesystem that ate up all the RAM. Pointing the backup to a *disk* cured that problem.
 

ron

Member
How naive of me to imagine that working out how much memory various processes were consuming might be "easy". :) Ha!

I have read and tried-out all sorts of forum posts on the subject and I've come to the conclusion that there is a kind of "Uncertainty Principle" at play here. When I use ps and get %mem -- and combine it with the system total memory to calculate the memory used by each process it gives "reasonable-looking results" -- but the sum is far greater than the memory of the system. My initial guess is that ps is using information like that provided by pmap -- which itemises not only the memory that is dedicated to a process, but all shared libraries used by several other processes.

So -- in a "kinda sorta" way this gives an indication of how much memory each process is dealing with, although some (maybe most) of that memory is actually shared among many other processes.

So -- I think I will leave that issue alone for now.

This discussion has made me see that I can reduce the memory usage of a number of processes that usually run on the Development system, although I don't expect it to be a lot .... let's see. I'm going to press the System Admin to bump-up swap as well. It costs "nothing" and will stop future instances of the kernel panicking and killing-off two really (really!) important brokers. They are not Production -- but they do get used by some "important people".

Ron.
 

TomBascom

Curmudgeon
As you noticed you cannot get a useful view of global memory usage from “ps”. If it helps anyone feel better the same is true for taskmgr on Windows. In both cases it is because shared segments of various kinds are being counted repeatedly.
 

Rob Fitzpatrick

ProgressTalk.com Sponsor
However -- thank you for highlighting this, because I can reduce the tablerange and indexrange to "nil" on the Development system, which I have not done.
As a counterpoint, I have seen lots of bad code get into production precisely because developers weren't paying attention to their CRUD stats. I recommend configuring your DBs so you can look at your CRUD stats wherever you run, and especially where you develop, data-access code.

I do agree with Patrice that you have to pay careful attention in production (or production-like) databases, as the combination of very large rangesizes together with very large user counts (-n) can use a fair bit of memory. But I wouldn't expect you to have so many users in a development DB that this becomes a problem.
 

ron

Member
Yes, Rob, I agree with that. As I mentioned the Development server has many copies of the production DB, but only two of them get moderately robust configurations -- the ones you might refer to as "Test" and "UAT". They are the two involved with SLDC and do get (for example) full values for -tablerange and -indexrange. All others get these parameters set to be very low.
 
Hi,
I am agree with Rob,
You can set your parameters -baseindex and -basetableas" to -1,610 and -16,384.
But if you want to minimze the cost in memory you could set -baseuserindex and -baseusertable to 1 and 1 (or 1 and -362).
With a setting like this one , you elimante the big(ger) part of the records in the VST _UserTableStat and _UserIndexStat, but the most insterresting record for the developper are still there (those for your own tables).

Patrice
 

ron

Member
Yes, Patrice, I appreciate what you're saying. I do need a very expansive range for collecting table and index statistics on the production system -- but it has quite enough memory to support that. The system where I am having a memory problem is the Development/Testing server and on that server the table and index range sizes are allowed to default -- and I will even make them 1, because that information is unwanted on this server.

Having said that -- I may set the ranges large enough for the application (but not Auditing) for the UAT environment. I'll think about that one.

Ron.
 

ron

Member
I appreciate that this is really a Sys Admin question -- but in my experience Progress DBAs are experts on nearly everything that opens-and-shuts ... so here goes.

Following on from the above discussion, my Sys Admin and I appear to have a different understanding of virtual memory. I would like to outline my fairly simplistic view on how it works.

Many processes will start and they get space allocated in swap FIRST. And then -- if there is available space in RAM the process will "swap in" and execute. This continues with all the processes in RAM and executing -- their "virtual image" (ie space) remains allocated in swap.

If a point is reached where a process is to start and there is not enough space in RAM to accommodate it then (according to complex rules that are not important here) the kernel will choose one or more processes currently in RAM and "swap out" -- copying their memory image to swap space. That frees-up some memory that can be used by the new process.

And so on it goes.

Now -- the most important point I'm trying to get resolved is that even though there is enough real memory for all processes to execute without any "swapping" -- every process will have space allocated in swap space.

Also .... if my Development system has 82 GB of RAM -- but only 21 GB of swap space ... I think I have a big problem.


Please tell me -- am I on the right track? Or not?

Ron.
 

Rob Fitzpatrick

ProgressTalk.com Sponsor
I confess I don't know a lot about Linux memory management. But I believe your understanding is not quite correct, for a few reasons. First, my understanding is that the swap file is used only when there is memory pressure and pages of physical memory must be freed to service new requests. (Intuitively, this makes more sense to me given that memory is thousands of times faster than disk; why write to swap every time if it is not necessary?) Second, you should think of swap operations in terms of individual pages of memory rather than entire processes. Third, not all of the pages of memory of a process necessarily contain data that needs to be written to the swap file if the memory manager needs to free physical pages. Code and unmodified data do not need to be swapped as they are already on disk and can be re-read from their original locations if necessary. Only modified private process data should need to be paged. For that reason, I also believe that your swap file doesn't need to be the same size as your RAM.

This is a link to a fairly recent article on the subject: https://opensource.com/article/18/9/swap-space-linux-systems. It is written about Fedora. I'm sure you're using a different distro, but of course they share the Linux kernel in common.

Hopefully others here have more educated opinions than me. :)
 
Top