Rocksolid Light

Welcome to RetroBBS

mail  files  register  newsreader  groups  login

Message-ID:  

"Beware of programmers carrying screwdrivers." -- Chip Salzenberg


devel / comp.unix.bsd.freebsd.misc / Re: swap space used but nothing obvious using it

SubjectAuthor
* swap space used but nothing obvious using itMike Scott
+* Re: swap space used but nothing obvious using itbob prohaska
|`* Re: swap space used but nothing obvious using itMike Scott
| `* Re: swap space used but nothing obvious using itbob prohaska
|  `* Re: swap space used but nothing obvious using itJohn D Groenveld
|   `* Re: swap space used but nothing obvious using itMike Scott
|    `* Re: swap space used but nothing obvious using itbob prohaska
|     `* Re: swap space used but nothing obvious using itMike Scott
|      +- Re: swap space used but nothing obvious using itbob prohaska
|      `* Re: swap space used but nothing obvious using itJohn D Groenveld
|       `- Re: swap space used but nothing obvious using itMike Scott
`* Re: swap space used but nothing obvious using itAndreas Kempe
 `- (resolved, "gone away") Re: swap space used but nothing obvious usingMike Scott

1
swap space used but nothing obvious using it

<tcfuj1$2oscj$1@dont-email.me>

 copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=255&group=comp.unix.bsd.freebsd.misc#255

 copy link   Newsgroups: comp.unix.bsd.freebsd.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader01.eternal-september.org!.POSTED!not-for-mail
From: usenet.16@scottsonline.org.uk.invalid (Mike Scott)
Newsgroups: comp.unix.bsd.freebsd.misc
Subject: swap space used but nothing obvious using it
Date: Thu, 4 Aug 2022 09:04:48 +0100
Organization: Scott family
Lines: 32
Message-ID: <tcfuj1$2oscj$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Thu, 4 Aug 2022 08:04:49 -0000 (UTC)
Injection-Info: reader01.eternal-september.org; posting-host="8eaaaf8a7f27223667da799c04daf0ee";
logging-data="2912659"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19W2ZHlgEYN7JGqoFQ5f6yZhJzHBaGqGb0="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
Thunderbird/91.11.0
Cancel-Lock: sha1:58OAp6GTugGQAfINMLD+S3B1hoQ=
Content-Language: en-GB
 by: Mike Scott - Thu, 4 Aug 2022 08:04 UTC

Hi all, a bit of a conundrum; maybe I'm missing something.

RPI4 (4Gb) running 13.1, no humungous tasks running on it, just a small
mail, web and file- server for the family.

But the swap is all eaten up, and there seem to be no swapped processes
to account:

root@kirk:~ # uptime
9:02AM up 20 days, 21:21, 2 users, load averages: 1.17, 0.87, 0.51

root@kirk:~ # swapinfo -h
Device Size Used Avail Capacity
/dev/da0s2b 2.0G 1.7G 267M 87%

root@kirk:~ # ps axwju | grep W
root 12 0 0 0 0 WL - 196:45.89 [intr]
23.9 0.0 0 368 14Jul22
root 19009 18964 19008 53800 2 S+ 2 0:00.01 grep W
0.0 0.0 12672 1900 09:01

It completely ran out of swap last night, so I'll reboot. But what's
chewing it all up? I've had much the same system running on i386 for
years without seeing this problem.

TIA.

--
Mike Scott
Harlow, England

Re: swap space used but nothing obvious using it

<tcp6l9$m5bq$1@dont-email.me>

 copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=256&group=comp.unix.bsd.freebsd.misc#256

 copy link   Newsgroups: comp.unix.bsd.freebsd.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader01.eternal-september.org!.POSTED!not-for-mail
From: bp@www.zefox.net (bob prohaska)
Newsgroups: comp.unix.bsd.freebsd.misc
Subject: Re: swap space used but nothing obvious using it
Date: Sun, 7 Aug 2022 20:17:46 -0000 (UTC)
Organization: A noiseless patient Spider
Lines: 38
Message-ID: <tcp6l9$m5bq$1@dont-email.me>
References: <tcfuj1$2oscj$1@dont-email.me>
Injection-Date: Sun, 7 Aug 2022 20:17:46 -0000 (UTC)
Injection-Info: reader01.eternal-september.org; posting-host="8e86ddfe38635a67328461649aec6472";
logging-data="726394"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX18xIbWQHhHLYy8xFNiE99EHU4MKiymOLhk="
User-Agent: tin/2.4.4-20191224 ("Millburn") (FreeBSD/12.3-STABLE (arm))
Cancel-Lock: sha1:s3WIq529dz3LD+bD964RwlIu7zw=
 by: bob prohaska - Sun, 7 Aug 2022 20:17 UTC

Mike Scott <usenet.16@scottsonline.org.uk.invalid> wrote:
>
> RPI4 (4Gb) running 13.1, no humungous tasks running on it, just a small
> mail, web and file- server for the family.
>
> But the swap is all eaten up, and there seem to be no swapped processes
> to account:
>
> root@kirk:~ # uptime
> 9:02AM up 20 days, 21:21, 2 users, load averages: 1.17, 0.87, 0.51
>
> root@kirk:~ # swapinfo -h
> Device Size Used Avail Capacity
> /dev/da0s2b 2.0G 1.7G 267M 87%
>
> root@kirk:~ # ps axwju | grep W
> root 12 0 0 0 0 WL - 196:45.89 [intr]
> 23.9 0.0 0 368 14Jul22
> root 19009 18964 19008 53800 2 S+ 2 0:00.01 grep W
> 0.0 0.0 12672 1900 09:01
>
>
> It completely ran out of swap last night, so I'll reboot. But what's
> chewing it all up? I've had much the same system running on i386 for
> years without seeing this problem.
>
Does top have anything interesting to say?

This question might be worth putting on the freebsd-arm mailing list:
https://lists.freebsd.org/subscription/freebsd-arm

I read the list and don't recall seeing anything similar, nor have
I experienced anything similar on a 13.1 Pi3 or -current Pi4.

HTH,

bob prohaska

Re: swap space used but nothing obvious using it

<tcqfpv$snpa$1@dont-email.me>

 copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=257&group=comp.unix.bsd.freebsd.misc#257

 copy link   Newsgroups: comp.unix.bsd.freebsd.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader01.eternal-september.org!.POSTED!not-for-mail
From: usenet.16@scottsonline.org.uk.invalid (Mike Scott)
Newsgroups: comp.unix.bsd.freebsd.misc
Subject: Re: swap space used but nothing obvious using it
Date: Mon, 8 Aug 2022 08:59:58 +0100
Organization: Scott family
Lines: 93
Message-ID: <tcqfpv$snpa$1@dont-email.me>
References: <tcfuj1$2oscj$1@dont-email.me> <tcp6l9$m5bq$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Mon, 8 Aug 2022 07:59:59 -0000 (UTC)
Injection-Info: reader01.eternal-september.org; posting-host="4a014ec62cc3fdfccab0b3bf956ad456";
logging-data="941866"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX18b80KAPWUEXAJj/Zljh6IPtrQttjrORHY="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
Thunderbird/91.11.0
Cancel-Lock: sha1:L9maSdO2lcnrCg5SV90anyk8IJI=
Content-Language: en-GB
In-Reply-To: <tcp6l9$m5bq$1@dont-email.me>
 by: Mike Scott - Mon, 8 Aug 2022 07:59 UTC

On 07/08/2022 21:17, bob prohaska wrote:
> Mike Scott <usenet.16@scottsonline.org.uk.invalid> wrote:
>>
>> RPI4 (4Gb) running 13.1, no humungous tasks running on it, just a small
>> mail, web and file- server for the family.
>>
>> But the swap is all eaten up, and there seem to be no swapped processes
>> to account:
>>
>> root@kirk:~ # uptime
>> 9:02AM up 20 days, 21:21, 2 users, load averages: 1.17, 0.87, 0.51
>>
>> root@kirk:~ # swapinfo -h
>> Device Size Used Avail Capacity
>> /dev/da0s2b 2.0G 1.7G 267M 87%
>>
>> root@kirk:~ # ps axwju | grep W
>> root 12 0 0 0 0 WL - 196:45.89 [intr]
>> 23.9 0.0 0 368 14Jul22
>> root 19009 18964 19008 53800 2 S+ 2 0:00.01 grep W
>> 0.0 0.0 12672 1900 09:01
>>
>>
>> It completely ran out of swap last night, so I'll reboot. But what's
>> chewing it all up? I've had much the same system running on i386 for
>> years without seeing this problem.
>>
> Does top have anything interesting to say?
>
> This question might be worth putting on the freebsd-arm mailing list:
> https://lists.freebsd.org/subscription/freebsd-arm
>
> I read the list and don't recall seeing anything similar, nor have
> I experienced anything similar on a 13.1 Pi3 or -current Pi4.
>
> HTH,
>
> bob prohaska
>

Thanks for the comment. I'm wary about the discussion lists: too much is
over my head, but I'll take a look, thanks.

I've just this moment experimented a bit. After posting the above, the
system's been running for a few days.

top says:

last pid: 74530; load averages: 1.21, 1.14, 0.80 up 3+23:34:36
08:41:25
108 processes: 2 running, 106 sleeping
CPU: 0.4% user, 0.0% nice, 0.3% system, 0.1% interrupt, 99.3% idle
Mem: 138M Active, 1052M Inact, 1278M Laundry, 580M Wired, 260M Buf, 809M
Free
Swap: 2048M Total, 179M Used, 1869M Free, 8% Inuse

I just did swapoff -a (took a few seconds) and top said:

last pid: 74538; load averages: 0.71, 1.03, 0.78 up 3+23:35:25
08:42:14
97 processes: 1 running, 96 sleeping
CPU: 0.4% user, 0.0% nice, 0.3% system, 0.1% interrupt, 99.3% idle
Mem: 128M Active, 679M Inact, 1447M Laundry, 490M Wired, 171M Buf, 1114M
Free

I've never known what "Laundry" might be :-{, but I would observe that
swap usage went from 179M to none, while Laundry increased by /roughly/
the same amount. Also, the number of "sleeping" process dropped a lot.

(Although as the system's been running a dump script while doing this, I
can't be sure about the figures)

The swap usage just seems to increase monotonically with time. If
there's enough memory, a swapoff/swapon fixes it for a while. If the
swap gets too full, it's can't all be brought into main memory, so the
swapoff won't work. It's not a truly busy machine - 4Gb memory should be
way more than it needs for operation (used to all run with just 1Gb
(i386) a couple of years ago without issue).

I'm tempted to run swapoff/swapon overnight, but then I'd never find
what was wrong :-}

Thanks.

--
Mike Scott
Harlow, England

Re: swap space used but nothing obvious using it

<tcrvmj$11jmd$1@dont-email.me>

 copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=258&group=comp.unix.bsd.freebsd.misc#258

 copy link   Newsgroups: comp.unix.bsd.freebsd.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader01.eternal-september.org!.POSTED!not-for-mail
From: bp@www.zefox.net (bob prohaska)
Newsgroups: comp.unix.bsd.freebsd.misc
Subject: Re: swap space used but nothing obvious using it
Date: Mon, 8 Aug 2022 21:37:23 -0000 (UTC)
Organization: A noiseless patient Spider
Lines: 107
Message-ID: <tcrvmj$11jmd$1@dont-email.me>
References: <tcfuj1$2oscj$1@dont-email.me> <tcp6l9$m5bq$1@dont-email.me> <tcqfpv$snpa$1@dont-email.me>
Injection-Date: Mon, 8 Aug 2022 21:37:23 -0000 (UTC)
Injection-Info: reader01.eternal-september.org; posting-host="1fc0de520e5548fa886bc3580b81d74b";
logging-data="1101517"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19XZ3T/bgeyRfKarD29HkOSpwT49FpSKLc="
User-Agent: tin/2.4.4-20191224 ("Millburn") (FreeBSD/12.3-STABLE (arm))
Cancel-Lock: sha1:iMSa+pXzKX0A+1OfBX/RzMzOWMI=
 by: bob prohaska - Mon, 8 Aug 2022 21:37 UTC

Mike Scott <usenet.16@scottsonline.org.uk.invalid> wrote:
> On 07/08/2022 21:17, bob prohaska wrote:
>> Mike Scott <usenet.16@scottsonline.org.uk.invalid> wrote:
>>>
>>> RPI4 (4Gb) running 13.1, no humungous tasks running on it, just a small
>>> mail, web and file- server for the family.
>>>
>>> But the swap is all eaten up, and there seem to be no swapped processes
>>> to account:
>>>
>>> root@kirk:~ # uptime
>>> 9:02AM up 20 days, 21:21, 2 users, load averages: 1.17, 0.87, 0.51
>>>
>>> root@kirk:~ # swapinfo -h
>>> Device Size Used Avail Capacity
>>> /dev/da0s2b 2.0G 1.7G 267M 87%
>>>
>>> root@kirk:~ # ps axwju | grep W
>>> root 12 0 0 0 0 WL - 196:45.89 [intr]
>>> 23.9 0.0 0 368 14Jul22
>>> root 19009 18964 19008 53800 2 S+ 2 0:00.01 grep W
>>> 0.0 0.0 12672 1900 09:01
>>>
>>>
>>> It completely ran out of swap last night, so I'll reboot. But what's
>>> chewing it all up? I've had much the same system running on i386 for
>>> years without seeing this problem.
>>>
>> Does top have anything interesting to say?
>>
>> This question might be worth putting on the freebsd-arm mailing list:
>> https://lists.freebsd.org/subscription/freebsd-arm
>>
>> I read the list and don't recall seeing anything similar, nor have
>> I experienced anything similar on a 13.1 Pi3 or -current Pi4.
>>
>> HTH,
>>
>> bob prohaska
>>
>
> Thanks for the comment. I'm wary about the discussion lists: too much is
> over my head, but I'll take a look, thanks.
>
If you've found a memory leak it'll be of interest to the developers.
Almost certainly you'll be asked for the output of uname -a and a
description of the storage hardware and the software running.

> I've just this moment experimented a bit. After posting the above, the
> system's been running for a few days.
>
> top says:
>
> last pid: 74530; load averages: 1.21, 1.14, 0.80 up 3+23:34:36
> 08:41:25
> 108 processes: 2 running, 106 sleeping
> CPU: 0.4% user, 0.0% nice, 0.3% system, 0.1% interrupt, 99.3% idle
> Mem: 138M Active, 1052M Inact, 1278M Laundry, 580M Wired, 260M Buf, 809M
> Free
> Swap: 2048M Total, 179M Used, 1869M Free, 8% Inuse
>
>
> I just did swapoff -a (took a few seconds) and top said:
>
> last pid: 74538; load averages: 0.71, 1.03, 0.78 up 3+23:35:25
> 08:42:14
> 97 processes: 1 running, 96 sleeping
> CPU: 0.4% user, 0.0% nice, 0.3% system, 0.1% interrupt, 99.3% idle
> Mem: 128M Active, 679M Inact, 1447M Laundry, 490M Wired, 171M Buf, 1114M
> Free
>
>
> I've never known what "Laundry" might be :-{, but I would observe that
> swap usage went from 179M to none, while Laundry increased by /roughly/
> the same amount. Also, the number of "sleeping" process dropped a lot.
>

AIUI, laundry is memory (virtual or physical) that holds data which will
never be used because the originating process is gone. There some amount
of work in putting it back in the free pool, so the system doesn't bother
so long as it does not need to.

> (Although as the system's been running a dump script while doing this, I
> can't be sure about the figures)
>
> The swap usage just seems to increase monotonically with time. If
> there's enough memory, a swapoff/swapon fixes it for a while. If the
> swap gets too full, it's can't all be brought into main memory, so the
> swapoff won't work. It's not a truly busy machine - 4Gb memory should be
> way more than it needs for operation (used to all run with just 1Gb
> (i386) a couple of years ago without issue).
>
Times have changed considerably 8-) 64-bit does not help.

> I'm tempted to run swapoff/swapon overnight, but then I'd never find
> what was wrong :-}

Top reports (at least on my machines) SIZE and RES. I think size is the
total memory used by a process and RES is how much is actually resident
in phyiscal RAM, with the difference being (I think) in swap. Those values
might bear watching.

hth,

bob prohaska

Re: swap space used but nothing obvious using it

<duhIK.864621$JVi.592054@fx17.iad>

 copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=259&group=comp.unix.bsd.freebsd.misc#259

 copy link   Newsgroups: comp.unix.bsd.freebsd.misc
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!feed1.usenet.blueworldhosting.com!peer03.iad!feed-me.highwinds-media.com!news.highwinds-media.com!fx17.iad.POSTED!not-for-mail
From: groenveld@acm.org (John D Groenveld)
Newsgroups: comp.unix.bsd.freebsd.misc
Subject: Re: swap space used but nothing obvious using it
References: <tcfuj1$2oscj$1@dont-email.me> <tcp6l9$m5bq$1@dont-email.me> <tcqfpv$snpa$1@dont-email.me> <tcrvmj$11jmd$1@dont-email.me>
Organization: Groenveld.US
Lines: 30
Message-ID: <duhIK.864621$JVi.592054@fx17.iad>
X-Complaints-To: abuse@frugalusenet.com
NNTP-Posting-Date: Tue, 09 Aug 2022 00:02:17 UTC
Date: Tue, 09 Aug 2022 00:02:17 GMT
X-Received-Bytes: 1829
 by: John D Groenveld - Tue, 9 Aug 2022 00:02 UTC

In article <tcrvmj$11jmd$1@dont-email.me>,
bob prohaska <bp@www.zefox.net> wrote:
>If you've found a memory leak it'll be of interest to the developers.
>Almost certainly you'll be asked for the output of uname -a and a
>description of the storage hardware and the software running.

Is the OP using tmpfs?
# mount -t tmpfs

>Top reports (at least on my machines) SIZE and RES. I think size is the
>total memory used by a process and RES is how much is actually resident
>in phyiscal RAM, with the difference being (I think) in swap. Those values
>might bear watching.

<URL:https://docs.freebsd.org/en/books/handbook/basics/#basics-processes>
| Below the header is a series of columns containing similar
| information to the output from ps(1), such as the PID, username,
| amount of CPU time, and the command that started the process. By
| default, top(1) also displays the amount of memory space taken by
| the process. This is split into two columns: one for total size and
| one for resident size. Total size is how much memory the application
| has needed and the resident size is how much it is actually using
| now.

I think top(1) output would be useful when swapinfo(8) is reporting
high usage.
# top -d 1 -o res -w

John
groenveld@acm.org

Re: swap space used but nothing obvious using it

<tcvrgv$1pd4p$1@dont-email.me>

 copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=260&group=comp.unix.bsd.freebsd.misc#260

 copy link   Newsgroups: comp.unix.bsd.freebsd.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader01.eternal-september.org!.POSTED!not-for-mail
From: usenet.16@scottsonline.org.uk.invalid (Mike Scott)
Newsgroups: comp.unix.bsd.freebsd.misc
Subject: Re: swap space used but nothing obvious using it
Date: Wed, 10 Aug 2022 09:50:39 +0100
Organization: Scott family
Lines: 74
Message-ID: <tcvrgv$1pd4p$1@dont-email.me>
References: <tcfuj1$2oscj$1@dont-email.me> <tcp6l9$m5bq$1@dont-email.me>
<tcqfpv$snpa$1@dont-email.me> <tcrvmj$11jmd$1@dont-email.me>
<duhIK.864621$JVi.592054@fx17.iad>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Wed, 10 Aug 2022 08:50:39 -0000 (UTC)
Injection-Info: reader01.eternal-september.org; posting-host="c830aebc9e2d15d3b85fa777bb3a772e";
logging-data="1881241"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19cSws9LAHVKwNknd8uPTjHwOiPAYTNf2s="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
Thunderbird/91.11.0
Cancel-Lock: sha1:OHxUjNRu9oea6Eoqj6Ru8ENAIOQ=
In-Reply-To: <duhIK.864621$JVi.592054@fx17.iad>
Content-Language: en-GB
 by: Mike Scott - Wed, 10 Aug 2022 08:50 UTC

On 09/08/2022 01:02, John D Groenveld wrote:

(Thanks to you and to Bob for replying)

> In article <tcrvmj$11jmd$1@dont-email.me>,
> bob prohaska <bp@www.zefox.net> wrote:
>> If you've found a memory leak it'll be of interest to the developers.
>> Almost certainly you'll be asked for the output of uname -a and a
>> description of the storage hardware and the software running.
>
> Is the OP using tmpfs?
> # mount -t tmpfs

Yes - and I'd forgotten that so thanks for the reminder. But it's not
the cause, as it's limited to 50Mb, and currently using only 4.5k

>>> Top reports (at least on my machines) SIZE and RES. I think size is the
>> total memory used by a process and RES is how much is actually resident
>> in phyiscal RAM, with the difference being (I think) in swap. Those
values
>> might bear watching.
> ....>
> I think top(1) output would be useful when swapinfo(8) is reporting
> high usage.
> # top -d 1 -o res -w

That just returns nothing for me. However,
# top -w -o swap 10000 | cat

returns a list of 106 processes, all with 0B swap usage listed. Yet
swapinfo says:

Device Size Used Avail Capacity
/dev/da0s2b 2.0G 246M 1.8G 12%

(wrote that yesterday, did a swapoff/swapon to clear things out, and
left a monitor script running swapinfo overnight. Then today.....)

At ~3.05 am, something started gobbling up swap space: it cranks up from
0 at 03:05 to 129M by 03:20. Then at 04:03 it started increasing again
reaching 372Mb by 04:16.

That's what it was this morning when I did a 'top -w 10000 |cat', which
tells me every process has 0B swap usage.

Yet, AFAICS crontab is free of anything running at those times apart
from periodic.daily (03:01); maybe I need to shift the time of that to
check.

I do have 2 jails running, for web and sendmail servers. stopping the
mail jail reduces usage to 143Mb, stopping web jail takes it down to
94Mb (restarting them keeps the reduced level). Yet clamd (which takes,
I've found, a horrendous amount of memory) I stopped yesterday without
affecting the swap usage one jot.

Presumably there's something big (not necessarily jailed) that runs that
forces other processes into swap before it exits, but if that's the
case, everything left ought to fit into main memory, which it clearly
doesn't as swap usage just goes monotonically upwards to the point where
swapoff can't work.

I guess I'll have to change the monitor script to do a ps when swapinfo
says swap is on the up.

Tedious.

--
Mike Scott
Harlow, England

Re: swap space used but nothing obvious using it

<td1m5u$1vn35$1@dont-email.me>

 copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=261&group=comp.unix.bsd.freebsd.misc#261

 copy link   Newsgroups: comp.unix.bsd.freebsd.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader01.eternal-september.org!.POSTED!not-for-mail
From: bp@www.zefox.net (bob prohaska)
Newsgroups: comp.unix.bsd.freebsd.misc
Subject: Re: swap space used but nothing obvious using it
Date: Thu, 11 Aug 2022 01:31:42 -0000 (UTC)
Organization: A noiseless patient Spider
Lines: 24
Message-ID: <td1m5u$1vn35$1@dont-email.me>
References: <tcfuj1$2oscj$1@dont-email.me> <tcp6l9$m5bq$1@dont-email.me> <tcqfpv$snpa$1@dont-email.me> <tcrvmj$11jmd$1@dont-email.me> <duhIK.864621$JVi.592054@fx17.iad> <tcvrgv$1pd4p$1@dont-email.me>
Injection-Date: Thu, 11 Aug 2022 01:31:42 -0000 (UTC)
Injection-Info: reader01.eternal-september.org; posting-host="aa27523d67c54df921d7d4a16446b719";
logging-data="2088037"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19LEOsvraWU4nXuyFcMgm5Wy+DIhURc9pM="
User-Agent: tin/2.4.4-20191224 ("Millburn") (FreeBSD/12.3-STABLE (arm))
Cancel-Lock: sha1:j1UAxaF9xgzhMoRRn1hQYTMGu6o=
 by: bob prohaska - Thu, 11 Aug 2022 01:31 UTC

Mike Scott <usenet.16@scottsonline.org.uk.invalid> wrote:

> Presumably there's something big (not necessarily jailed) that runs that
> forces other processes into swap before it exits, but if that's the
> case, everything left ought to fit into main memory, which it clearly
> doesn't as swap usage just goes monotonically upwards to the point where
> swapoff can't work.
>
Are you using a custom kernel, perchance?

[risking carrying coals to Newcastle]
Here's a script prepared with the help of Mark Millard which was used to
investigate problems during self-hosting on Pi2 and Pi3 systems. It'll
have to be hacked to look at what you want, but might be a starting point.

I found it not-too-hard to grep the swapinfo lines, sort them and then
search buildworld.log to find what was happening when swap use soared.
You'd probably want to replace the sysctl with ps -aux or something like
it (apparently I was looking for power supply problems at last use).
hth,

bob prohaska

Re: swap space used but nothing obvious using it

<td2g5g$24nfl$1@dont-email.me>

 copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=262&group=comp.unix.bsd.freebsd.misc#262

 copy link   Newsgroups: comp.unix.bsd.freebsd.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader01.eternal-september.org!.POSTED!not-for-mail
From: usenet.16@scottsonline.org.uk.invalid (Mike Scott)
Newsgroups: comp.unix.bsd.freebsd.misc
Subject: Re: swap space used but nothing obvious using it
Date: Thu, 11 Aug 2022 09:55:11 +0100
Organization: Scott family
Lines: 70
Message-ID: <td2g5g$24nfl$1@dont-email.me>
References: <tcfuj1$2oscj$1@dont-email.me> <tcp6l9$m5bq$1@dont-email.me>
<tcqfpv$snpa$1@dont-email.me> <tcrvmj$11jmd$1@dont-email.me>
<duhIK.864621$JVi.592054@fx17.iad> <tcvrgv$1pd4p$1@dont-email.me>
<td1m5u$1vn35$1@dont-email.me>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Thu, 11 Aug 2022 08:55:12 -0000 (UTC)
Injection-Info: reader01.eternal-september.org; posting-host="f6e83ac0e40d71976350c8d80d6cb17c";
logging-data="2252277"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1/2uaLISV6M1lSUteEfvXPubE2+t1RJH8g="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
Thunderbird/91.11.0
Cancel-Lock: sha1:Bow4YyFGo65jPKQhx/xoyrRLSuY=
In-Reply-To: <td1m5u$1vn35$1@dont-email.me>
Content-Language: en-GB
 by: Mike Scott - Thu, 11 Aug 2022 08:55 UTC

On 11/08/2022 02:31, bob prohaska wrote:
> Mike Scott <usenet.16@scottsonline.org.uk.invalid> wrote:
>
>> Presumably there's something big (not necessarily jailed) that runs that
>> forces other processes into swap before it exits, but if that's the
>> case, everything left ought to fit into main memory, which it clearly
>> doesn't as swap usage just goes monotonically upwards to the point where
>> swapoff can't work.
>>
> Are you using a custom kernel, perchance?

Thanks for comments. No. Can't face the thought of building one on the
Pi :-| Works fine as is though.

>
> [risking carrying coals to Newcastle]
> Here's a script prepared with the help of Mark Millard which was used to
> investigate problems during self-hosting on Pi2 and Pi3 systems. It'll
> have to be hacked to look at what you want, but might be a starting point.

Seems to have been left off :(

However, here's my offering give or take line wrap:

#!/bin/sh

LOG="/var/tmp/swap.topinfo"
DELAY=60

echo >>$LOG
echo "starting...." >>$LOG

lastswap="XXXX"

while true ; do
swap=`swapinfo -h | grep /dev | awk '{print $3}'`
if [ "$swap" != "$lastswap" ] ; then
echo "==================" >>$LOG

echo -n `date +"%Y-%m-%dT%H:%M:%S "` >>$LOG
swapinfo -h | grep /dev >>$LOG

top -wS 10000 >>$LOG # make terminal wide
enough!!!!!!!! truncates if not
echo >>$LOG

ps axwwud >>$LOG
echo >>$LOG

lastswap="$swap"
fi

sleep $DELAY
done

I had it running last night - or rather managed to accidentally run 2
copies using the same log file -- and top had truncated the last column
anyway. Very muddly - but it's nevertheless clear that /everything/ is
listed as 0B in the swap used column, even when swapinfo shows the space
rocketed to >400Mb used (later dropped to 190Mb). Nothing obvious amiss
otherwise.

I've restarted (one copy this time!) and swap used forced back to 0.
Have to await what happens tonight.

--
Mike Scott
Harlow, England

Re: swap space used but nothing obvious using it

<td4n04$2e7mc$1@dont-email.me>

 copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=263&group=comp.unix.bsd.freebsd.misc#263

 copy link   Newsgroups: comp.unix.bsd.freebsd.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader01.eternal-september.org!.POSTED!not-for-mail
From: bp@www.zefox.net (bob prohaska)
Newsgroups: comp.unix.bsd.freebsd.misc
Subject: Re: swap space used but nothing obvious using it
Date: Fri, 12 Aug 2022 05:04:04 -0000 (UTC)
Organization: A noiseless patient Spider
Lines: 92
Message-ID: <td4n04$2e7mc$1@dont-email.me>
References: <tcfuj1$2oscj$1@dont-email.me> <tcp6l9$m5bq$1@dont-email.me> <tcqfpv$snpa$1@dont-email.me> <tcrvmj$11jmd$1@dont-email.me> <duhIK.864621$JVi.592054@fx17.iad> <tcvrgv$1pd4p$1@dont-email.me> <td1m5u$1vn35$1@dont-email.me> <td2g5g$24nfl$1@dont-email.me>
Injection-Date: Fri, 12 Aug 2022 05:04:04 -0000 (UTC)
Injection-Info: reader01.eternal-september.org; posting-host="56eaf62fc3c49d87b6722196d8e81407";
logging-data="2563788"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1+t3O7gcO9mWtxWTNr0IMih8JDigrp2oMY="
User-Agent: tin/2.4.4-20191224 ("Millburn") (FreeBSD/12.3-STABLE (arm))
Cancel-Lock: sha1:hoOhb+ULefb60U7kcT3TYVetD8A=
 by: bob prohaska - Fri, 12 Aug 2022 05:04 UTC

Mike Scott <usenet.16@scottsonline.org.uk.invalid> wrote:
> On 11/08/2022 02:31, bob prohaska wrote:
>> Are you using a custom kernel, perchance?
>
> Thanks for comments. No. Can't face the thought of building one on the
> Pi :-| Works fine as is though.
>
Kernels aren't bad. Clang can be difficult.
If you've got a standard kernel using all its swap the folks
at freebsd.org would probably want to know it.

>>
>> [risking carrying coals to Newcastle]
>
> Seems to have been left off :(
>
Here's another try:
bob@www:~/fbsd % more swapscript
#!/bin/sh
while true
sysctl hw.regulator.5v0.min_uvolt ; do vmstat ; gstat -abd -I 10s ; date ; swapinfo ; tail \
-n 2 /var/log/messages ; netstat -m | grep "mbuf clusters" ; tail -n 1 \
/usr/src/buildworld.log
done

The idea is to log things of interest (swap use) with things like swap use
numbers and (in this case) make output lines that can be searched for in
the log file. Grep and sort find the interesting swap numbers, and a search
in more on the log files finds what the system was doing at the time.

It's tedious.....

> However, here's my offering give or take line wrap:
>
> #!/bin/sh
>
> LOG="/var/tmp/swap.topinfo"
> DELAY=60
>
> echo >>$LOG
> echo "starting...." >>$LOG
>
> lastswap="XXXX"
>
> while true ; do
> swap=`swapinfo -h | grep /dev | awk '{print $3}'`
> if [ "$swap" != "$lastswap" ] ; then
> echo "==================" >>$LOG
>
Are you looking for changes in swap use here?
> echo -n `date +"%Y-%m-%dT%H:%M:%S "` >>$LOG
> swapinfo -h | grep /dev >>$LOG
>
> top -wS 10000 >>$LOG # make terminal wide
> enough!!!!!!!! truncates if not
> echo >>$LOG
>
> ps axwwud >>$LOG
> echo >>$LOG
>
> lastswap="$swap"
> fi
>
> sleep $DELAY
> done
>
>
> I had it running last night - or rather managed to accidentally run 2
> copies using the same log file -- and top had truncated the last column
> anyway. Very muddly - but it's nevertheless clear that /everything/ is
> listed as 0B in the swap used column, even when swapinfo shows the space
> rocketed to >400Mb used (later dropped to 190Mb). Nothing obvious amiss
> otherwise.
>

I don't claim to understand the intent of your script, but I'd suggest
not worrying if the output is readable, as in trying to find the problem.
Rather make it a series of snapshots of the machine state over time.
Let the resulting file be big, with long lines. Then tease out the
times when swap use is high and and use the swap numbers as indexes
to the log file to see what's going on.

I can't help but wonder if the swap use is somehow coming from the kernel.

Apologies in advance if I'm missing your point! This does seem worth
a post or two on the freebsd mailing list(s)

bob prohaska

Re: swap space used but nothing obvious using it

<td5quh$2gaj$1@nyheter.lysator.liu.se>

 copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=264&group=comp.unix.bsd.freebsd.misc#264

 copy link   Newsgroups: comp.unix.bsd.freebsd.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader01.eternal-september.org!nyheter.lysator.liu.se!.POSTED!not-for-mail
From: kempe@lysator.liu.se (Andreas Kempe)
Newsgroups: comp.unix.bsd.freebsd.misc
Subject: Re: swap space used but nothing obvious using it
Date: Fri, 12 Aug 2022 15:17:37 -0000 (UTC)
Organization: Lysator ACS
Message-ID: <td5quh$2gaj$1@nyheter.lysator.liu.se>
References: <tcfuj1$2oscj$1@dont-email.me>
Injection-Date: Fri, 12 Aug 2022 15:17:37 -0000 (UTC)
Injection-Info: nyheter.lysator.liu.se; posting-account="kempe";
logging-data="82259"; mail-complaints-to="newsmaster@lysator.liu.se"
User-Agent: slrn/1.0.3 (FreeBSD)
 by: Andreas Kempe - Fri, 12 Aug 2022 15:17 UTC

Den 2022-08-04 skrev Mike Scott <usenet.16@scottsonline.org.uk.invalid>:
> Hi all, a bit of a conundrum; maybe I'm missing something.
>
> RPI4 (4Gb) running 13.1, no humungous tasks running on it, just a small
> mail, web and file- server for the family.
>
> But the swap is all eaten up, and there seem to be no swapped processes
> to account:
>
> root@kirk:~ # uptime
> 9:02AM up 20 days, 21:21, 2 users, load averages: 1.17, 0.87, 0.51
>
> root@kirk:~ # swapinfo -h
> Device Size Used Avail Capacity
> /dev/da0s2b 2.0G 1.7G 267M 87%
>
> root@kirk:~ # ps axwju | grep W
> root 12 0 0 0 0 WL - 196:45.89 [intr]
> 23.9 0.0 0 368 14Jul22
> root 19009 18964 19008 53800 2 S+ 2 0:00.01 grep W
> 0.0 0.0 12672 1900 09:01
>
>
> It completely ran out of swap last night, so I'll reboot. But what's
> chewing it all up? I've had much the same system running on i386 for
> years without seeing this problem.
>

If you aren't seeing processes in top eating more and more memory, you
could try having a look at vmstat -m which will show how the kernel
has allocated heap memory. It could be something in the kernel being
the culprit.

> TIA.
>
>

Re: swap space used but nothing obvious using it

<qKvJK.650852$70j.536396@fx16.iad>

 copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=265&group=comp.unix.bsd.freebsd.misc#265

 copy link   Newsgroups: comp.unix.bsd.freebsd.misc
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!feed1.usenet.blueworldhosting.com!peer01.iad!feed-me.highwinds-media.com!news.highwinds-media.com!fx16.iad.POSTED!not-for-mail
From: groenveld@acm.org (John D Groenveld)
Newsgroups: comp.unix.bsd.freebsd.misc
Subject: Re: swap space used but nothing obvious using it
References: <tcfuj1$2oscj$1@dont-email.me> <tcvrgv$1pd4p$1@dont-email.me> <td1m5u$1vn35$1@dont-email.me> <td2g5g$24nfl$1@dont-email.me>
Organization: Groenveld.US
Lines: 15
Message-ID: <qKvJK.650852$70j.536396@fx16.iad>
X-Complaints-To: abuse@frugalusenet.com
NNTP-Posting-Date: Fri, 12 Aug 2022 17:04:22 UTC
Date: Fri, 12 Aug 2022 17:04:22 GMT
X-Received-Bytes: 1048
 by: John D Groenveld - Fri, 12 Aug 2022 17:04 UTC

In article <td2g5g$24nfl$1@dont-email.me>,
Mike Scott <usenet.16@scottsonline.org.uk.invalid> wrote:
>Thanks for comments. No. Can't face the thought of building one on the
>Pi :-| Works fine as is though.

Ok, have you updated to p1 as advised?
<URL:https://www.freebsd.org/security/advisories/FreeBSD-SA-22:11.vm.asc>

>Seems to have been left off :(

You have left off the full output of ps and top when your swapinfo
usage exceeds your preferred maximum.

John
groenveld@acm.org

Re: swap space used but nothing obvious using it

<tdgr3c$4ofr$1@dont-email.me>

 copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=266&group=comp.unix.bsd.freebsd.misc#266

 copy link   Newsgroups: comp.unix.bsd.freebsd.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader01.eternal-september.org!.POSTED!not-for-mail
From: usenet.16@scottsonline.org.uk.invalid (Mike Scott)
Newsgroups: comp.unix.bsd.freebsd.misc
Subject: Re: swap space used but nothing obvious using it
Date: Tue, 16 Aug 2022 20:27:39 +0100
Organization: Scott family
Lines: 47
Message-ID: <tdgr3c$4ofr$1@dont-email.me>
References: <tcfuj1$2oscj$1@dont-email.me> <tcvrgv$1pd4p$1@dont-email.me>
<td1m5u$1vn35$1@dont-email.me> <td2g5g$24nfl$1@dont-email.me>
<qKvJK.650852$70j.536396@fx16.iad>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Tue, 16 Aug 2022 19:27:40 -0000 (UTC)
Injection-Info: reader01.eternal-september.org; posting-host="8ad66bca2ceb639c5910d00a529fc825";
logging-data="156155"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX186ZOCKqPr+7p1aClp7ZcVNHrDoQsGJ9oU="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
Thunderbird/91.11.0
Cancel-Lock: sha1:IB4QkglAi1RZKoeoJtXbftjI8FE=
In-Reply-To: <qKvJK.650852$70j.536396@fx16.iad>
Content-Language: en-GB
 by: Mike Scott - Tue, 16 Aug 2022 19:27 UTC

On 12/08/2022 18:04, John D Groenveld wrote:
> In article <td2g5g$24nfl$1@dont-email.me>,
> Mike Scott <usenet.16@scottsonline.org.uk.invalid> wrote:
>> Thanks for comments. No. Can't face the thought of building one on the
>> Pi :-| Works fine as is though.
>
> Ok, have you updated to p1 as advised?
> <URL:https://www.freebsd.org/security/advisories/FreeBSD-SA-22:11.vm.asc>
>
>> Seems to have been left off :(
>
> You have left off the full output of ps and top when your swapinfo
> usage exceeds your preferred maximum.

Yes, they seemed long, and I wasn't sure about the security of
publishing the entire contents, which would divulge exactly what
software my server runs.

But, meanwhile, I've been checking, changing and testing. It seems (and
IMBW) that clamd may be part of the problem. It's the biggest job on the
system at around 1.3G, and it turns out (which I didn't know before)
that it replicates itself when updating its virus tables, to ensure
continuity of service. That would chew up most of the available main memory.

So I've changed clamd's config to not do that, and the monotonic
increase in swap seems to have gone away. At least, it's been stable at
up to only ~280M after running for a few days.

FWIW I've now done a top and ps listing, then swapoff/swapon to clear
swap usage, and repeated the top/ps listing. Swap dropped from 284M to
0, but a diff on the two sets of listings shows nothing (that I can
spot) untoward.

I'll run for some days more and see what happens, then put clamd back to
being 'memory hungry' and see if the problem recurs. Can't think of
anything better ATM.

Thanks all.

>
> John
> groenveld@acm.org

--
Mike Scott
Harlow, England

(resolved, "gone away") Re: swap space used but nothing obvious using it

<teldm5$1gvbn$1@dont-email.me>

 copy mid

https://www.rocksolidbbs.com/devel/article-flat.php?id=269&group=comp.unix.bsd.freebsd.misc#269

 copy link   Newsgroups: comp.unix.bsd.freebsd.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader01.eternal-september.org!.POSTED!not-for-mail
From: usenet.16@scottsonline.org.uk.invalid (Mike Scott)
Newsgroups: comp.unix.bsd.freebsd.misc
Subject: (resolved, "gone away") Re: swap space used but nothing obvious using
it
Date: Tue, 30 Aug 2022 17:25:40 +0100
Organization: Scott family
Lines: 47
Message-ID: <teldm5$1gvbn$1@dont-email.me>
References: <tcfuj1$2oscj$1@dont-email.me>
<td5quh$2gaj$1@nyheter.lysator.liu.se>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Tue, 30 Aug 2022 16:25:41 -0000 (UTC)
Injection-Info: reader01.eternal-september.org; posting-host="4792d2580f6804b6e702776d41e1e477";
logging-data="1604983"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1+Eq4UPjn9mN8bD4D+6FsgyTxRXuAhrvbc="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
Thunderbird/91.11.0
Cancel-Lock: sha1:xr+r/APhEvRrh1W7Co/bkId7WkI=
Content-Language: en-GB
In-Reply-To: <td5quh$2gaj$1@nyheter.lysator.liu.se>
 by: Mike Scott - Tue, 30 Aug 2022 16:25 UTC

On 12/08/2022 16:17, Andreas Kempe wrote:
> Den 2022-08-04 skrev Mike Scott <usenet.16@scottsonline.org.uk.invalid>:
>> Hi all, a bit of a conundrum; maybe I'm missing something.
>>
>> RPI4 (4Gb) running 13.1, no humungous tasks running on it, just a small
>> mail, web and file- server for the family.
>>
>> But the swap is all eaten up, and there seem to be no swapped processes
>> to account:
......

>
> If you aren't seeing processes in top eating more and more memory, you
> could try having a look at vmstat -m which will show how the kernel
> has allocated heap memory. It could be something in the kernel being
> the culprit.
>
>> TIA.
>>
>>

Thanks to all for comments.

I tinkered with the setup for clamav's update, first removing the
memory-greedy update stuff - which seemed to fix the problem - then
putting it back, which didn't bring the problem back.

So I have a system now the same as when the problem reared its head, but
behaving reasonably. Swap use goes up to around 500Mb, then drops to
250Mb or thereabouts. There have been system updates in the interim, I
believe, so maybe something changed there.

I can't troubleshoot further as it's "gone away" and I remain bemused by
the original symptoms of programs that should fit main memory, but
don't, and nothing being marked as swapped out.

It's irritating that there seems no easy way of finding a list of stuff
using the swap file.

But at least things are now manageable, and I've learned a bit.

Thanks again.

--
Mike Scott
Harlow, England

1
server_pubkey.txt

rocksolid light 0.9.7
clearnet tor