Where does RasPiOS put log files related to ssh connections?

I've got a Pi4 that's dropping individual ssh connections. One
particular connection out of a dozen or so drops, so it's
probably related to the server end, not the client.

It'd be helpful to examine the client log for clues, but
the logging setup on RasPiOS is very different from
BSD and and I'm unsure where to look.

The only message displayed on LxTerminal at the RasPiOS end is

......
client_loop: send disconnect: Broken pipe
......

Thanks for reading, and any guidance!

bob prohaska

bob prohaska <bp@www.zefox.net> wrote:
> Where does RasPiOS put log files related to ssh connections?
>
> I've got a Pi4 that's dropping individual ssh connections. One
> particular connection out of a dozen or so drops, so it's
> probably related to the server end, not the client.
>
> It'd be helpful to examine the client log for clues, but
> the logging setup on RasPiOS is very different from
> BSD and and I'm unsure where to look.
>
> The only message displayed on LxTerminal at the RasPiOS end is
>
> .....
> client_loop: send disconnect: Broken pipe
> .....
>
> Thanks for reading, and any guidance!

It's normally in /var/log/auth.log, but that's for the setup part of SSH
where it does authentication. Otherwise in general syslog, as accessed via
'journalctl'. You can increase the amount of debug logging in
/etc/ssh/sshd_config

But usually a broken pipe means the TCP socket dropped for some reason,
which is not the fault of SSH. It can be due to aggressive timeouts on NAT
routers - mobile networks are notorious for that. Search for 'ssh
keepalive' for details of how to make SSH send null packets back and forth
every so often to prevent timeouts.

It could also be wifi related, if the wifi dropped out for some while then
eventually the TCP connection would drop.

If you affected by TCP drops, 'mosh' is a tool that runs on top of SSH that
automatically reconnects and maintains the connection as it was before. It
also does things like caching keystrokes and replaying them when the
connection comes back up.

Theo

On 1/12/24 08:14, bob prohaska wrote:
> Where does RasPiOS put log files related to ssh connections?
>
> I've got a Pi4 that's dropping individual ssh connections. One
> particular connection out of a dozen or so drops, so it's
> probably related to the server end, not the client.
>
> It'd be helpful to examine the client log for clues, but
> the logging setup on RasPiOS is very different from
> BSD and and I'm unsure where to look.
>
> The only message displayed on LxTerminal at the RasPiOS end is
>
> .....
> client_loop: send disconnect: Broken pipe
> .....
>
> Thanks for reading, and any guidance!
>
> bob prohaska
>

In the journal.

journalctl -u ssh

will get you what you want. Also be sure to look at the man page for
journalctl.

--

Knute Johnson

Knute Johnson <knute2023@585ranch.com> wrote:
> On 1/12/24 08:14, bob prohaska wrote:
>> Where does RasPiOS put log files related to ssh connections?
>>
>> I've got a Pi4 that's dropping individual ssh connections. One
>> particular connection out of a dozen or so drops, so it's
>> probably related to the server end, not the client.
>>
>> It'd be helpful to examine the client log for clues, but
>> the logging setup on RasPiOS is very different from
>> BSD and and I'm unsure where to look.
>>
>> The only message displayed on LxTerminal at the RasPiOS end is
>>
>> .....
>> client_loop: send disconnect: Broken pipe
>> .....
>>
>> Thanks for reading, and any guidance!
>>
>> bob prohaska
>>
>
> In the journal.
>
> journalctl -u ssh
>
> will get you what you want. Also be sure to look at the man page for
> journalctl.

That's a good start. The -u ssh arguments seem to produce records
of ssh server activity but nothing about outgoing client connections.
In my case the problem is with an outgoing (client) session. The
journalctl -u ssh command reported no ssh client activity at all.
Is there a list of valid unit names somewhere?

Many thanks!

bob prohaska

On 1/12/24 09:10, bob prohaska wrote:
>
> Is there a list of valid unit names somewhere?
>
> Many thanks!
>
> bob prohaska
>

If you find it, please post it here.

--

Knute Johnson

bob prohaska <bp@www.zefox.net> wrote:
> That's a good start. The -u ssh arguments seem to produce records
> of ssh server activity but nothing about outgoing client connections.
> In my case the problem is with an outgoing (client) session. The
> journalctl -u ssh command reported no ssh client activity at all.

Ah, you didn't make it clear it was the client. Try:

ssh -vvv user@host

bob prohaska <bp@www.zefox.net> writes:
> That's a good start. The -u ssh arguments seem to produce records
> of ssh server activity but nothing about outgoing client connections.
> In my case the problem is with an outgoing (client) session. The
> journalctl -u ssh command reported no ssh client activity at all.

The client doesn’t normally log to the journal. Run your client with
‘ssh -vvv’ for debug logging from the client . However Theo has already
interpreted the diagnostic; you’re probably not going to get much more
out of it than that.

Server logging seems to be quite thin by default. You could try
increasing the LogLevel in /etc/ssh/sshd_config (see ‘man sshd_config’
for possible values). You’ll need to restart the server for this to take
effect (systemctl restart ssh).

However, empirically, it doesn’t seem to have much to say when a
connection is terminated abnormally, so don’t get your hopes up.

I think Theo is probably right about what’s going on here.

> Is there a list of valid unit names somewhere?

systemctl list-units

--
https://www.greenend.org.uk/rjk/

On Fri, 12 Jan 2024 15:10:42 -0000 (UTC), bob prohaska wrote:

> Is there a list of valid unit names somewhere?

systemctl list-unit-files

On 2024-01-12, Theo <theom+news@chiark.greenend.org.uk> wrote:
> bob prohaska <bp@www.zefox.net> wrote:
>> I've got a Pi4 that's dropping individual ssh connections. One
>> particular connection out of a dozen or so drops, so it's
>> probably related to the server end, not the client.
>
....snip...
>
> But usually a broken pipe means the TCP socket dropped for some reason,
> which is not the fault of SSH. It can be due to aggressive timeouts on NAT
> routers - mobile networks are notorious for that. Search for 'ssh
> keepalive' for details of how to make SSH send null packets back and forth
> every so often to prevent timeouts.

I have

ServerAliveInterval 30

in my ~/.ssh/config in the header so it is set for all my connections,
precisely because some of my connections were being dropped because of
inactivity causing an intermediate firewall/router to close down the
connection.

It's been that way for longer than I care to remember, so my memory is a
bit dim, but I know I played around with some other options that didn't
always seem to work.

>
> It could also be wifi related, if the wifi dropped out for some while then
> eventually the TCP connection would drop.
>
> If you affected by TCP drops, 'mosh' is a tool that runs on top of SSH that
> automatically reconnects and maintains the connection as it was before. It
> also does things like caching keystrokes and replaying them when the
> connection comes back up.
>
> Theo

Richard Kettlewell <invalid@invalid.invalid> wrote:
> bob prohaska <bp@www.zefox.net> writes:
>> That's a good start. The -u ssh arguments seem to produce records
>> of ssh server activity but nothing about outgoing client connections.
>> In my case the problem is with an outgoing (client) session. The
>> journalctl -u ssh command reported no ssh client activity at all.
>
> The client doesn?t normally log to the journal. Run your client with
> ?ssh -vvv? for debug logging from the client . However Theo has already
> interpreted the diagnostic; you?re probably not going to get much more
> out of it than that.
>
> Server logging seems to be quite thin by default. You could try
> increasing the LogLevel in /etc/ssh/sshd_config (see ?man sshd_config?
> for possible values). You?ll need to restart the server for this to take
> effect (systemctl restart ssh).
>
> However, empirically, it doesn?t seem to have much to say when a
> connection is terminated abnormally, so don?t get your hopes up.
>
> I think Theo is probably right about what?s going on here.
>
>> Is there a list of valid unit names somewhere?
>
> systemctl list-units
>
Thank you!

bob prohaska