remote port forwarding unstable

[Adriana Rodean]

Hi,

I'm not sure if this is a bug or not, maybe someone noticed it also...

I always do a remote port forwarding with openssh on 1026 port let's
say ( ssh -R 1026:localhost:55555 ). Most times the port is opened on
remote machine. But sometimes i notice that ssh can't do remote port
forwarding to that port 1026. I looked on the remote machine (netstat
-an) and no one is using that port, so the port is free.
Only way to fix this is do a remote port forwarding to another port
lets say 1056, successfully done, then try again and do it for 1026,
this time remote port forwarding successfully works... Sometimes it
works if i try again with 1026, but other times i need to open another
port then try again with 1026 port...

What can cause this instability to remote port forwarding?
Is there another command than "netstat -an" to see if that port is
really free or something is using it?
If is a bug can it be fixed?

I use version of OpenSSH 5.1p1 on remote machine and the client is
OpenSSH for Windows 3.8.1p1

Thanks in advance,
Adriana

[Adriana Rodean]

Hi,

Here is some more info after little investigation :)

Message i get is: "Warning: remote port forwarding failed for listen port 1026"

SSH makes connection but port isn't opened and in server logs i see:

IPP-Linux:~# cat /var/log/auth.log | grep 18722
Oct 19 13:37:20 IPP-Linux sshd[18722]: error: bind: Address already in use
Oct 19 13:37:20 IPP-Linux sshd[18722]: error:
channel_setup_fwd_listener: cannot listen to port: 1026
IPP-Linux:~# cat /var/log/auth.log | grep 18737
Oct 19 13:37:47 IPP-Linux sshd[18737]: error: bind: Address already in use
Oct 19 13:37:47 IPP-Linux sshd[18737]: error:
channel_setup_fwd_listener: cannot listen to port: 1026

But i closed the previous console with ssh listening to that port, and
is no ssh process on client when i want to connect again on the same
port. So how come it still says that port is in use on server?

This happens in only 2% of the cases, very rare, i change nothing in
the way i connect or close the client, but somehow it seems that port
still hangs on server after closing ssh console ... or at least that's
what sshd says because with "netstat -an" i don't see that port busy

Thank you Rabbi for that command, next time when it happens i'll use it :)

Thanks,
Adriana

[Quintin Beukes]

Hey,

How do you close the console? And, can you share the command with the
list please.

For some reason replying on this list does so to the sender and not to
the list by default. So the list didn't receive the reply you did.

Quintin Beukes

Post by Adriana Rodean
Hi,
Here is some more info after little investigation :)
Message i get is: "Warning: remote port forwarding failed for listen port 1026"
IPP-Linux:~# cat /var/log/auth.log | grep 18722
Oct 19 13:37:20 IPP-Linux sshd[18722]: error: bind: Address already in use
channel_setup_fwd_listener: cannot listen to port: 1026
IPP-Linux:~# cat /var/log/auth.log | grep 18737
Oct 19 13:37:47 IPP-Linux sshd[18737]: error: bind: Address already in use
channel_setup_fwd_listener: cannot listen to port: 1026
But i closed the previous console with ssh listening to that port, and
is no ssh process on client when i want to connect again on the same
port.  So how come it still says that port is in use on server?
This happens in only 2% of the cases, very rare, i change nothing in
the way i connect or close the client, but somehow it seems that port
still hangs on server after closing ssh console ... or at least that's
what sshd says because with "netstat -an" i don't see that port busy
Thank you Rabbi for that command, next time when it happens i'll use it :)
Thanks,
Adriana

[Greg Wooledge]

Post by Adriana Rodean
IPP-Linux:~# cat /var/log/auth.log | grep 18737
Oct 19 13:37:47 IPP-Linux sshd[18737]: error: bind: Address already in use
channel_setup_fwd_listener: cannot listen to port: 1026
But i closed the previous console with ssh listening to that port, and
is no ssh process on client when i want to connect again on the same
port. So how come it still says that port is in use on server?

Use "lsof -i :1026" (lsof is not standard, but it's very common)
or "netstat -antp | grep :1026" (netstat -p is Linux only) to see what
is listening on port 1026.

[Adriana Rodean]

Thank you all for your replies :)

This is the command i use: ssh -L 30300:localhost:8080 -R
1026:localhost:55555 -F ssh_config -N ***@10.55.40.120
And i close the ssh process by closing the windows console, or by
killing the ssh process from another application. Either way when i
look in processes list after closing ssh process is gone when i try to
reconnect again.

I guess that's the explanation suggested above why sometimes it
doesn't allow me to reconnect with same port, TCP connection staying
in a wait state even after previous ssh client process terminates and
it keeps that remote port busy.
Maybe that's why it didn't showed with "netstat -an" command.
Next time i will use the other commands suggested :)

Thank you again,
Adriana

[Quintin Beukes]

Post by Quintin Beukes
Hey,
How do you close the console? And, can you share the command with the
list please.

The reason I'm asking this is that the fact it is a bind: address
already in use error, means the bind() call failed. So according to
the networking stack that port is still bound. A netstat command on
the remote server should definitely show this. The commands Greg
listed (the lsof -i :1026 and netstat -antp) will give you this
information.

Remember to run the 2 commands as root (lsof needs to be root, and for
netstat's -p to work as well).

Further, you can also try: netstat -antpl | grep :1026
This will filter into only showing matching listening ports. I often
do this because it's so easy to miss it among all those ports.

Q

[Alexander Klimov]

Post by Adriana Rodean
I always do a remote port forwarding with openssh on 1026 port let's
say ( ssh -R 1026:localhost:55555 ). Most times the port is opened on
remote machine. But sometimes i notice that ssh can't do remote port
forwarding to that port 1026. I looked on the remote machine (netstat
-an) and no one is using that port, so the port is free.
Only way to fix this is do a remote port forwarding to another port
lets say 1056, successfully done, then try again and do it for 1026,
this time remote port forwarding successfully works... Sometimes it
works if i try again with 1026, but other times i need to open another
port then try again with 1026 port...
[..]
I use version of OpenSSH 5.1p1 on remote machine and the client is
OpenSSH for Windows 3.8.1p1

Nobody uses the port, but it is still in TIME_WAIT state.
Usually openssh uses SO_REUSEADDR to say the kernel that the
port can be reused while in TIME_WAIT state, but to avoid X11
man-in-the-middle attack the portable version of OpenSSH
5.1 does not set it if you have X11UseLocalhost=no.

So, you should either wait a little after each closing of the
port before trying to use it again, or set X11UseLocalhost=yes.