Question about SCP stalling over VPN

[Matthew Case]

Greetings,

I am attempting to SSH a series of relatively small files (tens of
megabytes) over a VPN connection to a remote linux server. I am able to
SSH between the servers without a problem, but when I use SCP to copy
the files I start seeing the following message more and more until
finally the copy grinds to a halt and eventually times out:

debug2: channel 0: rcvd adjust 131072
debug2: channel 0: rcvd adjust 131072
debug2: channel 0: rcvd adjust 131072
debug2: channel 0: rcvd adjust 131072

I've looked high and low and haven't really come up with anything
definitive. Someone somewhere had mentioned fiddling with MTU settings,
but I'm not really sure what that will accomplish as I am unfamiliar
with what MTU is and does. If this question has been answered
previously, I apologize ahead of time. Thanks!

I am running CentOS 5.3 with OpenSSH 4.3p2-29 on the server I am copying
the files from, and CentOS 5.4 with OpenSSH 4.3p2.36 on the server I am
copying to.
--
Matthew Case
Specialized Business Software
Software Engineer
SCJP 5 Certified
Phone: [phone]
Fax: [phone]




This message and any files transmitted with it may contain information that is privileged, confidential, and exempt from disclosure under applicable law. They are intended solely for the use of the intended recipient. If you are not the intended recipient, distributing, copying, disclosing, or reliance on the contents of this communication is strictly prohibited. If this has reached you in error, kindly destroy this message and notify the sender immediately. Thank you for your assistance.

We attempt to sweep harmful content (e.g. viruses) from e-mail and attachments, however we cannot guarantee their safety and can accept no liability for any resulting damage. The recipient is responsible to verify the safety of this message and any attachments before accepting them.

[Darren Tucker]

Matthew Case wrote:
[...]

Post by Matthew Case
I've looked high and low and haven't really come up with anything
definitive. Someone somewhere had mentioned fiddling with MTU settings,
but I'm not really sure what that will accomplish as I am unfamiliar
with what MTU is and does. If this question has been answered
previously, I apologize ahead of time. Thanks!

This does sound like the MTU problem to which you refer. See
http://www.snailbook.com/faq/mtu-mismatch.auto.html for details.

[John Morrison]

Matt,

If you are using ssh do you need to use scp as well? Or is just plain copy ok?

[...]

Post by Matthew Case
I've looked high and low and haven't really come up with anything
definitive. Someone somewhere had mentioned fiddling with MTU settings, but
I'm not really sure what that will accomplish as I am unfamiliar with what
MTU is and does. If this question has been answered previously, I apologize
ahead of time. Thanks!

This does sound like the MTU problem to which you refer.  See
http://www.snailbook.com/faq/mtu-mismatch.auto.html for details.
--
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4  37C9 C982 80C7 8FF4 FA69
   Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

[Matthew Case]

First and foremost, thank you to everyone for your responses. I checked
the MTU on both sides and it's currently 1500 so I'm assuming it's not a
mismatch. My VPN is a pair of old Netscreen 5xp boxes, and I can't find
anything relating to MTU or packet size in the configuration, but I'm
still looking.

Secondly, to answer your question John, There is no persistent
connection between the servers. I could feasibly set up an NFS share
between the two but I have a sneaking suspicion that if the problem is
some sort of packet mangling by the VPN during file transfers, the
actual mechanism used to transfer the file will be irrelevant. However,
I will set this up and test it and report back my results, most likely
next Monday.

Post by John Morrison
Matt,
If you are using ssh do you need to use scp as well? Or is just plain copy ok?

Post by Darren Tucker
[...]

Post by Matthew Case
I've looked high and low and haven't really come up with anything
definitive. Someone somewhere had mentioned fiddling with MTU settings, but
I'm not really sure what that will accomplish as I am unfamiliar with what
MTU is and does. If this question has been answered previously, I apologize
ahead of time. Thanks!

This does sound like the MTU problem to which you refer. See
http://www.snailbook.com/faq/mtu-mismatch.auto.html for details.
--
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

--
Matthew Case
Specialized Business Software
Software Engineer
SCJP 5 Certified
Phone: [phone]
Fax: [phone]




This message and any files transmitted with it may contain information that is privileged, confidential, and exempt from disclosure under applicable law. They are intended solely for the use of the intended recipient. If you are not the intended recipient, distributing, copying, disclosing, or reliance on the contents of this communication is strictly prohibited. If this has reached you in error, kindly destroy this message and notify the sender immediately. Thank you for your assistance.

We attempt to sweep harmful content (e.g. viruses) from e-mail and attachments, however we cannot guarantee their safety and can accept no liability for any resulting damage. The recipient is responsible to verify the safety of this message and any attachments before accepting them.

[Robin, Robin]

Matt,

To do full diagnosis, do captures at both ends:
tcpdump -w client.pcap -s0 host <IP> and port 22
## so that we don't capture unnecessary stuff, if you are at the client, the <IP> should be the server side

At server side,
tcpdump -w server.pcap -s0 host <IP> and port 22
## so that we don't capture unnecessary stuff, if you are at the server, the <IP> should be the client side

Upload the {client,server}.pcap somewhere and provide the links, that way someone might be able to take a look at it.

This is one thing you can give it a try easily:

Certain network devices I have seen have bugs dealing with TCP SACK.
CentOS 5.3 comes enabled with TCP SACK.
As root, try issuing "sysctl -w net.ipv4.tcp_sack=0", then scp to check if turning SACK off resolve your issues.

Robin



________________________________________
From: ***@securityfocus.com [***@securityfocus.com] On Behalf Of Matthew Case [***@specializedbusinesssoftware.com]
Sent: Tuesday, March 09, 2010 12:44 PM
To: ***@securityfocus.com
Subject: Question about SCP stalling over VPN

Greetings,

I am attempting to SSH a series of relatively small files (tens of
megabytes) over a VPN connection to a remote linux server. I am able to
SSH between the servers without a problem, but when I use SCP to copy
the files I start seeing the following message more and more until
finally the copy grinds to a halt and eventually times out:

debug2: channel 0: rcvd adjust 131072
debug2: channel 0: rcvd adjust 131072
debug2: channel 0: rcvd adjust 131072
debug2: channel 0: rcvd adjust 131072

I've looked high and low and haven't really come up with anything
definitive. Someone somewhere had mentioned fiddling with MTU settings,
but I'm not really sure what that will accomplish as I am unfamiliar
with what MTU is and does. If this question has been answered
previously, I apologize ahead of time. Thanks!

I am running CentOS 5.3 with OpenSSH 4.3p2-29 on the server I am copying
the files from, and CentOS 5.4 with OpenSSH 4.3p2.36 on the server I am
copying to.
--

Matthew Case
Specialized Business Software
Software Engineer
SCJP 5 Certified
Phone: [phone]
Fax: [phone]




This message and any files transmitted with it may contain information that is privileged, confidential, and exempt from disclosure under applicable law. They are intended solely for the use of the intended recipient. If you are not the intended recipient, distributing, copying, disclosing, or reliance on the contents of this communication is strictly prohibited. If this has reached you in error, kindly destroy this message and notify the sender immediately. Thank you for your assistance.

We attempt to sweep harmful content (e.g. viruses) from e-mail and attachments, however we cannot guarantee their safety and can accept no liability for any resulting damage. The recipient is responsible to verify the safety of this message and any attachments before accepting them.