Discussion:
logging file names with sftp
(too old to reply)
Agile Aspect
2009-09-02 16:39:20 UTC
Permalink
Hi - I' m trying to log file names of the files which are transferred
(basically in the same manner as vsftp, proftpd, etc.) but regardless
of how high I turn on debugging, the file name won't show up!

I'm running openssh-5.2p1.

Any help would be greatly appreciated.

-- Agile
--
"We are drowning in information and starving for knowledge."

--
Rutherford D. Roger
Darren Tucker
2009-09-03 15:31:10 UTC
Permalink
Post by Agile Aspect
Hi - I' m trying to log file names of the files which are transferred
(basically in the same manner as vsftp, proftpd, etc.) but regardless
of how high I turn on debugging, the file name won't show up!
How did you try configuring it?

Basically you just need to add the -l and -f options to the sftp-server
line in sshd_config to specify the appropriate syslog level and facility
and restart sshd to pick up the changes. See sftp-server(8) man page
for details. syslog will also need to be configured appropriately.
--
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
Agile Aspect
2009-09-03 15:33:37 UTC
Permalink
Hi Darren - thanks for the reply!

The so-called "normal" logging is working - the problem is it's tragically
incomplete.

We store IP information on the server. In addition, it's not possible to ha=
ve
security without accountability.

I was able to modify sftp-server.c to our needs, i.e., basically log every
sftp command issued - including the names of the files transferred - tagged
with user name and stored in a seperate log file.

If anyone is interested in what I did I would more than happy to share it -
otherwise I won't bore you to death.

In short, I solved the my problem and thanks!

-- Cinaed
Post by Darren Tucker
Post by Agile Aspect
Hi - I' m trying to log file names of the files which are transferred
(basically in the same manner as vsftp, proftpd, etc.) but regardless
of how high I turn on debugging, the file name won't show up!
How did you try configuring it?
Basically you just need to add the -l and -f options to the sftp-server l=
ine
Post by Darren Tucker
in sshd_config to specify the appropriate syslog level and facility and
restart sshd to pick up the changes. =A0See sftp-server(8) man page for
details. =A0syslog will also need to be configured appropriately.
--
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 =A037C9 C982 80C7 8FF4 FA69
=A0 =A0Good judgement comes with experience. Unfortunately, the experienc=
e
Post by Darren Tucker
usually comes from bad judgement.
--=20
"We are drowning in information and starving for knowledge."

--
Rutherford D. Roger
Derek Martin
2009-09-04 16:02:42 UTC
Permalink
--Q6Ii71d/u7QX3MLh
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Post by Agile Aspect
The so-called "normal" logging is working - the problem is it's
tragically incomplete.
We store IP information on the server. In addition, it's not
possible to have security without accountability.
The logging of individual file transfers arguably buys you very little
though, because the users are legitimate users who are authenticated.
This is generally quite a different situation from FTP installations,
where often the users are anonymous, and tracking downloads of files
may be interesting from a purely statistical point of view (e.g. how
many downloads of a particular game, application, etc., to determine
its popularity). An individual FTP site may not fall into this
category, but FTP software generally needs to cope with this very
common usage.

With sftp sites, the users are (in some sense) people that you know,
and access to data can and should be carefully regulated via file
system permissions. Users should not be physically able to access
anything they shouldn't have access to, and logging file transfers of
files they legitimately should have access to is in most cases little
more than spying on them.

If your site is *especially* security sensitive, this may be called
for (though, if that's the case, you might also want to re-evaluate
whether you really should be providing file access this way), but most
of the time it probably isn't warranted, and may be considered by
some as an unnecessary invasion of privacy.
--
Derek D. Martin
http://www.pizzashack.org/
GPG Key ID: 0x81CFE75D


--Q6Ii71d/u7QX3MLh
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFKn+5RdjdlQoHP510RAvNUAKC4cQF+L1qoujeDBiO8m9dyulxdGACgt6xb
H1nCzQf8qI0pl2kbDLiB5es=
=3XaU
-----END PGP SIGNATURE-----

--Q6Ii71d/u7QX3MLh--
Darren Tucker
2009-09-04 16:05:44 UTC
Permalink
Post by Agile Aspect
Hi Darren - thanks for the reply!
The so-called "normal" logging is working - the problem is it's tragically
incomplete.
We store IP information on the server. In addition, it's not possible to have
security without accountability.
I was able to modify sftp-server.c to our needs, i.e., basically log every
sftp command issued - including the names of the files transferred - tagged
with user name and stored in a seperate log file.
If anyone is interested in what I did I would more than happy to share it -
otherwise I won't bore you to death.
I'm curious about what you needed that wasn't provided out of the box.

sftp-server(8) says about loglevel: "INFO and VERBOSE log transactions
that sftp-server performs on behalf of the client", and setting this in
sshd_config:

Subsystem sftp /usr/local/libexec/sftp-server -f LOCAL7 -l INFO

gives:

2009-09-04T11:07:35.660499+10:00 gate sftp-server[24692]: session opened
for local user dtucker from [127.0.0.1]
2009-09-04T11:07:39.723595+10:00 gate sftp-server[24692]: open
"/home/dtucker/foo" flags READ mode 0666
2009-09-04T11:07:39.725142+10:00 gate sftp-server[24692]: close
"/home/dtucker/foo" bytes read 4763 written 0
2009-09-04T11:07:41.100064+10:00 gate sftp-server[24692]: session closed
for local user dtucker from [127.0.0.1]

Everything you mentioned (user, source IP address, files accessed) is
all there. What did you find missing?
Post by Agile Aspect
In short, I solved the my problem and thanks!
--
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4 37C9 C982 80C7 8FF4 FA69
Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.
Greg Wooledge
2009-09-04 22:00:35 UTC
Permalink
Post by Derek Martin
The logging of individual file transfers arguably buys you very little
though, because the users are legitimate users who are authenticated.
This is generally quite a different situation from FTP installations,
where often the users are anonymous
If I understand correctly, many people run an sftp service which is
essentially an encrypted, NAT-capable version of anonymous FTP. They
offer files (or file hosting space) to a large group of barely-trusted
people, and want to limit or track abuse of the service.

The encryption may be used to prevent spying upon the traffic by
people outside the group.

The ability of sftp to sit behind a NAT firewall (which FTP cannot do --
not with a straight NAT without special hacks) may be essential to
many sites.
Dennis Taylor
2009-09-04 23:15:35 UTC
Permalink
=20
-----Original Message-----
Sent: Friday, September 04, 2009 12:13 PM
Subject: Re: logging file names with sftp
=20
The logging of individual file transfers arguably buys you=20
very little
though, because the users are legitimate users who are=20
authenticated.
This is generally quite a different situation from FTP=20
installations,
where often the users are anonymous
=20
If I understand correctly, many people run an sftp service which is
essentially an encrypted, NAT-capable version of anonymous FTP. They
offer files (or file hosting space) to a large group of barely-trusted
people, and want to limit or track abuse of the service.
=20
The encryption may be used to prevent spying upon the traffic by
people outside the group.
=20
The ability of sftp to sit behind a NAT firewall (which FTP=20
cannot do --
not with a straight NAT without special hacks) may be essential to
many sites.
=20
When one of our technicians or engineers accesses one of our
embedded systems in the field, we have no reasonable expectation
of privacy. All sensitive data on the system belongs to someone
else. We need to be trusted, but also accountable. We have gone
to great lengths to make it impractical for even an authorized
user to steal others' information. Having logs that we cannot tamper
with is one more layer of security, mostly to help track who compromised
the system if they do find away around the other layers. It is
also an industry requirement, and our products are audited by independent
firms to verify that we do in fact log accesses.

I have not yet enabled SFTP, but most likely will in the future, with
logging of all file accesses turned on.
Derek Martin
2009-09-06 15:57:11 UTC
Permalink
--6B2tGqeUN3NZMHUe
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Post by Greg Wooledge
Post by Derek Martin
The logging of individual file transfers arguably buys you very little
though, because the users are legitimate users who are authenticated.
This is generally quite a different situation from FTP installations,
where often the users are anonymous
If I understand correctly, many people run an sftp service which is
essentially an encrypted, NAT-capable version of anonymous FTP. They
offer files (or file hosting space) to a large group of barely-trusted
people, and want to limit or track abuse of the service.
Absolutely; of course I was generalizing, and acknowledged that there
exist cases where the logging is more useful than in others. When the
OP posted, he did not describe his intended purpose; all we knew was
that he wanted to log the file names. I was simply trying to point
out that, depending on his use case, this may be unnecessary,
unwarranted, and un-useful.

I would also point out that even in the case you describe, the
situation is different from an actual anonymous FTP server, and I
think that it's worth pointing out what is available without tracking
individual file transfers.

In general with OpenSSH, for determining abuse, without logging file
names, one already knows the identity of the user, where they logged
in from, and how long they were connected. If the user uses up the
available space on the storage device, you know who they are, and you
can find the files. If you enable system accounting (if the server
has those facilities -- devices based on any of the free unix kernels
generally at least have the possibility) then you can identify when
the user downloads more data then they should be allowed to. All
without logging individual file names. Though admittedly, logging
file names may more efficient in terms of space consumed, and it's
easier to count log lines than process the system accounting file.
But you could certainly build a tool that processes the accounting for
you, and if this is a product, include that as part of your management
tools.
Post by Greg Wooledge
When one of our technicians or engineers accesses one of our
embedded systems in the field, we have no reasonable expectation of
privacy.
Clearly this is for support purposes, and in that case such a policy
makes sense; you don't own the data, and are not an agent of the
person who owns the data -- you have no right to it. But in general
I'm philosophically opposed to the idea of "no reasonable expectation
of privacy." As you might have guessed. ;-) It's become the modus
operandi of the day, but I think that's wrong, from both a moral and
practical standpoint. People tend to feel somewhat stifled if they
know they are being watched. And we have a variety of laws against
watching people without their knowledge not without reason, though of
late they have been lessened and weakened, sadly.

Honestly, what I personally would like to see is more software with
logic that automatically starts logging but only when a behavior
surpasses some configured notion of what's acceptable. This has
multiple benefits (including reducing disk usage of logs). I like my
privacy, and I think we should all generally have more of it than we
typically do today, at the hands of technology.
Post by Greg Wooledge
We need to be trusted, but also accountable.
Above, I'm trying to point out that not logging individual file
transfers does not necessarily prevent accountability. In many cases,
all it does is provide additional detail that's not strictly
necessary or even especially useful. In your case, it may be
particularly useful, or even required.
Post by Greg Wooledge
We have gone to great lengths to make it impractical for even an
authorized user to steal others' information. Having logs that we
cannot tamper with is one more layer of security, mostly to help
track who compromised the system if they do find away around the
other layers.
Sure. Of course, if they find a way around the other layers, most
likely they've managed to escalate their privileges, in which case
your logging is suspect (unless it's to some sort of write-only
media).
Post by Greg Wooledge
It is also an industry requirement, and our products are audited by
independent firms to verify that we do in fact log accesses.
You can log *access* without necessarily logging individual file
transfers. Though again, in your case it may be appropriate or
required.

To be clear, I'm not trying to suggest that one should never log file
transfers, or that it is never useful. I'm simply saying that it is
not always clear that it is necessary, and it is clear that it is not
always necessary. :) Just because you can do something, doesn't mean
you should. I think all of us as technologists should take the time
to consider if what we are doing is ethical and necessary. From a
purely philosophical point of view, I think it is better not to log
individual file transfers unless you can determine that you have a
justifiable need for the data so obtained that you can not obtain
another way. A legal requirement would certainly qualify as a
justifiable need... And (obviously), I think we should apply this
philosophy everywhere, not just to file transfers.

Of course, if you have a bonifide need for a high level of security,
then you do what you have to do. But you ought to at least make sure
that the parties involved have appropriate notice. If you're going to
log transfers of legitimate users, you should notify the users that
you do so.

Just my $.00002
--
Derek D. Martin
http://www.pizzashack.org/
GPG Key ID: 0x81CFE75D


--6B2tGqeUN3NZMHUe
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)

iD8DBQFKob4BdjdlQoHP510RAi0NAJ4yKhYEd82ZkYq2JCAp8i+1qlv9NACeLuHZ
1Kwc4n+9iQqukcRSG3/IvOw=
=Khwf
-----END PGP SIGNATURE-----

--6B2tGqeUN3NZMHUe--
Dennis Taylor
2009-09-08 17:05:55 UTC
Permalink
=20
-----Original Message-----
Sent: Friday, September 04, 2009 9:25 PM
Subject: Re: logging file names with sftp
=20
Post by Greg Wooledge
The logging of individual file transfers arguably buys=20
you very little
Post by Greg Wooledge
though, because the users are legitimate users who are=20
authenticated.
Post by Greg Wooledge
This is generally quite a different situation from FTP=20
installations,
Post by Greg Wooledge
where often the users are anonymous
=20
If I understand correctly, many people run an sftp service which is
essentially an encrypted, NAT-capable version of anonymous=20
FTP. They
Post by Greg Wooledge
offer files (or file hosting space) to a large group of=20
barely-trusted
Post by Greg Wooledge
people, and want to limit or track abuse of the service.
=20
Absolutely; of course I was generalizing, and acknowledged that there
exist cases where the logging is more useful than in others. When the
OP posted, he did not describe his intended purpose; all we knew was
that he wanted to log the file names. I was simply trying to point
out that, depending on his use case, this may be unnecessary,
unwarranted, and un-useful.
Agreed.
=20
I would also point out that even in the case you describe, the
situation is different from an actual anonymous FTP server, and I
think that it's worth pointing out what is available without tracking
individual file transfers. =20
=20
In general with OpenSSH, for determining abuse, without logging file
names, one already knows the identity of the user, where they logged
in from, and how long they were connected. If the user uses up the
available space on the storage device, you know who they are, and you
can find the files. If you enable system accounting (if the server
has those facilities -- devices based on any of the free unix kernels
generally at least have the possibility) then you can identify when
the user downloads more data then they should be allowed to. All
without logging individual file names. Though admittedly, logging
file names may more efficient in terms of space consumed, and it's
easier to count log lines than process the system accounting file.
But you could certainly build a tool that processes the accounting for
you, and if this is a product, include that as part of your management
tools.
=20
Post by Greg Wooledge
When one of our technicians or engineers accesses one of our
embedded systems in the field, we have no reasonable expectation of
privacy. =20
=20
Clearly this is for support purposes, and in that case such a policy
makes sense; you don't own the data, and are not an agent of the
person who owns the data -- you have no right to it. But in general
I'm philosophically opposed to the idea of "no reasonable expectation
of privacy." As you might have guessed. ;-) It's become the modus
operandi of the day, but I think that's wrong, from both a moral and
practical standpoint. People tend to feel somewhat stifled if they
The people being monitored usually are downloading files requested
by the developers. It's really not an issue here. Obviously, I cannot
speak for the original poster.
know they are being watched. And we have a variety of laws against
watching people without their knowledge not without reason, though of
late they have been lessened and weakened, sadly.
=20
I agree in principle, but in this specific case it may be your credit
card number (albeit encrypted) that I am accessing. That certainly is
none of our business. The auditors want to be able to forensically
determine if those files were accessed by any means, with records
kept at least a year. In a perfect world, we would not ever need
to look at the files. To be clear, we do NOT decrypt the sensitive
data, and the keys are not accessible to even authenticated users.
Honestly, what I personally would like to see is more software with
logic that automatically starts logging but only when a behavior
surpasses some configured notion of what's acceptable. This has
multiple benefits (including reducing disk usage of logs). I like my
privacy, and I think we should all generally have more of it than we
typically do today, at the hands of technology.
I like the idea of selective logging. It is just difficult to do
and be sure we do not miss something we should have included. Eventually,
I may have a way to realiably implement that.
=20
Post by Greg Wooledge
We need to be trusted, but also accountable. =20
=20
Above, I'm trying to point out that not logging individual file
transfers does not necessarily prevent accountability. In many cases,
all it does is provide additional detail that's not strictly
necessary or even especially useful. In your case, it may be
particularly useful, or even required.=20
=20
Yes, required.
Post by Greg Wooledge
We have gone to great lengths to make it impractical for even an
authorized user to steal others' information. Having logs that we
cannot tamper with is one more layer of security, mostly to help
track who compromised the system if they do find away around the
other layers. =20
=20
Sure. Of course, if they find a way around the other layers, most
likely they've managed to escalate their privileges, in which case
your logging is suspect (unless it's to some sort of write-only
media).
=20
We enable sending log files to a host that we do not have access to
tamper with. The folks with access to that host do not have
access to our equipment. When they log in using a shell, commands
are logged as well. They have the privileges necessary to stop
the logging if they know how, but the last thing logged will be
them stopping the logging.
Post by Greg Wooledge
It is also an industry requirement, and our products are audited by
independent firms to verify that we do in fact log accesses.
=20
You can log *access* without necessarily logging individual file
transfers. Though again, in your case it may be appropriate or
required.
=20
Definitely required by outside forces. (Card Information Security
Program, Payment Card Industry Data Storage Standards, and so on.)
Under the best of circumstances, all we need to download is the
debug logs for the developers to study over to see what is wrong.
Sometimes we need more.
To be clear, I'm not trying to suggest that one should never log file
transfers, or that it is never useful. I'm simply saying that it is
not always clear that it is necessary, and it is clear that it is not
always necessary. :) Just because you can do something, doesn't mean
you should. I think all of us as technologists should take the time
to consider if what we are doing is ethical and necessary. From a
purely philosophical point of view, I think it is better not to log
individual file transfers unless you can determine that you have a
justifiable need for the data so obtained that you can not obtain
another way. A legal requirement would certainly qualify as a
justifiable need... And (obviously), I think we should apply this
philosophy everywhere, not just to file transfers.
=20
Of course, if you have a bonifide need for a high level of security,
then you do what you have to do. But you ought to at least make sure
that the parties involved have appropriate notice. If you're going to
log transfers of legitimate users, you should notify the users that
you do so.
=20
All dozen or so are aware of why we recently made significant changes
to their access methods.
Just my $.00002
=20
--=20
Derek D. Martin
http://www.pizzashack.org/
GPG Key ID: 0x81CFE75D
=20
=
Continue reading on narkive:
Loading...