Discussion:
rsync buffer overflow detected
Boris Savelev via rsync
2017-04-14 17:22:29 UTC
Permalink
Hello!

I use rsync from python on my Debian Jessie amd64 and get this error:
*** buffer overflow detected ***: <snip>/rsync terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x731af)[0x7ffff78971af]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7ffff791caa7]
/lib/x86_64-linux-gnu/libc.so.6(+0xf6cc0)[0x7ffff791acc0]
/lib/x86_64-linux-gnu/libc.so.6(+0xf8a17)[0x7ffff791ca17]
<snip>/rsync(+0x30c78)[0x555555584c78]
<snip>/rsync(+0x31cfe)[0x555555585cfe]
<snip>/rsync(+0x31ef6)[0x555555585ef6]
<snip>/rsync(+0x336ed)[0x5555555876ed]
<snip>/rsync(+0x22417)[0x555555576417]
<snip>/rsync(+0x2395e)[0x55555557795e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7ffff7845b45]
<snip>/rsync(+0x7f89)[0x55555555bf89]

I guess that problem is about too many open fds
STR for this is a small script on python:
import os
import subprocess

F = 'test'
OPENS = 1600

cmd = [
#'gdb', '--args',
'./rsync',
'-aviH',
'/etc/passwd',
'/tmp/passwd'
]

for i in xrange(OPENS):
fd = os.open(F, os.O_WRONLY | os.O_CREAT)
print(cmd)
subprocess.check_call(cmd)

I rebuild rsync-3.1.1 from Debian source with debug and -O1 and get bt from gdb:
(gdb) bt
#0 0x00007ffff7859067 in __GI_raise (sig=***@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007ffff785a448 in __GI_abort () at abort.c:89
#2 0x00007ffff78971b4 in __libc_message (do_abort=***@entry=2,
fmt=***@entry=0x7ffff7989cb3 "*** %s ***: %s terminated\n")
at ../sysdeps/posix/libc_fatal.c:175
#3 0x00007ffff791caa7 in __GI___fortify_fail
(msg=***@entry=0x7ffff7989c4a "buffer overflow detected") at
fortify_fail.c:31
#4 0x00007ffff791acc0 in __GI___chk_fail () at chk_fail.c:28
#5 0x00007ffff791ca17 in __fdelt_chk (d=***@entry=1606) at fdelt_chk.c:25
#6 0x0000555555584c78 in safe_read (fd=***@entry=1606,
buf=***@entry=0x7fffffffa810 "\037", len=***@entry=4) at io.c:245
#7 0x0000555555585cfe in read_buf (f=***@entry=1606,
buf=***@entry=0x7fffffffa810 "\037", len=***@entry=4) at io.c:1815
#8 0x0000555555585ef6 in read_int (f=***@entry=1606) at io.c:1711
#9 0x00005555555876ed in setup_protocol (f_out=1605, f_in=1606) at compat.c:158
#10 0x0000555555576417 in client_run (f_in=1606, f_out=1605,
pid=24793, argc=1, argv=0x5555557d5240) at main.c:1128
#11 0x000055555557795e in start_client (argv=0x5555557d5240, argc=1)
at main.c:1423
#12 main (argc=2, argv=0x5555557d5240) at main.c:1651

It looks like a bug, but I'm not sure)

--
Boris
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
devzero--- via rsync
2017-04-16 06:43:08 UTC
Permalink
What's the value of "i" when this happens and what are the system ulimit values for the user running that?

Roland
Gesendet: Freitag, 14. April 2017 um 19:22 Uhr
Betreff: rsync buffer overflow detected
Hello!
*** buffer overflow detected ***: <snip>/rsync terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x731af)[0x7ffff78971af]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7ffff791caa7]
/lib/x86_64-linux-gnu/libc.so.6(+0xf6cc0)[0x7ffff791acc0]
/lib/x86_64-linux-gnu/libc.so.6(+0xf8a17)[0x7ffff791ca17]
<snip>/rsync(+0x30c78)[0x555555584c78]
<snip>/rsync(+0x31cfe)[0x555555585cfe]
<snip>/rsync(+0x31ef6)[0x555555585ef6]
<snip>/rsync(+0x336ed)[0x5555555876ed]
<snip>/rsync(+0x22417)[0x555555576417]
<snip>/rsync(+0x2395e)[0x55555557795e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7ffff7845b45]
<snip>/rsync(+0x7f89)[0x55555555bf89]
I guess that problem is about too many open fds
import os
import subprocess
F = 'test'
OPENS = 1600
cmd = [
#'gdb', '--args',
'./rsync',
'-aviH',
'/etc/passwd',
'/tmp/passwd'
]
fd = os.open(F, os.O_WRONLY | os.O_CREAT)
print(cmd)
subprocess.check_call(cmd)
(gdb) bt
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007ffff785a448 in __GI_abort () at abort.c:89
at ../sysdeps/posix/libc_fatal.c:175
#3 0x00007ffff791caa7 in __GI___fortify_fail
fortify_fail.c:31
#4 0x00007ffff791acc0 in __GI___chk_fail () at chk_fail.c:28
#9 0x00005555555876ed in setup_protocol (f_out=1605, f_in=1606) at compat.c:158
#10 0x0000555555576417 in client_run (f_in=1606, f_out=1605,
pid=24793, argc=1, argv=0x5555557d5240) at main.c:1128
#11 0x000055555557795e in start_client (argv=0x5555557d5240, argc=1)
at main.c:1423
#12 main (argc=2, argv=0x5555557d5240) at main.c:1651
It looks like a bug, but I'm not sure)
--
Boris
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Boris Savelev via rsync
2017-04-17 08:14:05 UTC
Permalink
ulimit:
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 8192
coredump(blocks) 0
memory(kbytes) unlimited
locked memory(kbytes) 64
process 64098
nofiles 65536
vmemory(kbytes) unlimited
locks unlimited


Rsync call at the end, so i = 1600

modified test script:

---8<---
import os
import subprocess
#import psutil

F = 'test'
OPENS = 1600

cmd = [
#'gdb', '--args',
#'strace', '-f', '-olog',
'./rsync',
'-aviH',
'/etc/passwd',
'/tmp/passwd'
]

for i in xrange(OPENS):
fd = os.open(F, os.O_WRONLY | os.O_CREAT)

os.system('ulimit -a')
#proc = psutil.Process()
#print len(proc.open_files())
subprocess.check_call(cmd)
---8<---

strace:
https://pastebin.com/Xu9Pq4R9

If system limit excided it may fall with EMFILE, I think
Post by devzero--- via rsync
What's the value of "i" when this happens and what are the system ulimit values for the user running that?
Roland
Gesendet: Freitag, 14. April 2017 um 19:22 Uhr
Betreff: rsync buffer overflow detected
Hello!
*** buffer overflow detected ***: <snip>/rsync terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x731af)[0x7ffff78971af]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7ffff791caa7]
/lib/x86_64-linux-gnu/libc.so.6(+0xf6cc0)[0x7ffff791acc0]
/lib/x86_64-linux-gnu/libc.so.6(+0xf8a17)[0x7ffff791ca17]
<snip>/rsync(+0x30c78)[0x555555584c78]
<snip>/rsync(+0x31cfe)[0x555555585cfe]
<snip>/rsync(+0x31ef6)[0x555555585ef6]
<snip>/rsync(+0x336ed)[0x5555555876ed]
<snip>/rsync(+0x22417)[0x555555576417]
<snip>/rsync(+0x2395e)[0x55555557795e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7ffff7845b45]
<snip>/rsync(+0x7f89)[0x55555555bf89]
I guess that problem is about too many open fds
import os
import subprocess
F = 'test'
OPENS = 1600
cmd = [
#'gdb', '--args',
'./rsync',
'-aviH',
'/etc/passwd',
'/tmp/passwd'
]
fd = os.open(F, os.O_WRONLY | os.O_CREAT)
print(cmd)
subprocess.check_call(cmd)
(gdb) bt
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007ffff785a448 in __GI_abort () at abort.c:89
at ../sysdeps/posix/libc_fatal.c:175
#3 0x00007ffff791caa7 in __GI___fortify_fail
fortify_fail.c:31
#4 0x00007ffff791acc0 in __GI___chk_fail () at chk_fail.c:28
#9 0x00005555555876ed in setup_protocol (f_out=1605, f_in=1606) at compat.c:158
#10 0x0000555555576417 in client_run (f_in=1606, f_out=1605,
pid=24793, argc=1, argv=0x5555557d5240) at main.c:1128
#11 0x000055555557795e in start_client (argv=0x5555557d5240, argc=1)
at main.c:1423
#12 main (argc=2, argv=0x5555557d5240) at main.c:1651
It looks like a bug, but I'm not sure)
--
Boris
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--
Boris
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Wayne Davison via rsync
2017-04-29 21:40:06 UTC
Permalink
On Fri, Apr 14, 2017 at 10:22 AM, Boris Savelev via rsync <
#9 0x00005555555876ed in setup_protocol (f_out=1605, f_in=1606) at
Post by Boris Savelev via rsync
compat.c:158
Since rsync is just trying to setup the protocol, I'd imagine that your
shell is outputting extraneous characters. You can try running something
manually, such as "ssh host echo hello", which should output only "hello",
and "ssh host rsync --server . ." (note the 2 trailing dot args) which
should output nothing, and require a Ctrl-C to kill it. You might be able
to work around the issue with something like -e'ssh -q'.

..wayne..
Lars Ellenberg via rsync
2017-05-04 14:41:44 UTC
Permalink
Post by Boris Savelev via rsync
*** buffer overflow detected ***: <snip>/rsync terminated
(gdb) bt
That is FD_SET(fd, &r_fds); with fd >= FD_SETSIZE, which is 1024.
You cannot use select with file descriptor numbers >= FD_SETSIZE (or < 0),
and glibc is catching that.

The "buffer" that would overflow is the fd_set.

Maybe rsync could simply close all inherited file descriptors,
first things first, before it does anything else,
possibly after making sure fds 0,1,2 are open to somewhere,
to avoid any output to "supposedly" stdout/stderr to clobber
fds opened only later. Similar to what lvm tools do in their
_check_standard_fds() and _close_stray_fds()?

But of course rsync could also say: not my problem, *you* (whatever
entity was spawning rsync) leaked file descriptors, learn to use
O_CLOEXEC resp. set FD_CLOEXEC, so only 0,1,2 will be inherited.

quick and dirty workaround:
use a wrapper script, close all fds >= 3 "just in case",
then exec rsync.
Post by Boris Savelev via rsync
It looks like a bug, but I'm not sure)
Thanks,

Lars Ellenberg
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Loading...