rsync and broken symlinks

[Yakov Hrebtov]

Hello!

I need a suggestion.
How to copy broken symlink as symlink with rsync?

Example:
# mkdir src
# ln -s file-that-doesnt-exist src/broken-link
# rsync -a src/ dst/
# ls -l src/
total 0
lrwxrwxrwx 1 root root 22 Dec 16 10:31 broken-link -> file-that-doesnt-exist
# ls -l dst
total 0

I want src/ and dst/ to be identical after sync.

[Wayne Davison]

Post by Yakov Hrebtov
How to copy broken symlink as symlink with rsync?

Your example works fine for me, so run "rsync --version" and ensure that
it even has symlink support compiled into it. E.g.:

rsync version 2.6.3 protocol version 28
Copyright (C) 1996-2004 by Andrew Tridgell and others
<http://rsync.samba.org/>
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
inplace, IPv6, 64-bit system inums, 64-bit internal inums

If it doesn't mention "symlinks", you'll need to compile a new version
(rsync depends on the readlink() function for its symlink support, so
make sure that configure defined HAVE_READLINK).

If the output does mention symlinks, try running rsync without -l set
and make sure that it mentions that it is skipping a non-regular file.
Then, try running it something like -v options and ensure that it
appears in the sender's and receiver's file list.

..wayne..

[Yakov Hrebtov]

WD> Your example works fine for me, so run "rsync --version" and ensure that

My results:
# rsync --version
rsync version 2.6.3 protocol version 28
Copyright (C) 1996-2004 by Andrew Tridgell and others
<http://rsync.samba.org/>
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
inplace, IPv6, 64-bit system inums, 64-bit internal inums

WD> If it doesn't mention "symlinks", you'll need to compile a new version
WD> (rsync depends on the readlink() function for its symlink support, so
WD> make sure that configure defined HAVE_READLINK).

See my results above

WD> If the output does mention symlinks, try running rsync without -l set
WD> and make sure that it mentions that it is skipping a non-regular file.
WD> Then, try running it something like -v options and ensure that it
WD> appears in the sender's and receiver's file list.

# ls -l src/
total 4
lrwxrwxrwx 1 root root 22 Dec 16 10:31 broken-link -> file-that-doesnt-exist
-rw-r--r-- 1 root root 4 Dec 16 13:31 file
lrwxrwxrwx 1 root root 4 Dec 16 13:32 good-link -> file

# rsync -rv src/ dst/
building file list ... done
file
skipping non-regular file "good-link"

sent 121 bytes received 40 bytes 322.00 bytes/sec
total size is 8 speedup is 0.05

what about symlink "broken-link"?
where is the message -- skipping non-regular file "broken-link"?

[Wayne Davison]

Post by Yakov Hrebtov
where is the message -- skipping non-regular file "broken-link"?

I can only imagine that this is some kind of OS misfeature. Try running
rsync under something like "strace -f" (or your OS's equivalent) and see
what is going wrong with the system calls. If you see readlink() return
-1 for the file, that would cause rsync to skip it as non-existent.

..wayne..

[Yakov Hrebtov]

Hello, Wayne.

WD> I can only imagine that this is some kind of OS misfeature. Try running
WD> rsync under something like "strace -f" (or your OS's equivalent) and see
WD> what is going wrong with the system calls. If you see readlink() return
WD> -1 for the file, that would cause rsync to skip it as non-existent.

25737 lstat64("file", {st_mode=S_IFREG|0644, st_size=4, ...}) = 0
25737 access("file", R_OK) = 0
25737 lstat64("good-link", {st_mode=S_IFLNK|0777, st_size=4, ...}) = 0
25737 readlink("good-link", "file", 4095) = 4
25737 access("good-link", R_OK) = 0
25737 lstat64("broken-link", {st_mode=S_IFLNK|0777, st_size=22, ...}) = 0
25737 readlink("broken-link", "file-that-doesnt-exist", 4095) = 22
25737 access("broken-link", R_OK) = -1 ENOENT (No such file or directory)

You see, readlink call is successful. But rsync then trying to access
the referent of symlink... But referent doesnt exist.

my system is GNU/Linux running kernel 2.4.25
Can you give me any advices?

Thank you

[Wayne Davison]

Post by Yakov Hrebtov
You see, readlink call is successful. But rsync then trying to access
the referent of symlink... But referent doesnt exist.

The only time that rsync calls access() on the files it is sending is
when the "ignore nonreadable" parameter is set in a rsyncd.conf file (or
if the default for this value was changed in the loadparam.c file from
the normal value of "False"). So, if you're really doing a copy from an
rsync daemon, check that daemon's setting of the "ignore nonreadable"
parameter. If you're not talking to a daemon, you'll have to figure out
how the value of sDefault.ignore_nonreadable got to be non-zero.

..wayne..

[Dmitry V. Levin]

Hi,

Post by Wayne Davison

Post by Yakov Hrebtov
You see, readlink call is successful. But rsync then trying to access
the referent of symlink... But referent doesnt exist.

The only time that rsync calls access() on the files it is sending is
when the "ignore nonreadable" parameter is set in a rsyncd.conf file (or
if the default for this value was changed in the loadparam.c file from
the normal value of "False"). So, if you're really doing a copy from an
rsync daemon, check that daemon's setting of the "ignore nonreadable"
parameter. If you're not talking to a daemon, you'll have to figure out
how the value of sDefault.ignore_nonreadable got to be non-zero.

I see, it happens due to the patched loadparam.c; I'll change that back.
Thanks,

[Wayne Davison]

Post by Dmitry V. Levin
I see, it happens due to the patched loadparam.c; I'll change that back.

OK -- I'm glad the cause is resolved.

As for the fix, one could argue that "ignore nonreadable" has a bug in
it when dealing with symlinks -- if readlink() worked, the symlink was
readable, so there is no need to call access() on it. The appended
patch changes this so that rsync doesn't ever use access() on a symlink.
Comments or disagreement?

..wayne..

[Paul Slootman]

Post by Wayne Davison
As for the fix, one could argue that "ignore nonreadable" has a bug in
it when dealing with symlinks -- if readlink() worked, the symlink was
readable, so there is no need to call access() on it. The appended
patch changes this so that rsync doesn't ever use access() on a symlink.
Comments or disagreement?

Just to be certain - readlink() is only called if --copy-links is NOT
specified? In that case, I agree.


Paul Slootman

[Wayne Davison]

Post by Paul Slootman
Just to be certain - readlink() is only called if --copy-links is NOT
specified? In that case, I agree.

Correct. If --copy-links was specified, then rsync will either do a
normal file copy of the referent of the symlink (and rsync won't even
know that a symlink was involved), or it will complain that the referent
was missing.

..wayne..

[Dmitry V. Levin]

Hi,

Post by Paul Slootman

Post by Wayne Davison
As for the fix, one could argue that "ignore nonreadable" has a bug in
it when dealing with symlinks -- if readlink() worked, the symlink was
readable, so there is no need to call access() on it. The appended
patch changes this so that rsync doesn't ever use access() on a symlink.

Surely good fix, thanks.

Post by Paul Slootman
Just to be certain - readlink() is only called if --copy-links is NOT
specified? In that case, I agree.

Yes, if copy_links is set, then stat() is called instead.

[Dmitry V. Levin]

Hi,

Post by Wayne Davison

Post by Yakov Hrebtov
where is the message -- skipping non-regular file "broken-link"?

I can only imagine that this is some kind of OS misfeature. Try running
rsync under something like "strace -f" (or your OS's equivalent) and see
what is going wrong with the system calls. If you see readlink() return
-1 for the file, that would cause rsync to skip it as non-existent.

Here is strace log for this case:

$ strace -fF -e trace=readlink,access rsync -a src/ dst/
Process 1328 attached
[pid 1328] --- SIGSTOP (Stopped (signal)) @ 0 (0) ---
[pid 1327] access(".", R_OK) = 0
[pid 1327] readlink("broken-link", "missing", 4095) = 7
[pid 1327] access("broken-link", R_OK) = -1 ENOENT (No such file or directory)
Process 1329 attached
[pid 1329] --- SIGSTOP (Stopped (signal)) @ 0 (0) ---
[pid 1329] --- SIGUSR2 (User defined signal 2) @ 0 (0) ---
Process 1329 detached
[pid 1328] --- SIGCHLD (Child exited) @ 0 (0) ---
Process 1328 detached
--- SIGCHLD (Child exited) @ 0 (0) ---