Discussion:
[Bug 12732] New: hard links can cause rsync to block or to silently skip files
L A Walsh via rsync
2017-04-05 20:05:42 UTC
Permalink
Hard link handling seems to be broken when using "rsync -aH --compare-dest". I
1) rsync completes without error message and exit code 0, although some files
are missing from the backup
2) rsync blocks and must be interrupted/killed
....
Further information
===================
This problem exists at least for rsync versions 3.1.0, 3.1.1, and 3.1.2 for
https://lists.samba.org/archive/rsync/2015-April/030092.html
---

I ran rsync 3.1.1 for over a year to help generate
snapshots. I can't say if it copied all the files or not, as
it was backing up a large "/home" partition, BUT, it never hung.
It did take 45min to a few hours to do the compare, but it
was comparing a large amount of data (>750G) w/a snapshot
(another 750G) to dump diffs to a third, and my /home partion
has a *very* large number of hard links.

So I know that hardlinks are handled 'fine' on comparing
'xfs' to 'xfs'.
Latest test on openSUSE 42.2 (x86_64) on ext4 + on nfs with
----
Ah... I'd suspect nfs...

Why are you using nfs? rsync was designed to compare
against local file systems. You should try running rsync
directly from the nfs-host machine to the client and bypassing
NFS. I.e. -- you need to bypass NFS, since local->local
with hardlinks works.

Just checked my /home partition.
find shows 9295431 names (of any type), but du shows
(using du --inodes) shows 4407458 inodes. That means over
half of the filenames are hard linked. While my home
partition takes up 60% more space now, even cutting
those counts in half would still a large number of
hard links -- and rsync didn't crash doing an
rsync of the partition to an empty one, but first comparing
to a previous snapshot (the empty partition ended up
with differences between the main partition & the snapshot.

I'd remove NFS...
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Hansjoerg Lipp via rsync
2017-04-05 21:25:50 UTC
Permalink
Post by L A Walsh via rsync
I ran rsync 3.1.1 for over a year to help generate
snapshots. I can't say if it copied all the files or not, as
it was backing up a large "/home" partition, BUT, it never hung.
It did take 45min to a few hours to do the compare, but it
was comparing a large amount of data (>750G) w/a snapshot
(another 750G) to dump diffs to a third, and my /home partion
has a *very* large number of hard links.
I've been using rsync for many years and it works fine most of the time.
I'm not sure if all of the occasional hangs have the same reason, these
are really hard to track down as they usually occur during large
transfers (e.g. when synchronizing large backup disks). That's why I was
happy that I could find a small test case which triggers this problem.

Does your rsync hang after the sequence of commands described in section
"How to reproduce (2)"?
Post by L A Walsh via rsync
Latest test on openSUSE 42.2 (x86_64) on ext4 + on nfs with
----
Ah... I'd suspect nfs...
Why are you using nfs?
In order to find out if there is a difference when using another file
system type. The most recent tests were on ext4 and on nfs
(independently), older tests were on at least ext3 and xfs. IIRC I only
tested on different OpenSUSE and Debian versions on x86_64 systems, though.
Post by L A Walsh via rsync
Just checked my /home partition.
find shows 9295431 names (of any type), but du shows
(using du --inodes) shows 4407458 inodes. That means over
half of the filenames are hard linked. While my home
partition takes up 60% more space now, even cutting
those counts in half would still a large number of
hard links -- and rsync didn't crash doing an
rsync of the partition to an empty one, but first comparing
to a previous snapshot (the empty partition ended up
with differences between the main partition & the snapshot.
Probably using different options? Can this be some sort of Heisenbug,
nobody can reproduce? Do the two sequences of shell commands work for
you as expected? Please note that both rsync commands in the mail
generated by bugzilla are split into two lines (each): Both rsync
commands should read
rsync PARAMS DIRS >> XXX.log 2>&1

Kind regards,
Hansjoerg
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
L A Walsh via rsync
2017-04-05 23:19:00 UTC
Permalink
Post by Hansjoerg Lipp via rsync
Probably using different options? Can this be some sort of Heisenbug,
nobody can reproduce? Do the two sequences of shell commands work for
you as expected? Please note that both rsync commands in the mail
generated by bugzilla are split into two lines (each): Both rsync
commands should read
---
Actually, I am reproducing your bug exactly as you entered
it.

And it is very sensitive to options.

When I take out the $PWD it works.

First your case:
/tmp> rsync -aHvv --compare-dest=$PWD/dstlt/. $PWD/srclt/. $PWD/baklt/.
|& tee testlt.log
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
./
a is uptodate
d is uptodate
b
a => b
/tmp> ll -i srclt dstlt baklt
baklt:
total 8
17305770 -rw-rw-r-- 2 2 Apr 5 15:46 a
17305770 -rw-rw-r-- 2 2 Apr 5 15:46 b
dstlt:
total 8
8474888 -rw-rw-r-- 1 2 Apr 5 15:46 a
8474896 -rw-rw-r-- 1 2 Apr 5 15:46 d
srclt:
total 16
8866 -rw-rw-r-- 2 2 Apr 5 15:46 a
8866 -rw-rw-r-- 2 2 Apr 5 15:46 b
8867 -rw-rw-r-- 2 2 Apr 5 15:46 c
8867 -rw-rw-r-- 2 2 Apr 5 15:46 d
----
Then w/o $PWD:
/tmp> rsync -aHvv --compare-dest=dstlt/. srclt/. baklt/. |& tee testlt.log
sending incremental file list
--compare-dest arg does not exist: dstlt/.
delta-transmission disabled for local transfer or --whole-file
./
b
d
a => b
c => d
/tmp> ll -i srclt dstlt baklt
baklt:
total 16
17305770 -rw-rw-r-- 2 2 Apr 5 15:46 a
17305770 -rw-rw-r-- 2 2 Apr 5 15:46 b
17305782 -rw-rw-r-- 2 2 Apr 5 15:46 c
17305782 -rw-rw-r-- 2 2 Apr 5 15:46 d
dstlt:
total 8
8474888 -rw-rw-r-- 1 2 Apr 5 15:46 a
8474896 -rw-rw-r-- 1 2 Apr 5 15:46 d
srclt:
total 16
8866 -rw-rw-r-- 2 2 Apr 5 15:46 a
8866 -rw-rw-r-- 2 2 Apr 5 15:46 b
8867 -rw-rw-r-- 2 2 Apr 5 15:46 c
8867 -rw-rw-r-- 2 2 Apr 5 15:46 d
--------------
Post by Hansjoerg Lipp via rsync
rsync -aHv --compare-dest=~/dstlt/. $PWD/srclt/. $PWD/baklt/. |& tee
testlt.log
sending incremental file list
--compare-dest arg does not exist: ~/dstlt/.
./
b
d
a => b
c => d

In my usage both the compare-dest & baklt were 'technically'
different file systems (though the compare-dest was a snap
shot).
-----------
ARG: if baklt is on a different FS, but src+dst are same:
/tmp> rsync -aHv --compare-dest=$PWD/dstlt/. $PWD/srclt/. ~/baklt/. |&
tee testlt.log
sending incremental file list
./
b
a => b
/tmp> ll -i srclt dstlt ~/baklt
/home/law/baklt:
total 8
3677653445 -rw-rw-r-- 2 2 Apr 5 15:46 a
3677653445 -rw-rw-r-- 2 2 Apr 5 15:46 b

dstlt:
total 8
8474888 -rw-rw-r-- 1 2 Apr 5 15:46 a
8474896 -rw-rw-r-- 1 2 Apr 5 15:46 d

srclt:
total 16
8866 -rw-rw-r-- 2 2 Apr 5 15:46 a
8866 -rw-rw-r-- 2 2 Apr 5 15:46 b
8867 -rw-rw-r-- 2 2 Apr 5 15:46 c
8867 -rw-rw-r-- 2 2 Apr 5 15:46 d

----
Ok... something's screwy here. But no hangs.

Like I said in my 1st response, if some hard links
were missing in the bak directory, I really would be
unlikely to notice, as I didn't even try to check
them (way too many files), but I never got a hang --
and that's the part I'm thinking might be nfs
related, since I've seen several issues with nfs not
working the same way as a local fs.

I usually use smbfs -- in my usage, it's faster and I
usually have fewer compatibility problems.
(faster meaning ~ Reads~162MB/s, W~220MB/s,
though i've seen explorer hit over 400MB/s).
(using a 10Gb link).

But the above testing shows some unexplained behaviors
out of rsync that sure look like a bug.

Good test case!
:-)
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Kevin Korb via rsync
2017-04-05 23:29:42 UTC
Permalink
Note that in all --*-dest options if the path is relative it is relative
to the target dir not the "$PWD". I like to always use absolute paths
because of this. But essentially, the command with the instances of
$PWD vs without them the paths aren't the same. If you want all
relative to . paths it would be:
rsync -aHvv --compare-dest=../dstlt/. srclt/. baklt/.

Also, all these params that end with /. the . is kinda pointless despite
being points ;)
Post by L A Walsh via rsync
Post by Hansjoerg Lipp via rsync
Probably using different options? Can this be some sort of Heisenbug,
nobody can reproduce? Do the two sequences of shell commands work for
you as expected? Please note that both rsync commands in the mail
generated by bugzilla are split into two lines (each): Both rsync
commands should read
---
Actually, I am reproducing your bug exactly as you entered
it.
And it is very sensitive to options.
When I take out the $PWD it works.
/tmp> rsync -aHvv --compare-dest=$PWD/dstlt/. $PWD/srclt/. $PWD/baklt/.
|& tee testlt.log
sending incremental file list
delta-transmission disabled for local transfer or --whole-file
./
a is uptodate
d is uptodate
b
a => b
/tmp> ll -i srclt dstlt baklt
total 8
17305770 -rw-rw-r-- 2 2 Apr 5 15:46 a
17305770 -rw-rw-r-- 2 2 Apr 5 15:46 b
total 8
8474888 -rw-rw-r-- 1 2 Apr 5 15:46 a
8474896 -rw-rw-r-- 1 2 Apr 5 15:46 d
total 16
8866 -rw-rw-r-- 2 2 Apr 5 15:46 a
8866 -rw-rw-r-- 2 2 Apr 5 15:46 b
8867 -rw-rw-r-- 2 2 Apr 5 15:46 c
8867 -rw-rw-r-- 2 2 Apr 5 15:46 d
----
/tmp> rsync -aHvv --compare-dest=dstlt/. srclt/. baklt/. |& tee testlt.log
sending incremental file list
--compare-dest arg does not exist: dstlt/.
delta-transmission disabled for local transfer or --whole-file
./
b
d
a => b
c => d
/tmp> ll -i srclt dstlt baklt
total 16
17305770 -rw-rw-r-- 2 2 Apr 5 15:46 a
17305770 -rw-rw-r-- 2 2 Apr 5 15:46 b
17305782 -rw-rw-r-- 2 2 Apr 5 15:46 c
17305782 -rw-rw-r-- 2 2 Apr 5 15:46 d
total 8
8474888 -rw-rw-r-- 1 2 Apr 5 15:46 a
8474896 -rw-rw-r-- 1 2 Apr 5 15:46 d
total 16
8866 -rw-rw-r-- 2 2 Apr 5 15:46 a
8866 -rw-rw-r-- 2 2 Apr 5 15:46 b
8867 -rw-rw-r-- 2 2 Apr 5 15:46 c
8867 -rw-rw-r-- 2 2 Apr 5 15:46 d
--------------
Post by Hansjoerg Lipp via rsync
rsync -aHv --compare-dest=~/dstlt/. $PWD/srclt/. $PWD/baklt/. |& tee
testlt.log sending incremental file list
--compare-dest arg does not exist: ~/dstlt/.
./
b
d
a => b
c => d
In my usage both the compare-dest & baklt were 'technically'
different file systems (though the compare-dest was a snap
shot).
-----------
/tmp> rsync -aHv --compare-dest=$PWD/dstlt/. $PWD/srclt/. ~/baklt/. |&
tee testlt.log sending incremental file list
./
b
a => b
/tmp> ll -i srclt dstlt ~/baklt
total 8
3677653445 -rw-rw-r-- 2 2 Apr 5 15:46 a
3677653445 -rw-rw-r-- 2 2 Apr 5 15:46 b
total 8
8474888 -rw-rw-r-- 1 2 Apr 5 15:46 a
8474896 -rw-rw-r-- 1 2 Apr 5 15:46 d
total 16
8866 -rw-rw-r-- 2 2 Apr 5 15:46 a
8866 -rw-rw-r-- 2 2 Apr 5 15:46 b
8867 -rw-rw-r-- 2 2 Apr 5 15:46 c
8867 -rw-rw-r-- 2 2 Apr 5 15:46 d
----
Ok... something's screwy here. But no hangs.
Like I said in my 1st response, if some hard links
were missing in the bak directory, I really would be
unlikely to notice, as I didn't even try to check
them (way too many files), but I never got a hang --
and that's the part I'm thinking might be nfs
related, since I've seen several issues with nfs not
working the same way as a local fs.
I usually use smbfs -- in my usage, it's faster and I
usually have fewer compatibility problems.
(faster meaning ~ Reads~162MB/s, W~220MB/s,
though i've seen explorer hit over 400MB/s).
(using a 10Gb link).
But the above testing shows some unexplained behaviors
out of rsync that sure look like a bug.
Good test case!
:-)
--
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. ***@FutureQuest.net (work)
Orlando, Florida ***@sanitarium.net (personal)
Web page: http://www.sanitarium.net/
PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
L. A. Walsh via rsync
2017-04-06 00:50:18 UTC
Permalink
Post by Kevin Korb via rsync
Note that in all --*-dest options if the path is relative it is relative
to the target dir not the "$PWD". I like to always use absolute paths
because of this. But essentially, the command with the instances of
$PWD vs without them the paths aren't the same. If you want all
rsync -aHvv --compare-dest=../dstlt/. srclt/. baklt/.
---
Good point -- though it still doesn't explain the differing
output when dstlt or baklt are on different file systems (one
works, while the other was worse)....
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Loading...