preserving Mac OS X metadata in rsync backups and restores

Post by Moritz Heckscher
Hello all,
I'm new to the list, but have done quite a bit of researching before
regarding the support of Mac OS X specific features (resource forks,
extended attributes, ACLs, file creation & modification date).
By reading the archives, I get the impression that the current
version of rsync 3.0.0pre8 is quite far in this respect. At least it
sounds so, and I thank the developers very much for this! I like
your approach much more than the (very buggy) one originally pursued
by Apple (store metadata in separate ._ file).

Be careful and test, test, test. I tried using pre8 to sync two local
Xserve RAID's(about 2TB of data) and I'm seeing these errors.

rsync: writefd_unbuffered failed to write 4 bytes [sender]: Broken
pipe (32)
[receiver] internal abbrev error!
rsync error: error in rsync protocol data stream (code 12) at
xattrs.c(565) [receiver=3.0.0pre8]
rsync: connection unexpectedly closed (175959 bytes received so far)
[sender]
rsync error: error in rsync protocol data stream (code 12) at
io.c(600) [sender=3.0.0pre8]

I have another Xserve RAID(about 1.3TB) and I don't get those errors
when syncing with pre8. I'm trying to pin down what files/folders are
causing the problem now.

Post by Moritz Heckscher
* Run a Linux server (Ubuntu, I guess, on a ext3 partition) with two
separate internal ATA hard disks formatted in XFS and configured in
software RAID to store the actual backup data. (As I understand, I
should use XFS rather than ext3 because XFS supports extended
attributes large enough to hold also larger converted Mac resource
forks.)
* Back up from different Mac OS X clients (cuurently all on 10.4,
but I might upgrade them to 10.5 later) to the server using rsync
over ssh. This should hopefully preserve (most of) the Mac-specifif
metadata on the server. (Actually I plan to use rsnapshot, but I
believe if I have rsync installed in the newest version and possibly
tell rsnaphot to use the appropriate rsync options, things will be
the same.)
1) What would I have to do to ensure the metadata is also restored
correctly? I assume I will have to use rsync for restoring also, and
if I just copy over data (using, e.g, scp or over an AFP or CIFS or
NFS network mount), I will lose this metadata. Is this correct?

Why not use rsync3 for both backup and restore. Either use ssh (rsync -
azXA --delete /path/to/source server:/path/to/target) or setup an
rsync daemon server. This way you let rsync handle the metadata.

Post by Moritz Heckscher
Another problem I'm thinking about is that rsnapshot should be run
on the server to "pull" the backups over the network. One cannot run
it on the clients and "push" the data to the server over the network
-- which is what I'd prefer because I plan to not leave the server
on all day but rather have it woken up by the (laptop) clients when
needed who'll take care of the scheduling of the backups (using
anachron or launchd etc.) One could, however, run rsnapshot on the
clients to backup onto a locally attached storage device.

You don't need rsnapshot. Use the --link-dist option to create
incremental backups.

Post by Moritz Heckscher
2) If I mount the server as a network drive on the clients using
AFP, SMB/CIFS, NFS, ..., and then backup to this 'locally attached'
drive with rsync (using rsnapshot), will I lose the metadata because
of the transfer via the SMB/... layer?
Thanks a lot for a great program!
-Moritz
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

David Miller.

Moritz Heckscher

2008-01-19 00:37:15 UTC

Post by Moritz Heckscher
Hello all,
I'm new to the list, but have done quite a bit of researching
before regarding the support of Mac OS X specific features
(resource forks, extended attributes, ACLs, file creation &
modification date).
By reading the archives, I get the impression that the current
version of rsync 3.0.0pre8 is quite far in this respect. At least
it sounds so, and I thank the developers very much for this! I
like your approach much more than the (very buggy) one originally
pursued by Apple (store metadata in separate ._ file).

Be careful and test, test, test. I tried using pre8 to sync two
local Xserve RAID's(about 2TB of data) and I'm seeing these errors.
rsync: writefd_unbuffered failed to write 4 bytes [sender]: Broken
pipe (32)
[receiver] internal abbrev error!
rsync error: error in rsync protocol data stream (code 12) at
xattrs.c(565) [receiver=3.0.0pre8]
rsync: connection unexpectedly closed (175959 bytes received so
far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c
(600) [sender=3.0.0pre8]
I have another Xserve RAID(about 1.3TB) and I don't get those
errors when syncing with pre8. I'm trying to pin down what files/
folders are causing the problem now.

Thanks for your feedback. I will certainly test a lot before going
into deployment. (Currently I'm waiting for a hard disk so my machine
is not even physically built yet.) I have found some interesting
resources regarding the metadata problem which you might find useful
for isolating the problem with your machines also:

1) A detailed post about which kinds of metadata exist on Mac OS X
and about how poorly almost all programs handle them:

<http://blog.plasticsfuture.org/2006/03/05/the-state-of-backup-and-
cloning-tools-under-mac-os-x/>

The post is rather old (March 2006, i.e. OS X 10.4.5/10.4.6), however
from my research it seems that basically nothing has changed since
then (a real shame, I say!). (I am fairly certain nothing substatial
has changed in 10.4. I am also not sure if things have been fixed in
10.5 in the cp and rsync programs of OS X or if Apple has invested
all their energy to get things right in their own "Time Machine"
program, at the cost of neglecting the standard tools.)

2) Almost a year later, someone has built a test set containing files
will (almost?) all possible Max OS X metadata:

<http://inik.net/node/150>

One could transfer this small number of files from the set back and
forth and check where the problems are.

3) To help with the checking/comparing, someone else built a little
tool. It not only creates a collection of test files but can also
compare the original and transferred versions afterwards:

<http://www.n8gray.org/blog/2007/04/27/introducing-backup-bouncer/>

I haven't tested these yet, but will once my hardware is ready.

All in all, the metadata situation on Mac OS X is and has been a
total mess. It's almost impossible to generate true backups (true
meaning keeping all metadata intact with the data). Sure, usually you
don't need all metadata and people are happy if they can at least
recover the data fork. But on the other hand there is all this
metadata, used by all sorts of old and new programs, so keeping it
should be possible.

Post by Moritz Heckscher
* Run a Linux server (Ubuntu, I guess, on a ext3 partition) with
two separate internal ATA hard disks formatted in XFS and
configured in software RAID to store the actual backup data. (As I
understand, I should use XFS rather than ext3 because XFS supports
extended attributes large enough to hold also larger converted Mac
resource forks.)
* Back up from different Mac OS X clients (cuurently all on 10.4,
but I might upgrade them to 10.5 later) to the server using rsync
over ssh. This should hopefully preserve (most of) the Mac-
specifif metadata on the server. (Actually I plan to use
rsnapshot, but I believe if I have rsync installed in the newest
version and possibly tell rsnaphot to use the appropriate rsync
options, things will be the same.)
1) What would I have to do to ensure the metadata is also restored
correctly? I assume I will have to use rsync for restoring also,
and if I just copy over data (using, e.g, scp or over an AFP or
CIFS or NFS network mount), I will lose this metadata. Is this
correct?

Why not use rsync3 for both backup and restore. Either use ssh
(rsync -azXA --delete /path/to/source server:/path/to/target) or
setup an rsync daemon server. This way you let rsync handle the
metadata.

Yes, that's what I meant, using rsync in both directions should (!)
keep things intact. I was asking about the other alternatives because
while I will set up the server, it might be necessary for users to
restore their own backups. And I don't think they'd be able to
successfully use rsync on the command line.

So I was thinking of maybe publishing the backup directory on the
server as a read-only share on the network that users could mount on
their client machines. If they then restore files (over AFP,
SMB, ...), will this destroy the metadata that rsync had previously
stored? I assume so. (Yes, I know, I can test myself, and will when
possible, I'm just hoping to get advice from seasoned rsync/network/
Mac gurus here beforehand...)

Post by Moritz Heckscher
Another problem I'm thinking about is that rsnapshot should be run
on the server to "pull" the backups over the network. One cannot
run it on the clients and "push" the data to the server over the
network -- which is what I'd prefer because I plan to not leave
the server on all day but rather have it woken up by the (laptop)
clients when needed who'll take care of the scheduling of the
backups (using anachron or launchd etc.) One could, however, run
rsnapshot on the clients to backup onto a locally attached storage
device.

You don't need rsnapshot. Use the --link-dist option to create
incremental backups.

Thanks for pointing this out. I had looked through the man page of
rsync a few times, but, you know, it's a little complicated in the
beginning to understand all the options...

Anyway, I think rsnapshot would still be a good (better) solution for
me because it handles all the rotating of daily/weekly/monthly
backups. I browsed through the Perl code a few days ago and saw that
it was more than 5000 lines long. I think if several smart people
have worked on this problem for years and produced a script heavily
tested, this should be more reliable than rolling out my own 20 lines
shell script.

My clients are laptop machines which are not always on, so I expect a
lot of interrupted or skipped backups. That's difficult to deal with.

See above. If anyone can confirm I will lose the metadata, I'd be
grateful.

-Moritz

Post by Moritz Heckscher
Thanks a lot for a great program!
-Moritz
--
To unsubscribe or change options: https://lists.samba.org/mailman/
listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-
questions.html

David Miller.
--
To unsubscribe or change options: https://lists.samba.org/mailman/
listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-
questions.html

Moritz Heckscher

2008-01-19 04:12:41 UTC

Post by Moritz Heckscher
3) To help with the checking/comparing, someone else built a little
tool. It not only creates a collection of test files but can also
<http://www.n8gray.org/blog/2007/04/27/introducing-backup-bouncer/>

I have downloaded the tool and compiled it on my Mac with 10.4 to do
some testing. In this case, that's transfers using rsync 3.0.0pre8
from one HFS+ disk image to another HFS+ image (on a computer with an
HFS+ disk as well). Later I'd be using cross-platform cross-
filesystem transfers, so the results below should present something
like the "upper limit" of what is currently possible.

Using the following options for rsync:

--archive --hard-links --acls --xattrs --executability --numeric-ids

I get the following output from backup-bouncer:

------------------ rsync3.0.0pre8 ------------------
This copier exited with error code 23
This copier produced log output in:
/Volumes/Dst/21-rsync3.0.0pre8/log
Verifying: basic-permissions ... ok
Verifying: timestamps ...
Sub-test: modification time ... ok
ok
Verifying: symlinks ... ok
Verifying: symlink-ownership ... ok
Verifying: hardlinks ... ok
Verifying: resource-forks ... ok
Verifying: finder-flags ... ok
Verifying: finder-locks ... FAIL
Verifying: creation-date ... FAIL
Verifying: bsd-flags ... FAIL
Verifying: extended-attrs ...
Sub-test: on files ... ok
Sub-test: on directories ... ok
Sub-test: on symlinks ... ok
ok
Verifying: access-control-lists ...
Sub-test: on files ... ok
Sub-test: on dirs ... ok
ok
Verifying: fifo ... FAIL
Verifying: devices ... FAIL
Verifying: combo-tests ...
Sub-test: xattrs + rsrc forks ... ok
Sub-test: lots of metadata ... ok
ok

So that looks not so bad actually. The main problem for me are the
mangled creation dates. However, not even Apple's own cp with options
-pR (-p to "preserve in the copy as many of the modification time,
access time, file flags, file mode, user ID, and group ID as allowed
by permissions") passes this test:

------------------ cp-apple ------------------
Verifying: basic-permissions ... ok
Verifying: timestamps ...
Sub-test: modification time ... ok
ok
Verifying: symlinks ... ok
Verifying: symlink-ownership ... FAIL
Verifying: hardlinks ... FAIL
Verifying: resource-forks ... ok
Verifying: finder-flags ... ok
Verifying: finder-locks ... ok
Verifying: creation-date ... FAIL
Verifying: bsd-flags ... ok
Verifying: extended-attrs ...
Sub-test: on files ... ok
Sub-test: on directories ... ok
Sub-test: on symlinks ... FAIL
FAIL
Verifying: access-control-lists ...
Sub-test: on files ... ok
Sub-test: on dirs ... ok
ok
Verifying: fifo ... ok
Verifying: devices ... ok
Verifying: combo-tests ...
Sub-test: xattrs + rsrc forks ... ok
Sub-test: lots of metadata ... ok
ok

For comparison, this is Apple's 'fixed' rsync version 2.6.3
protocol version 28:

------------------ rsync-apple ------------------
This copier exited with error code 24
This copier produced log output in:
/Volumes/Dst/10-rsync-apple/log
Verifying: basic-permissions ... ok
Verifying: timestamps ...
Sub-test: modification time ... ok
ok
Verifying: symlinks ... ok
Verifying: symlink-ownership ... ok
Verifying: hardlinks ... ok
Verifying: resource-forks ... FAIL
Verifying: finder-flags ... FAIL
Verifying: finder-locks ... FAIL
Verifying: creation-date ... FAIL
Verifying: bsd-flags ... FAIL
Verifying: extended-attrs ...
Sub-test: on files ... FAIL
Sub-test: on directories ... FAIL
Sub-test: on symlinks ... FAIL
FAIL
Verifying: access-control-lists ...
Sub-test: on files ... FAIL
Sub-test: on dirs ... FAIL
FAIL
Verifying: fifo ... ok
Verifying: devices ... ok
Verifying: combo-tests ...
Sub-test: xattrs + rsrc forks ... FAIL
Sub-test: lots of metadata ... FAIL
FAIL

Adding --fake-super to the options of rsync doesn't help:

------------------ rsync3.0.0pre8 ------------------
This copier exited with error code 23
This copier produced log output in:
/Volumes/Dst/21-rsync3.0.0pre8/log
Verifying: basic-permissions ... FAIL
Verifying: timestamps ...
Sub-test: modification time ... ok
ok
Verifying: symlinks ... ok
Verifying: symlink-ownership ... FAIL
Verifying: hardlinks ... ok
Verifying: resource-forks ... ok
Verifying: finder-flags ... ok
Verifying: finder-locks ... FAIL
Verifying: creation-date ... FAIL
Verifying: bsd-flags ... FAIL
Verifying: extended-attrs ...
Sub-test: on files ... FAIL
Sub-test: on directories ... FAIL
Sub-test: on symlinks ... FAIL
FAIL
Verifying: access-control-lists ...
Sub-test: on files ... FAIL
Sub-test: on dirs ... FAIL
FAIL
Verifying: fifo ... FAIL
Verifying: devices ... FAIL
Verifying: combo-tests ...
Sub-test: xattrs + rsrc forks ... FAIL
cmp: ./many-metadata: Permission denied
Sub-test: lots of metadata ... FAIL
FAIL

-Moritz

Matt McCutchen

2008-01-19 04:31:11 UTC

Post by Moritz Heckscher
--archive --hard-links --acls --xattrs --executability --numeric-ids
I get the following output from backup-bouncer: [...]
Verifying: bsd-flags ... FAIL

BSD flags are handled by flags.diff in rsync-patches-3.0.0pre8.tar.gz .

Post by Moritz Heckscher
Verifying: finder-locks ... FAIL

A request for enhancement was recently entered for finder locks:

https://bugzilla.samba.org/show_bug.cgi?id=5188

But Wayne says he thinks they are handled by flags.diff too.

Post by Moritz Heckscher
Verifying: creation-date ... FAIL

Creation dates are handled by osx-create-time.diff .

Post by Moritz Heckscher
Verifying: fifo ... FAIL
Verifying: devices ... FAIL

These are supposed to work with --archive, at least if rsync runs as
root. In any event, you probably don't care about these if you're just
backing up user data.
Naturally, because that option makes rsync wrap the source metadata in
an rsync-specific extended attribute that it sets on the destination
file, whereas the checker tool is expecting the real destination
metadata to be set.

Matt

Moritz Heckscher

2008-01-19 04:38:33 UTC

Post by Moritz Heckscher
--archive --hard-links --acls --xattrs --executability --numeric-ids
I get the following output from backup-bouncer: [...]
Verifying: bsd-flags ... FAIL

BSD flags are handled by flags.diff in rsync-
patches-3.0.0pre8.tar.gz .

Post by Moritz Heckscher
Verifying: finder-locks ... FAIL

https://bugzilla.samba.org/show_bug.cgi?id=5188
But Wayne says he thinks they are handled by flags.diff too.

Post by Moritz Heckscher
Verifying: creation-date ... FAIL

Creation dates are handled by osx-create-time.diff .

Thanks for mentioning those diffs. I'll try tomorrow and report here.

Post by Moritz Heckscher
Verifying: fifo ... FAIL
Verifying: devices ... FAIL

These are supposed to work with --archive, at least if rsync runs as
root. In any event, you probably don't care about these if you're just
backing up user data.

Yes, I was expecting this to work. I did run as root (using sudo) and
also tried --device and --specials, but these are already included in
--archive. Hm, I'll run it again tomorrow, maybe I've messed up
somehow (it's 05:30 in the morning...).

Post by Matt McCutchen
Naturally, because that option makes rsync wrap the source metadata in
an rsync-specific extended attribute that it sets on the destination
file, whereas the checker tool is expecting the real destination
metadata to be set.

Yes, that absolutely makes sense. I shouldn't have included the
output here. Obviously one would need a second "recover" step using
rsync before checking with the tool.

Thanks for your fast help, I'm learning a lot!
-Moritz

Robert DuToit

2008-01-19 10:23:06 UTC

Post by Moritz Heckscher
--archive --hard-links --acls --xattrs --executability --numeric-ids
I get the following output from backup-bouncer: [...]
Verifying: bsd-flags ... FAIL

BSD flags are handled by flags.diff in rsync-
patches-3.0.0pre8.tar.gz .

Post by Moritz Heckscher
Verifying: finder-locks ... FAIL

https://bugzilla.samba.org/show_bug.cgi?id=5188
But Wayne says he thinks they are handled by flags.diff too.

Post by Moritz Heckscher
Verifying: creation-date ... FAIL

Creation dates are handled by osx-create-time.diff .

Thanks for mentioning those diffs. I'll try tomorrow and report here.

I did some extensive testing with the osx-create-time.diff patch on
OS10.4.11 and 10.5.1 and it works pretty well except it takes twice as
long for a 10+GB simple backup to an external drive (once it just
slowed down to a dead stop and I gave up.)I tried Mike Bombich's
version of rsync that he made for his new excellent Carbon Copy Cloner
(just the rsync tool itself) and it saved the creation dates with
little loss in efficiency. Not sure how he did that. I made some
wrapper apps for rsync and get a lot of requests for creation dates
saved from people and, though personally it isn't an issue for me, it
would be nice for OSX folk to have that work well. Besides that I am
grateful for rsync 3 and all the work you have all done. I also made a
wrapper for Nat Gray's backup-bouncer and my tests verify Moritz's
results too. Rsync 3 is a huge improvement over the apple version.

Also I use the current pre8 version for incremental backups of my 20GB
Home folder every night using --link-dest=DIR. I set it for a
revolving cycle of 30 backups and they work perfectly every time
creating exact snapshots that take a few minutes at most, after the
initial large copy. This is not the case with the apple supplied rsync
that re-copies certain files over and over without creating hard
links. I made a wrapper for this if any one is interested (not tested
much on networks.) So thanks!

Rob D

PS Time machine has not worked well for me, grinding to halt when
encountering certain troublesome files and havn't been able to work
out the problem so just use rsync 3 instead!

Post by Moritz Heckscher
Verifying: fifo ... FAIL
Verifying: devices ... FAIL

These are supposed to work with --archive, at least if rsync runs as
root. In any event, you probably don't care about these if you're just
backing up user data.

Yes, I was expecting this to work. I did run as root (using sudo)
and also tried --device and --specials, but these are already
included in --archive. Hm, I'll run it again tomorrow, maybe I've
messed up somehow (it's 05:30 in the morning...).

Yes, that absolutely makes sense. I shouldn't have included the
output here. Obviously one would need a second "recover" step using
rsync before checking with the tool.

Thanks for your fast help, I'm learning a lot!
-Moritz
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Mike Bombich

2008-01-20 21:17:30 UTC

With rsync 3 pre8 and Mac OS 10.5.1:

bash-3.2# patch -p1 <patches/osx-create-time.diff
bash-3.2# patch -p1 <patches/flags.diff
bash-3.2# ./prepare-source
bash-3.2# patch -p1 <patches/backup-dir-dels.diff
bash-3.2# ./configure
bash-3.2# make

bash-3.2# ~/rsync-3.0.0pre8/rsync -aHAX --fileflags /Volumes/Source/ /
Volumes/Target/

bash-3.2# bbouncer verify -d /Volumes/Source /Volumes/Target
Verifying: basic-permissions ... ok
Verifying: timestamps ...
Sub-test: modification time ... ok
ok
Verifying: symlinks ... ok
Verifying: symlink-ownership ... ok
Verifying: symlink-permissions ... ok
Verifying: hardlinks ... ok
Verifying: resource-forks ... ok
Verifying: finder-flags ... ok
Verifying: finder-locks ... ok
Verifying: creation-date ... ok
Verifying: bsd-flags ... ok
Verifying: extended-attrs ...
Sub-test: on files ... ok
Sub-test: on directories ... ok
Sub-test: on symlinks ... ok
ok
Verifying: access-control-lists ...
Sub-test: on files ... ok
Sub-test: on dirs ... ok
ok
Verifying: fifo ... ok
Verifying: devices ... ok
Verifying: combo-tests ...
Sub-test: xattrs + rsrc forks ... ok
Sub-test: lots of metadata ... ok
ok

As far as performance is concerned, I'm surprised how fast rsync 3pre8
is for no-change-incrementals. Incremental recursion alone doesn't
really explain it, I'm still trying to see if something is fishy or if
it really can be this good.

Mike

Post by Moritz Heckscher
--archive --hard-links --acls --xattrs --executability --numeric-ids
I get the following output from backup-bouncer: [...]
Verifying: bsd-flags ... FAIL

BSD flags are handled by flags.diff in rsync-
patches-3.0.0pre8.tar.gz .

Post by Moritz Heckscher
Verifying: finder-locks ... FAIL

https://bugzilla.samba.org/show_bug.cgi?id=5188
But Wayne says he thinks they are handled by flags.diff too.

Post by Moritz Heckscher
Verifying: creation-date ... FAIL

Creation dates are handled by osx-create-time.diff .

Thanks for mentioning those diffs. I'll try tomorrow and report here.

Post by Moritz Heckscher
Verifying: fifo ... FAIL
Verifying: devices ... FAIL

These are supposed to work with --archive, at least if rsync runs as
root. In any event, you probably don't care about these if you're just
backing up user data.

Yes, I was expecting this to work. I did run as root (using sudo)
and also tried --device and --specials, but these are already
included in --archive. Hm, I'll run it again tomorrow, maybe I've
messed up somehow (it's 05:30 in the morning...).

Yes, that absolutely makes sense. I shouldn't have included the
output here. Obviously one would need a second "recover" step using
rsync before checking with the tool.

Robert DuToit

2008-01-21 11:58:03 UTC

Hi All,
I ran the bouncer tests too and came up with same results except the
"fifo" and "devices" failed. I ran it from command line as well as
from "do shell script" command with same results. I used the same
patches. Nevertheless, I am thrilled with the results here for OSX and
can't thank you all enough for putting rsync 3 together. I did some
incremental speed tests with my 16GB Home folder from my old powerbook
G4:

from do shell script; destinations all to external firewire drive on
OS 10.5.1

with patches:
initial backup 54 minutes
no-change-incremental backups 7 - 14 minutes

without patches:
initial backup 52 minutes
no-change-incremental backups 5 - 7 minutes

the patches don't make a huge difference for the initial backups but
no-change-incrementals do seem to go on twice as long. Not sure why
really- The output showed nothing unusual. Things would go faster
from the command line since I am running rsync 3 through my wrapper
application with the "do shell script" command. . I changed one letter
on some deeply nested files and they all showed up fine in the
incremental snapshots.

By the way, after the initial 16+ GB copy, the incrementals took up an
incredible .05 +- Gb of space. The apple supplieed rsync on Tiger and
Leo both recopy a lot of data each time so the incrementals never
worked well.

Thanks Again, Rob D

Post by Mike Bombich
bash-3.2# patch -p1 <patches/osx-create-time.diff
bash-3.2# patch -p1 <patches/flags.diff
bash-3.2# ./prepare-source
bash-3.2# patch -p1 <patches/backup-dir-dels.diff
bash-3.2# ./configure
bash-3.2# make
bash-3.2# ~/rsync-3.0.0pre8/rsync -aHAX --fileflags /Volumes/
Source/ /Volumes/Target/
bash-3.2# bbouncer verify -d /Volumes/Source /Volumes/Target
Verifying: basic-permissions ... ok
Verifying: timestamps ...
Sub-test: modification time ... ok
ok
Verifying: symlinks ... ok
Verifying: symlink-ownership ... ok
Verifying: symlink-permissions ... ok
Verifying: hardlinks ... ok
Verifying: resource-forks ... ok
Verifying: finder-flags ... ok
Verifying: finder-locks ... ok
Verifying: creation-date ... ok
Verifying: bsd-flags ... ok
Verifying: extended-attrs ...
Sub-test: on files ... ok
Sub-test: on directories ... ok
Sub-test: on symlinks ... ok
ok
Verifying: access-control-lists ...
Sub-test: on files ... ok
Sub-test: on dirs ... ok
ok
Verifying: fifo ... ok
Verifying: devices ... ok
Verifying: combo-tests ...
Sub-test: xattrs + rsrc forks ... ok
Sub-test: lots of metadata ... ok
ok
As far as performance is concerned, I'm surprised how fast rsync
3pre8 is for no-change-incrementals. Incremental recursion alone
doesn't really explain it, I'm still trying to see if something is
fishy or if it really can be this good.
Mike

Post by Moritz Heckscher
--archive --hard-links --acls --xattrs --executability --numeric-
ids
I get the following output from backup-bouncer: [...]
Verifying: bsd-flags ... FAIL

BSD flags are handled by flags.diff in rsync-
patches-3.0.0pre8.tar.gz .

Post by Moritz Heckscher
Verifying: finder-locks ... FAIL

https://bugzilla.samba.org/show_bug.cgi?id=5188
But Wayne says he thinks they are handled by flags.diff too.

Post by Moritz Heckscher
Verifying: creation-date ... FAIL

Creation dates are handled by osx-create-time.diff .

Thanks for mentioning those diffs. I'll try tomorrow and report here.

Post by Moritz Heckscher
Verifying: fifo ... FAIL
Verifying: devices ... FAIL

These are supposed to work with --archive, at least if rsync runs as
root. In any event, you probably don't care about these if you're just
backing up user data.

Yes, I was expecting this to work. I did run as root (using sudo)
and also tried --device and --specials, but these are already
included in --archive. Hm, I'll run it again tomorrow, maybe I've
messed up somehow (it's 05:30 in the morning...).

Post by Matt McCutchen
Naturally, because that option makes rsync wrap the source
metadata in
an rsync-specific extended attribute that it sets on the destination
file, whereas the checker tool is expecting the real destination
metadata to be set.

Yes, that absolutely makes sense. I shouldn't have included the
output here. Obviously one would need a second "recover" step using
rsync before checking with the tool.

--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Victor Shoup

2008-01-19 12:12:28 UTC

The backup-bouncer tool looks very helpful.
I tried testing my own backup solution (see http://shoup.net/xattr )
with backup-bouncer.
It did fine -- it preserves everything on cross-platform backups,
except for those things
it was not designed to: ACLs (I don't know much about these) and BSD
flags other than uchg, uappnd
(I modified the backup-bouncer test suite to restrict to just these
BSD flags, to verify).

-- Victor Shoup

Post by Moritz Heckscher
3) To help with the checking/comparing, someone else built a
little tool. It not only creates a collection of test files but
<http://www.n8gray.org/blog/2007/04/27/introducing-backup-bouncer/>

I have downloaded the tool and compiled it on my Mac with 10.4 to
do some testing. In this case, that's transfers using rsync
3.0.0pre8 from one HFS+ disk image to another HFS+ image (on a
computer with an HFS+ disk as well). Later I'd be using cross-
platform cross-filesystem transfers, so the results below should
present something like the "upper limit" of what is currently
possible.
--archive --hard-links --acls --xattrs --executability --numeric-ids
------------------ rsync3.0.0pre8 ------------------
This copier exited with error code 23
/Volumes/Dst/21-rsync3.0.0pre8/log
Verifying: basic-permissions ... ok
Verifying: timestamps ...
Sub-test: modification time ... ok
ok
Verifying: symlinks ... ok
Verifying: symlink-ownership ... ok
Verifying: hardlinks ... ok
Verifying: resource-forks ... ok
Verifying: finder-flags ... ok
Verifying: finder-locks ... FAIL
Verifying: creation-date ... FAIL
Verifying: bsd-flags ... FAIL
Verifying: extended-attrs ...
Sub-test: on files ... ok
Sub-test: on directories ... ok
Sub-test: on symlinks ... ok
ok
Verifying: access-control-lists ...
Sub-test: on files ... ok
Sub-test: on dirs ... ok
ok
Verifying: fifo ... FAIL
Verifying: devices ... FAIL
Verifying: combo-tests ...
Sub-test: xattrs + rsrc forks ... ok
Sub-test: lots of metadata ... ok
ok
So that looks not so bad actually. The main problem for me are the
mangled creation dates. However, not even Apple's own cp with
options -pR (-p to "preserve in the copy as many of the
modification time, access time, file flags, file mode, user ID, and
------------------ cp-apple ------------------
Verifying: basic-permissions ... ok
Verifying: timestamps ...
Sub-test: modification time ... ok
ok
Verifying: symlinks ... ok
Verifying: symlink-ownership ... FAIL
Verifying: hardlinks ... FAIL
Verifying: resource-forks ... ok
Verifying: finder-flags ... ok
Verifying: finder-locks ... ok
Verifying: creation-date ... FAIL
Verifying: bsd-flags ... ok
Verifying: extended-attrs ...
Sub-test: on files ... ok
Sub-test: on directories ... ok
Sub-test: on symlinks ... FAIL
FAIL
Verifying: access-control-lists ...
Sub-test: on files ... ok
Sub-test: on dirs ... ok
ok
Verifying: fifo ... ok
Verifying: devices ... ok
Verifying: combo-tests ...
Sub-test: xattrs + rsrc forks ... ok
Sub-test: lots of metadata ... ok
ok
For comparison, this is Apple's 'fixed' rsync version 2.6.3
------------------ rsync-apple ------------------
This copier exited with error code 24
/Volumes/Dst/10-rsync-apple/log
Verifying: basic-permissions ... ok
Verifying: timestamps ...
Sub-test: modification time ... ok
ok
Verifying: symlinks ... ok
Verifying: symlink-ownership ... ok
Verifying: hardlinks ... ok
Verifying: resource-forks ... FAIL
Verifying: finder-flags ... FAIL
Verifying: finder-locks ... FAIL
Verifying: creation-date ... FAIL
Verifying: bsd-flags ... FAIL
Verifying: extended-attrs ...
Sub-test: on files ... FAIL
Sub-test: on directories ... FAIL
Sub-test: on symlinks ... FAIL
FAIL
Verifying: access-control-lists ...
Sub-test: on files ... FAIL
Sub-test: on dirs ... FAIL
FAIL
Verifying: fifo ... ok
Verifying: devices ... ok
Verifying: combo-tests ...
Sub-test: xattrs + rsrc forks ... FAIL
Sub-test: lots of metadata ... FAIL
FAIL
------------------ rsync3.0.0pre8 ------------------
This copier exited with error code 23
/Volumes/Dst/21-rsync3.0.0pre8/log
Verifying: basic-permissions ... FAIL
Verifying: timestamps ...
Sub-test: modification time ... ok
ok
Verifying: symlinks ... ok
Verifying: symlink-ownership ... FAIL
Verifying: hardlinks ... ok
Verifying: resource-forks ... ok
Verifying: finder-flags ... ok
Verifying: finder-locks ... FAIL
Verifying: creation-date ... FAIL
Verifying: bsd-flags ... FAIL
Verifying: extended-attrs ...
Sub-test: on files ... FAIL
Sub-test: on directories ... FAIL
Sub-test: on symlinks ... FAIL
FAIL
Verifying: access-control-lists ...
Sub-test: on files ... FAIL
Sub-test: on dirs ... FAIL
FAIL
Verifying: fifo ... FAIL
Verifying: devices ... FAIL
Verifying: combo-tests ...
Sub-test: xattrs + rsrc forks ... FAIL
cmp: ./many-metadata: Permission denied
Sub-test: lots of metadata ... FAIL
FAIL
-Moritz
--
To unsubscribe or change options: https://lists.samba.org/mailman/
listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-
questions.html

Moritz Heckscher

2008-01-19 04:13:12 UTC

Matt McCutchen

2008-01-19 02:11:05 UTC

Post by Moritz Heckscher
Another problem I'm thinking about is that rsnapshot should be run on
the server to "pull" the backups over the network. One cannot run it
on the clients and "push" the data to the server over the network --
which is what I'd prefer because I plan to not leave the server on
all day but rather have it woken up by the (laptop) clients when
needed who'll take care of the scheduling of the backups (using
anachron or launchd etc.) One could, however, run rsnapshot on the
clients to backup onto a locally attached storage device.

I would like to dispel once and for all the myth that rsnapshot cannot
be used with a push. While it is /designed/ to pull backups, with a few
extra steps one can push a backup to an rsnapshot installation on
another machine. I outlined the setup here:

http://lists.samba.org/archive/rsync/2007-December/019470.html

How are the clients going to wake the server? Does the server wake up
on any incoming network connection (I hope)?

I would strongly discourage you from pushing the backup via a network
drive. In addition to potential metadata mangling, you lose the benefit
of rsync's delta-transfer algorithm.

Matt

Moritz Heckscher

2008-01-19 03:39:59 UTC

Post by Moritz Heckscher
Another problem I'm thinking about is that rsnapshot should be run on
the server to "pull" the backups over the network. One cannot run it
on the clients and "push" the data to the server over the network --
which is what I'd prefer because I plan to not leave the server on
all day but rather have it woken up by the (laptop) clients when
needed who'll take care of the scheduling of the backups (using
anachron or launchd etc.) One could, however, run rsnapshot on the
clients to backup onto a locally attached storage device.

Good news indeed (for me at least)! Having read this FAQ on the
rsnapshot page in an early stage, without having much background info
about rsync and rsnapshot myself, I thought it was best to abandon
the idea of pushed backups with rsnapshot altogether:

Q: Can I set the snapshot_root to a remote SSH path? I want to push
my backups to a remote server, rather than pull them from a remote
server.

A: Currently this is not possible. This would be a nice feature, but
several questions remain unanswered:
* How can the integrity of the snapshot root be guaranteed if one or
more remote servers have write access to it?
* When snapshots are rotated, which of the potentially several remote
servers connecting is responsible for performing this task?
* One possibility is to have a "staging" area for files to be
transferred to, then have rsnapshot sync from this staging area into
the snapshot root. Can this be accomplished without taking up double
the disk space?

Your suggestion does indeed take care of the "doubled disk space"
issue nicely. It's only really possible with your very recent patch
regarding the --no-tweak-hlinked option, so maybe now this FAQ entry
could be update with the above link to your description! (By the way,
would this always remain a patch or eventually go into the normal
source?)

Regarding the other two points: If I have several clients, I guess I
should have several independent deamons, each with "max connections"
set to 1, and several independent "incoming" and "snapshots"
directories, to avoid conflicts:

/backup/
rsnapshot.laptop0.conf
kick-rsnapshot-laptop0
rsnapshot.laptop1.conf
kick-rsnapshot-laptop1
./incoming/
./laptop0/
./laptop1/
./snapshots/
./laptop0/
./laptop1/

Could there be any problems if laptop0 and laptop1 are both pushing
at the same time? Configurations are independent, and in that case
there should simply be two instances of rsnapshot running, right?

Post by Matt McCutchen
How are the clients going to wake the server? Does the server wake up
on any incoming network connection (I hope)?

Yes, that's the plan. I intent to have the clients send a wake-on-lan
magic packet to wake the server, wait a little and then try to push
the backup.

By they way, for the client to see if the backup operation succeeded,
this could be easily tested by looking at the return code from the
communication with the daemon, right?

I would strongly discourage you from pushing the backup via a network
drive. In addition to potential metadata mangling, you lose the benefit
of rsync's delta-transfer algorithm.

Yes, losing the metadata this way is likely to happen. I'll test it
once my hardware is ready, just to see what would happen.

Just out of curiosity: Why would rsync not use the delta-transfer in
that case? Is it because this would be the "local disk -> local disk"
transfer case for which a fast connection is assumed, so no resources
are wasted on trying to find differences?

Well, thank you so very much. That post you've referred to was really
one of the most helpful texts for me! Thank you also for the nice
patch and all the hard work. I very much appreciate it!

-Moritz

Moritz Heckscher

2008-01-19 04:32:01 UTC

Post by Moritz Heckscher
Your suggestion does indeed take care of the "doubled disk space"
issue nicely. It's only really possible with your very recent patch
regarding the --no-tweak-hlinked option, so maybe now this FAQ
entry could be update with the above link to your description! (By
the way, would this always remain a patch or eventually go into the
normal source?)

Sorry, I just see you've answered this before: <http://
lists.samba.org/archive/rsync/2007-December/019471.html>
Well, if it helps to convice, I'll vote for inclusion into the
source! :-)

-Moritz

Matt McCutchen

2008-01-19 04:55:10 UTC

Post by Moritz Heckscher
Good news indeed (for me at least)! Having read this FAQ on the
rsnapshot page in an early stage, without having much background info
about rsync and rsnapshot myself, I thought it was best to abandon
[...]
Your suggestion does indeed take care of the "doubled disk space"
issue nicely. It's only really possible with your very recent patch
regarding the --no-tweak-hlinked option, so maybe now this FAQ entry
could be update with the above link to your description!

Getting in touch with the rsnapshot people to get the FAQ updated is on
my long to-do list.

Post by Moritz Heckscher
Regarding the other two points: If I have several clients, I guess I
should have several independent deamons, each with "max connections"
set to 1, and several independent "incoming" and "snapshots"
directories, to avoid conflicts: [...]

Yes. Specifically:

- You definitely need a separate "incoming" module (including the lock
file used to enforce "max connections = 1"), rsnapshot configuration
file, and "snapshots" dir for each client.

- It is your choice whether to put all the modules in the same
rsyncd.conf (perhaps named "incoming.laptop0", etc.) or each in its own
rsyncd.laptop*.conf . But note: if you plan to secure the setup by
having each client log in with a different ssh authorized key with a
forced command, you want the latter so you can limit each client to its
own module.

- I wouldn't make lots of copies of "kick-rsnapshot" if I could avoid it
by having it check which daemon/module called it.

In any event, I encourage you to write scripts/templates to
automatically generate the necessary configuration files so it's easy to
add or remove clients over time. If you want to be really clever, you
can skip saving the files to disk and instead generate them on the fly
and pipe them to rsync and rsnapshot each time a client connects; if you
do this, put the magic in an ssh forced command so the client doesn't
have to specify a horrendous --rsync-path .

BTW, I just verified that the "max connections" limit covers daemon
processes that are in the post-xfer exec stage, which is important: we
don't want a client both pushing and rotating at the same time.

Post by Moritz Heckscher
Could there be any problems if laptop0 and laptop1 are both pushing
at the same time? Configurations are independent, and in that case
there should simply be two instances of rsnapshot running, right?

I don't foresee any problems with different clients pushing at the same
time.

Post by Moritz Heckscher
By they way, for the client to see if the backup operation succeeded,
this could be easily tested by looking at the return code from the
communication with the daemon, right?

The return code will tell you whether the push to "incoming" succeeded.
It will not reflect any failure in the rsnapshot rotation. This is
probably fine from the client's perspective, as it can't really do
anything to handle such a failure. Still, you might want to ensure that
you find out about any problems by adding code to "kick-rsnapshot" to
send you email if the rotation fails.