Xattr on FreeBSD ... Good Luck ?

[Ensel Sharon]

Thanks for the help on the missing '-p1' for the acl and xattr
patches. Things are configuring and making properly now.

However, I couldn't help noticing this during the configure:


checking whether to support ACLs... ACLs requested -- running tests
checking for acl_get_file in -lacl... no
checking for ACL support... yes
checking ACL test results... Using posix ACLs
checking for acl_get_perm_np... yes
checking attr/xattr.h usability... no
checking attr/xattr.h presence... no
checking for attr/xattr.h... no
checking whether to support extended attributes... Xattrs requested but
not linux. Good luck


Note the last line ... if my goal is to have OSX clients rsyncing their
extended attributes files on to my FreeBSD 6.1 UFS2 filesystem ... is that
a reasonable thing to be attempting ?

[Ensel Sharon]

One last detail...

Post by Ensel Sharon
checking whether to support ACLs... ACLs requested -- running tests
checking for acl_get_file in -lacl... no
checking for ACL support... yes
checking ACL test results... Using posix ACLs
checking for acl_get_perm_np... yes
checking attr/xattr.h usability... no
checking attr/xattr.h presence... no
checking for attr/xattr.h... no
checking whether to support extended attributes... Xattrs requested but
not linux. Good luck
Note the last line ... if my goal is to have OSX clients rsyncing their
extended attributes files on to my FreeBSD 6.1 UFS2 filesystem ... is that
a reasonable thing to be attempting ?

During the 'make' I also saw this notice:


gcc -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -I./popt -c acls.c -o acls.o
acls.c: In function `change_sacl_perms':
acls.c:735: warning: large integer implicitly truncated to unsigned type
acls.c: In function `set_acl':
acls.c:1163: warning: comparison is always false due to limited range of
data type
gcc -I. -I. -g -O2 -DHAVE_CONFIG_H -Wall -W -I./popt -c xattr.c -o
xattr.o


Comments ?

[Wayne Davison]

Post by Ensel Sharon
acls.c:735: warning: large integer implicitly truncated to unsigned type
acls.c:1163: warning: comparison is always false due to limited range of
data type

These are fixed in the latest version of the patch. With 2.6.9 being
released in the very near future (maybe even today), you may want to use
the 2.6.9pre3 release or the latest "nightly" tar file instead of 2.6.8.
The xattrs code has been ported to work with OS X, so it shouldn't be
hard to add the code necessary to get it to work with FreeBSD.

..wayne..

[Ensel Sharon]

Post by Wayne Davison

Post by Ensel Sharon
acls.c:735: warning: large integer implicitly truncated to unsigned type
acls.c:1163: warning: comparison is always false due to limited range of
data type

These are fixed in the latest version of the patch. With 2.6.9 being
released in the very near future (maybe even today), you may want to use
the 2.6.9pre3 release or the latest "nightly" tar file instead of 2.6.8.
The xattrs code has been ported to work with OS X, so it shouldn't be
hard to add the code necessary to get it to work with FreeBSD.

Thanks for your help.

Ok, so 2.6.9 will solve the problems I saw during the 'make', that you
quote above.

But you are saying that even with 2.6.9, there is still work to be done to
get xattrs safely across from OSX -> FreeBSD ?

I do not need to go in the other direction - I have no xattr on FreeBSD
that I need to send _anywhere_ - I just need to get xattr from OSX source
to freebsd destination.

Thanks again.

[Wesley W. Terpstra]

Post by Ensel Sharon
checking whether to support extended attributes... Xattrs requested but
not linux. Good luck

[Wesley W. Terpstra]

Give me a half hour! :-)

Drat! 10 minutes late. :-P This patch modifies also the
fake_super.patch, because I removed the unused (and unportable) flags
operand.
Changes:
remove flags from setxattr
add wrappers I hope work on FreeBSD (can't test this myself)
remove the 0S% prefix for MacOS (it seems the macos extended
attributes should be considered simply the user namespace)
renamed the special fake_super extended attribute to match the
naming recommendations from the freedesktop

If you could test this, Ensel, I'd be grateful. Try a local copy with
-X first. To build it:
cvs -d:pserver:***@pserver.samba.org:/cvsroot co rsync
cd rsync
patch -p1 < patches/acls.patch
patch -p1 < patches/xattrs.patch
patch -p1 < patches/fake_super.patch
patch -p1 < ~/freebsd-xattr.patch
./prepare-sources
./configure --enable-xattr-support
make

Just so you know: there's almost certainly a bug / typo /
misunderstanding somewhere in there.

[Wesley W. Terpstra]

Post by Wesley W. Terpstra
Just so you know: there's almost certainly a bug / typo /
misunderstanding somewhere in there.

Urp! Found a bug already. Sorry. It will segfault on FreeBSD without
this small little fix.
< memmove(&list[off], &list[off+keylen], keylen);
---

Post by Wesley W. Terpstra
memmove(&list[off], &list[off+1], keylen);

This patch is better.

[Wayne Davison]

Post by Wesley W. Terpstra
Urp! Found a bug already. Sorry. It will segfault on FreeBSD without
this small little fix.

One more fix was needed: the new sys_llistxattr() function needs to
return "len" when it is larger than "size", or longer lists of keys will
not get processed (i.e. we use a size of 0 to ask for the size of the
data).

I like your change that gets rid of the ".0S%" part of the Mac OS keys.

What is your reason for wanting to get rid of the '%' from the fake
stat attribute? Do some implementations not allow a % in the key?

..wayne..

[Wesley W. Terpstra]

Post by Wayne Davison

Post by Wesley W. Terpstra
Urp! Found a bug already. Sorry. It will segfault on FreeBSD without
this small little fix.

One more fix was needed: the new sys_llistxattr() function needs to
return "len" when it is larger than "size", or longer lists of keys will
not get processed (i.e. we use a size of 0 to ask for the size of the
data).

Ahh right. I added that to prevent possible buffer overrun when
parsing the string. Your way dodges the problem as well, so all is good.

Did you try it on FreeBSD? I don't have a machine.

Why did you move the loop increment in sys_llistxattr to inside the
loop? It's valid the way I had it, and expressed the concept of
iteration cleanly. The optimizer also detects uninitialized values in
this case as well, so it's not a maintenance problem. Also, my idiom
works with continue.
Not important, I know. :-)

I note that both you and I missed:
+ AC_MSG_RESULT(Xattrs requested but not Linux or OS X.
Good luck...)
maybe
+ AC_MSG_RESULT(Xattrs requested, but not implemented on
this platform. Good luck...)

Post by Wayne Davison
I like your change that gets rid of the ".0S%" part of the Mac OS keys.
What is your reason for wanting to get rid of the '%' from the fake
stat attribute? Do some implementations not allow a % in the key?

No, but others are using 'reverse-DNS naming style' and this seems to
be standard-ish. That means a '.' and not a '%'.
Did you check the second link in my original email?: http://
www.freedesktop.org/wiki/CommonExtendedAttributes

Here's a general question: should linux really be sync'ing system and
root namespaces?
I don't think even root has permission to write arbitrary stuff in
system..?

For the FreeBSD case, I dropped the system namespace.

If you want to continue to sync root.*, then there is a problem right
now with linux->{mac,bsd}->linux. I didn't catch that case before,
sorry.

root.foo (linux) -> root.foo (osx) -> user.root.foo (linux)

[Wesley W. Terpstra]

Post by Wesley W. Terpstra

Post by Wayne Davison
What is your reason for wanting to get rid of the '%' from the fake
stat attribute? Do some implementations not allow a % in the key?

No, but others are using 'reverse-DNS naming style' and this seems
to be standard-ish. That means a '.' and not a '%'.
Did you check the second link in my original email?: http://
www.freedesktop.org/wiki/CommonExtendedAttributes

One more detail: FAKE_XATTR should only include the "user." on linux.

Post by Wesley W. Terpstra
Here's a general question: should linux really be sync'ing system
and root namespaces?
I don't think even root has permission to write arbitrary stuff in
system..?
For the FreeBSD case, I dropped the system namespace.

I've read around a bit, and it seems that the tar people also took
this approach:
http://lists.freebsd.org/pipermail/posix1e/2006-March/thread.html#482
That is: don't copy the system namespace at all
Only sync the root namespace if explicitly requested. ==> In our case
I'd say that's -a run as root.

So, osx&bsd already skip over the system namespaces (osx hides it and
the bsd API needs a different 2nd parameter).
I propose linux do the same (rsync_xal_get):

for (left = name_size, name = namebuf; left > 0 ; left -=
len, name += l
rsync_xa *rxas;

len = strlen(name) + 1;

+#ifdef HAVE_LINUX_XATTRS
+ if (strncmp(name, USER_PREFIX, UPRE_LEN) != 0 &&
+ (strncmp(name, ROOT_PREFIX, RPRE_LEN) != 0 || !
am_root))
+ continue;
+#else
+ /* On systems without a root namespace, we emulate it
with a fake prefix */
+ if (strncmp(name, ROOT_PREFIX, RPRE_LEN) == 0 && !am_root)
+ continue;
+#endif

Post by Wesley W. Terpstra
If you want to continue to sync root.*, then there is a problem
right now with linux->{mac,bsd}->linux. I didn't catch that case
before, sorry.
root.foo (linux) -> root.foo (osx) -> user.root.foo (linux)

Since neither osx nor bsd have a root namespace, probably the second
step is the mistake.
When they transmit 'root.' *, they should not prepend 'user.'

#ifdef HAVE_LINUX_XATTRS
/* Linux already puts the namespace in the
key. */
write_int(f, rxa->name_len);
write_int(f, rxa->datum_len);
#else
/* Other platforms need the namespace added in,
* so use the "user." namespace for all
keys, except root. */
if (strncmp(rxa->name, ROOT_PREFIX,
RPRE_LEN) =! 0) {
write_int(f, rxa->name_len + UPRE_LEN);
write_int(f, rxa->datum_len);
write_buf(f, USER_PREFIX, UPRE_LEN);
} else {
write_int(f, rxa->name_len);
write_int(f, rxa->datum_len);
}
#endif

Of course, this begs the question of how many namespaces should we
allow.
I say: only 'user' and 'root' for now.

Oie. This interacts with the fake-super as well... You can't write to
root.* if you aren't root, so we would have to fake this too...

What do you think? Is this going in the right direction?

[Wayne Davison]

Post by Wesley W. Terpstra
That is: don't copy the system namespace at all

That's what my current patch does -- it discards any system namespace
attributes. If the receiver is not running as root (or with --super)
then we only try to set user attributes. This allows, for instance,
the security namespace to be synchronized when running as root

When running as a system that only has a user namespace, we store other
namespaces in our "rsync." hierarchy. When running as fake-root, we
do the same thing.

So, I think the current code has a good solution for what it should do
with the various namespaces, OS types, and options.

..wayne..

[Ensel Sharon]

Post by Wesley W. Terpstra
No, but others are using 'reverse-DNS naming style' and this seems
to be standard-ish. That means a '.' and not a '%'.
Did you check the second link in my original email?: http://
www.freedesktop.org/wiki/CommonExtendedAttributes

Heh ... just so we're all on the same page here ... you've _totally_ lost
me. I don't want to get in the way of this code-fest, but if you would
clarify something for me I would appreciate it greatly:


I don't know what acls are, honestly. I don't know what extended
attributes are. I don't much care.

All I know is, I have a FreeBSD backup server, and some dude in creative
wants to rsync his stuff up and _maintain_ the fancy little OSX meta-data,
etc.

So all I need to know are:

1. Is this even my problem ? Can I just stay at 2.6.6 and this is all
just a client side patch on his end ?

2. If it _is my problem_, does 2.6.9 with acls and xattr solve this, or do
I need to follow you down this road of "not sure if it works on freebsd
yet" ?

Again, I apologize for my cluelessness and for opening this can of worms -
I just wonder what the shortest path from A to B is ...

Many thanks.

[Wesley W. Terpstra]

Post by Ensel Sharon
All I know is, I have a FreeBSD backup server, and some dude in creative
wants to rsync his stuff up and _maintain_ the fancy little OSX meta-data,
etc.
1. Is this even my problem ? Can I just stay at 2.6.6 and this is all
just a client side patch on his end ?

There are two ways to backup osx meta-data to a non-osx machine with
rsync:

1. Use the absolute latest official rsync + acl and xattr patches.
That means 2.6.9 whenever it comes out, or the version in CVS. This
has to be upgraded on both the mac and on FreeBSD. The downside here
is that you can't read the meta-data back via nfs/samba. If the mac
only reads/writes the backup via rsync, then this doesn't matter.

=> upgrade mac+freebsd to 2.6.9+xattr patch once we fix it
=> no strange files created under freebsd
=> mac should not read the data via nfs/samba, only rsync

2. Hack apple's rsync. Apple changed rsync in Tiger (10.4) to support
backing up meta-data. Unfortunately, their version is very buggy.
There are patches that fix this, and I have an additional patch on
top of this which makes it possible to use this rsync with a normal
linux/bsd rsync server. In this case you would only need to change
the rsync on the mac, as the mac puts the meta-data into special
'appledouble' files.

=> screw with mac only (a special 'apple' 2.6.3)
=> creates ._filename files to hold meta-data
=> mac can read it via nfs/samaba as well as rsync

I can't really recommend either approach. #2 sounds superficially
better, and if you need samba/nfs access, it's your only option.
However, we're talking about using a completely tweaked rsync on the
mac that no one will support. Route #1 is likely to become more
'stable' in the future.

Post by Ensel Sharon
2. If it _is my problem_, does 2.6.9 with acls and xattr solve
this, or do
I need to follow you down this road of "not sure if it works on freebsd
yet" ?

At the moment either route you go will require you to run alpha
software.

Post by Ensel Sharon
Again, I apologize for my cluelessness and for opening this can of worms -
I just wonder what the shortest path from A to B is ...

There is no short path.
Backing up a mac properly to a non-mac is practically impossible.

[Victor Shoup]

There is a third way:

Simply keep all the files you want backed up in a UFS formatted
partition
on the mac. This way, all the metadata for file foo is in the
file ._foo,
and rsync backs these up perfectly, and all the data and metadata can
also be accessed vi NFS.

Restoring is a bit of a pain, as rsyncing from
the solaris to the mac gives some surprising results --- the ._foo
files tend to get erased
because of the way mac osx implements certain low-level file operations.
One can get around this by running rsync twice to restore the files
(or use apple's tar instead).

Anyway, this is my solution, and it works quite well for me.
However, an Apple engineer told me that UFS may not be maintained
forever
(but hopefully long enough until other solutions exist).

-- Victor

However, these issues are only for RESTORE, not for BACKUP.

Post by Wesley W. Terpstra
There is no short path.
Backing up a mac properly to a non-mac is practically impossible.