Axel Kittenberger
2010-11-20 21:56:13 UTC
Hello,
I'm author of Lsyncd - the live syncing deamon -
http://code.google.com/p/lsyncd/ - a daemon that uses Linux` inotify
to watch for filesystem changes - aggregates them for a few seconds
and then periodically calls rsync to transfer the changes to
target(s). Version 1 was simply aware of directories only, and it
called rsync once with --delete -d for every directory in which
anything has happened. I'm now coding Version 2, this goes further and
keeps track for every file. Beside doing this to optimize on moves, I
want to optimize performance also with its interface to rsync.
While running, Lsyncd knows exactly which files need transfer and
rsync's --files-from (with =- to get the filelist from stdin-pipe)
sounds like an ideal solution for this, also to reduce the number of
times rsync gets spawned. Coded that, now in beta stage there is a
nasty problem, rsync refuses to accept deletions that way!
It says e.g. "rsync: link_stat "/tmp/ltest.M2e/src/z/t/fq" failed: No
such file or directory (2)" and refuses to delete the file on the
target.
What good solutions can you suggest?
I thought about instead of "--files-from=-" to use "--recursive
--include-from=- --exclude=*". Beside I don't know how this evaluates
performance wise, it does not quite produce the effects, at least not
easily without further bloating the file-list. I suppose for every
directory on the include list, it needs to add rules for all parent
directories, so they get included in recursiveness.
Lsyncd does support a mode where ssh access is as well available on
the target. That way it can move moved files on the target, instead of
retransmitting them like a normal incremental rsync does. In that
case, it can as well simply log in there and call rm for removed
files. But I do want to support a rsync-only mode, for example if the
target runs the rsyncd daemon proving modules.
Could we get future versions of rsync accept "--delete --files-from=-"
rule to delete files and (recursively?) directories on the target that
are in the file-list but locally not existent?
Kind regards,
Axel
I'm author of Lsyncd - the live syncing deamon -
http://code.google.com/p/lsyncd/ - a daemon that uses Linux` inotify
to watch for filesystem changes - aggregates them for a few seconds
and then periodically calls rsync to transfer the changes to
target(s). Version 1 was simply aware of directories only, and it
called rsync once with --delete -d for every directory in which
anything has happened. I'm now coding Version 2, this goes further and
keeps track for every file. Beside doing this to optimize on moves, I
want to optimize performance also with its interface to rsync.
While running, Lsyncd knows exactly which files need transfer and
rsync's --files-from (with =- to get the filelist from stdin-pipe)
sounds like an ideal solution for this, also to reduce the number of
times rsync gets spawned. Coded that, now in beta stage there is a
nasty problem, rsync refuses to accept deletions that way!
It says e.g. "rsync: link_stat "/tmp/ltest.M2e/src/z/t/fq" failed: No
such file or directory (2)" and refuses to delete the file on the
target.
What good solutions can you suggest?
I thought about instead of "--files-from=-" to use "--recursive
--include-from=- --exclude=*". Beside I don't know how this evaluates
performance wise, it does not quite produce the effects, at least not
easily without further bloating the file-list. I suppose for every
directory on the include list, it needs to add rules for all parent
directories, so they get included in recursiveness.
Lsyncd does support a mode where ssh access is as well available on
the target. That way it can move moved files on the target, instead of
retransmitting them like a normal incremental rsync does. In that
case, it can as well simply log in there and call rm for removed
files. But I do want to support a rsync-only mode, for example if the
target runs the rsyncd daemon proving modules.
Could we get future versions of rsync accept "--delete --files-from=-"
rule to delete files and (recursively?) directories on the target that
are in the file-list but locally not existent?
Kind regards,
Axel