rsync over stdin/stdout

Post by Tony Mantler
So I've got this situation where I have two systems connected via a
protocol that looks nothing at all like IP, and I'd like to be able to
rsync between them.
I can, though, open a perfectly normal looking socket through various
magic and trickery.
I've managed to get rsync to work by using the "rsync
--rsh=mymagictransportprogram" style invocation, but for a number of
reasons this isn't really a viable long-term solution for me. What I'd
A: Set up stdin/stdout and exec rsync with some options that tell it
that there's a remote rsync on the other end of those FDs, or

ï»¿

Post by Tony Mantler
Basically I need to set up the socket beforehand and then hand it to
rsync somehow. This seems to be possible on the 'remote' end, but I
haven't managed to figure out how to do on the initiation side.

First, you should use an rsync daemon if you aren't already; with the
remote shell mode, you have to get the server arguments across the
socket yourself, whereas the daemon protocol takes care of that.

There's an enhancement request for a ï»¿"&3::module/src/" syntax to make
the client access a daemon over a socket supplied on a fd:

ï»¿https://bugzilla.samba.org/show_bug.cgi?id=5220

In the absence of this enhancement, you can accomplish the same thing
using the RSYNC_CONNECT_PROG environment variable (or --rsh, but
RSYNC_CONNECT_PROG is slightly more convenient). Have the
RSYNC_CONNECT_PROG inherit the socket from rsync and bridge the socket
to its own stdin/stdout for use by rsync. Here's an example, assuming
the socket is already on fd 3 of the shell:

RSYNC_CONNECT_PROG='cat >&3 & cat <&3; wait' rsync -a foo::module/src/ dest/

The hostname, here "foo", doesn't matter, since RSYNC_CONNECT_PROG
overrides it.

Post by Tony Mantler
B: Call rsync as a library, passing it a socket and directory(^Hies) to
sync with. (librsync doesn't seem to be quite what I want)

Rsync doesn't come as a library. Librsync is perhaps misnamed: it
contains only rsync's delta-transfer algorithm for individual files.

Matt

Tony Mantler

2008-06-02 21:29:06 UTC

The program I have handles that fairly well, actually. It's not too far
removed from complete rsh functionality.

Post by Matt McCutchen
There's an enhancement request for a "&3::module/src/" syntax to make
https://bugzilla.samba.org/show_bug.cgi?id=5220

That's pretty much exactly what I'm looking for. It's a shame it's just
sitting as an enhancement request, but I suppose the need for it is
fairly rare.

Post by Matt McCutchen
In the absence of this enhancement, you can accomplish the same thing
using the RSYNC_CONNECT_PROG environment variable (or --rsh, but
RSYNC_CONNECT_PROG is slightly more convenient). Have the
RSYNC_CONNECT_PROG inherit the socket from rsync and bridge the socket
to its own stdin/stdout for use by rsync. Here's an example, assuming
RSYNC_CONNECT_PROG='cat >&3 & cat <&3; wait' rsync -a foo::module/src/ dest/
The hostname, here "foo", doesn't matter, since RSYNC_CONNECT_PROG
overrides it.

That probably won't work very well in my case.

Matt McCutchen

2008-06-02 21:49:20 UTC

The program I have handles that fairly well, actually. It's not too far
removed from complete rsh functionality.

I don't understand. If your connection program is like rsh, it passes
arguments and then gives you a socket to talk to the invoked program.
But the server arguments are not determined until you start the rsync
client, after the socket has been set up. So how could the server
arguments be passed by the connection program? Or do you just mean that
the connection program passes "rsync --server --daemon ." without you
having to hardcode that on the remote side, in which case we're really
talking about the same thing?

Post by Matt McCutchen
In the absence of this enhancement, you can accomplish the same thing
using the RSYNC_CONNECT_PROG environment variable [...]
RSYNC_CONNECT_PROG='cat >&3 & cat <&3; wait' rsync -a foo::module/src/ dest/

That probably won't work very well in my case.

Why not? It does exactly the same thing as the enhancement would, but
with slightly higher overhead to pass the data back and forth.

Matt

Tony Mantler

2008-06-02 21:59:02 UTC

The program I have handles that fairly well, actually. It's not too far
removed from complete rsh functionality.

Well, when operating in the 'rsh'-like mode, rsync passes a bunch of
arguments to my program which my program hands across to the other side
to invoke the remote-side rsh.

What I need to do is turn this whole thing inside out, so that my
program launches rsync rather than rsync launching my program. The
reason for this is that my program needs to operate as a plugin as well
as standalone.

That probably won't work very well in my case.

Why not? It does exactly the same thing as the enhancement would, but
with slightly higher overhead to pass the data back and forth.

Well, it won't work on windows for one thing, and that's apparently one
of my target platforms. (not by choice, I assure you)

Matt McCutchen

2008-06-02 22:08:54 UTC

Post by Matt McCutchen
First, you should use an rsync daemon if you aren't already; with the
remote shell mode, you have to get the server arguments across the
socket yourself, whereas the daemon protocol takes care of that.

Well, when operating in the 'rsh'-like mode, rsync passes a bunch of
arguments to my program which my program hands across to the other side
to invoke the remote-side rsh.
What I need to do is turn this whole thing inside out, so that my
program launches rsync rather than rsync launching my program. The
reason for this is that my program needs to operate as a plugin as well
as standalone.

Right. I was suggesting an rsync daemon for the inside-out case where
you can't easily pass the server arguments using your program.

That probably won't work very well in my case.

Why not? It does exactly the same thing as the enhancement would, but
with slightly higher overhead to pass the data back and forth.

Well, it won't work on windows for one thing, and that's apparently one
of my target platforms. (not by choice, I assure you)

It might work under Cygwin, but you're right, that's iffy. I suggest
you compile your own rsync with the proposed patch:

https://bugzilla.samba.org/attachment.cgi?id=3116&action=view

Matt

Tony Mantler

2008-06-02 22:19:17 UTC

Post by Tony Mantler
What I need to do is turn this whole thing inside out, so that my
program launches rsync rather than rsync launching my program. The
reason for this is that my program needs to operate as a plugin as well
as standalone.

Right. I was suggesting an rsync daemon for the inside-out case where
you can't easily pass the server arguments using your program.

Ah, ok. Sure.