[prev in list] [next in list] [prev in thread] [next in thread] 

List:       puppet-dev
Subject:    [Puppet-dev] Using HTTP to download files
From:       luke () madstop ! com (Luke Kanies)
Date:       2007-01-29 16:12:25
Message-ID: 64709B3E-23CE-4E0F-9A1B-9DAF5CD08D42 () madstop ! com
[Download RAW message or body]

On Jan 28, 2007, at 8:44 PM, Ian Burrell wrote:

> In bug #451, I ran into some problems downloading large (26 MB) files
> through the puppet protocol.  The cause seems to be that the entire
> file contents is transfered through XML-RPC.  The file contents are
> also CGI escaped for the transfer.  Encoding the XML-RPC messages
> could also be a performance bottleneck.

Yep.  XML-RPC is a horrible way to transfer large files.

> A suggested solution is to download the files through a separate HTTP
> server.  But then I realized that Puppet already uses an HTTP server.
> It should be possible to stream the files through HTTP directly
> instead of wrapping them in XML-RPC.  By going through the WEBrick
> server, the file downloads would have access to the SSL authentication
> and fileserver mappings.
>
> Is this worth pursuing?

I *think* so.  I'm actually hoping to be able to switch from XML-RPC  
to some kind of RESTian interface, which should make most of these  
problems go away, but I'm not exactly on a schedule to do that, and I  
don't even know how it will work yet.  I also want to find a way to  
support other web servers, since webrick's scaling is very limited.

If you can come up with a simple patch that does file transfer over  
https instead of xmlrpc, I'd gladly apply it immediately.

If anyone's interested in helping to switch from xmlrpc to REST  
system-wide, I'd be even happier.  I don't even know what the best  
solution is in this space, since Puppet differs from most web servers  
in that (1) it isn't user-facing and (2) it requires information from  
the certificates.  I'm not sure if most of the common HTTP setups  
will work for it, and I haven't had the time to do all of the  
research necessary to determine this.  I don't see the need for  
separate application and web tiers, but then, (as evidenced by my  
choice of xmlrpc in the first place) network transport isn't exactly  
my strong suit.

  --
  You've achieved success in your field when you don't know whether what
  you're doing is work or play.  -- Warren Beatty
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com



[prev in list] [next in list] [prev in thread] [next in thread]