[prev in list] [next in list] [prev in thread] [next in thread] 

List:       infrastructures
Subject:    Re: [Infrastructures] how to build an internal (local) filestructure?
From:       stevegt () terraluna ! org (Steve Traugott)
Date:       2005-03-31 20:49:11
Message-ID: 20050331204911.GC9441 () terraluna ! org
[Download RAW message or body]

Hi Ivan,

This must be the week for the list founders to get pummelled.  ;-)

I said these tree structures can be extremely expensive "if not done
right, or if done for the wrong reasons"...  I think you are doing this
right, for the right reasons, and others can learn a great deal from
what you're doing.  That's why I mentioned you as an exception case.  I
think if I were to write this more formally I'd need to more clearly
state what "right" means; I think you would probably come up with nearly
the same criteria that I would.

I hope you do also discuss the "expense" side though -- the Konvalo FAQ
covers some of the issues inherent in trying to force apps and tools to
deal with an unconventional tree structure, so I know you've put a lot
of thought into this.  (Building on the Gnome example from the FAQ,
imagine a world in which the head of a client business unit is a
physicist-turned-trader who's living on coffee, can't hold a
conversation for more than 30 seconds, and is pulling down $15 million a
year in salary plus bonus.  For some insane reason he can't live without
Gnome, so you *have* to make it work, and keep it up to date with the
latest bells and whistles with every release, or he's going to go find
another IT department to support his group.)

For anyone who's interested in another example of people who thought it
through, had the resources, and went into the project with their eyes
open to what they were getting into, see Xev Gittler, Phil Moore, and J.
Rambhaskar's paper about Morgan Stanley's Aurora system:

  http://www.usenix.org/publications/library/proceedings/lisa95/gittler.html

I can testify that Aurora works.  Ivan, Xev's paper also provides a good
description of the sort of "mission-critical" we've been talking about
over the last several days.  (To show how the Gnome example above is
typical, see the CDE discussion in the paper -- "We have users that
actually make use of up to 60 workspaces"...)  

My cautions are aimed at cases where shops are *not* willing to deploy
AFS, Coda, or a globally consistent NFS/amd tree, they *don't* have 1800
applications, they *don't* have, or want to have, enough skilled people
or political will to sustain the upkeep effort forever, they *don't*
have a good network, and they *don't* want to rely on an external third
party for file and application service.  The worst case I've experienced
personally is the one I described in my previous message, in which the
skills were there, the network was there, but the tree was maintained on
the local disk of each machine -- so they had *both* the expense of the
tree-maintenance tools as well as the expense of local disk tools; worst
of both worlds.

There is another hard problem here that doesn't get talked about enough;
what is the right thing to do for perimeter and DMZ machines, bastion
hosts, firewalls?  Folks tend to say these should be minimal,
hardenened, *never* mounting any sort of network filesystem.  But of all
the hosts in the org these are usually the most demanding in terms of
the need for frequent security patches and uptime requirements.   I've
tended to think that this means you still need to maintain a suite of
tools capable of managing the bits on local disks, regardless of what
you do on internal machines.  This probably influences the way I think.

Ivan, do you have any way of managing the Coda client, kernel, and other
bits which are still on local disk?  I get the impression from
konvalo.org that right now you state the minimum requirements for local
disk content, and it's up to the user to manage that however they see
fit.  Have you worked out any ways for doing it for your own machines,
and how would you scale up?

Steve

On Thu, Mar 31, 2005 at 04:27:53PM +0200, Ivan Popov wrote:
> Hi Steve,
> 
> it is nice to see a solid analysis of the problems.
> 
> Nevertheless, I cannot agree with some of your arguments.
> 
> On Wed, Mar 30, 2005 at 10:22:09PM -0800, Steve Traugott wrote:
> > ...local disk is *cheap*, the methods for dropping an application onto
> > the local disk of the right machines have gotten much better in the last
> 
> It is not for the sake of local disk space we move things to a global
> place, it is to avoid the need for "methods for dropping an application onto".
> 
> With a global installation the time to deploy an application on
> another host is about zero. That amount is impossible to beat, with any
> approach.
> 
> > These tree structures normally prevent you from using vanilla .deb,
> > .rpm, .pkg, 'make install', and other vendor-supplied installation code;
> 
> Nothing prevent us to _use_ them even if we have a global installation.
> [but why would we bother doing that in many different ways on different
> hosts (with Debian use apt-get, with RedHat use up2date, otherwise
> make install, take different measures to ensure the programs are
> updated as necessary...) ?]
> 
> We make compilation and configuration once and use it for an unlimited number
> of hosts, updating when necessary as well in one sole place.
> Note that updates are both upstream upgrades and configuration changes
> (say, replacing the company logo as it is acquired by another :)
> 
> > you instead have to painstakingly spend hours or even days repackaging
> > something that you otherwise could have deployed to your users in a few
> > minutes.
> 
> I see it from another perspective. You can run any of 1800 programs
> from Konvalo.org in minutes on any of thousands hosts.
> It would cost many hours to push the applications to the local disks...
> 
> I can start my full environment (with those 1800 apps) at home over ADSL
> after full reinit faster than run RedHat's kickstart over a 100Mbit line.
> 
> Of course it took time to install all of the programs, but on the other
> side it takes time and money to build an alternative method of distributing
> programs as well.
> An apt-get install or make install does not help unless you can do it in a
> congruent way on all your hosts. That congruent way does not come for granted.
> It is also always less general, as you use someone's else structure (apt-get)
> or assumptions (make install).
> 
> > Most users on most machines in most
> > environments use a tiny set of applications, and benefit best from
> 
> That's why you win enormously by letting them run applications
> directly from the global file space instead of ensuring that tens
> of gigabytes are copied to - and regularly updated! - on each host,
> without any real need.
> 
> > tool-driven local disk installs, because it takes less of your labor per
> > application to do that, freeing you up to do other things.
> 
> It takes definitely more of my labor to let all users on all different
> platforms to successfully use the all different local disk installs.
> That is what drove me to put things globally.
> 
> > Think of it like this -- in most environments you need to manage that
> > local disk on every desktop machine anyway.  You need to ensure that the
> > bootloader, the kernel, the network configuration, libc, the
> > authentication and authorization subsystems,
> 
> Right.
> 
> >  and the 15,000 other files
> 
> No, here I definitely can not agree.
> The 15000 files are haunting you if you use the local-disk-based applications.
> 
> I run things on bare-bones hosts, the kernel and bourne sh, that's all
> what is necessary on the local disk. The rest can reside globally.
> Even libc that your applications use.
> 
> >     - You can afford to forever devote an entire, dedicated small army
> >       of people to managing the application tree, *and* that tree is in
> >       a globally mounted network filesystem,
> 
> Of course - it is the same people who would do the same thing by other means
> anyway. May be by buying the work in form of prebuilt packages from a software
> distributor like Red Hat - but you can buy the work of global placement as well
> (or better and cheaper?).
> 
> >                                             *and* you control local
> >       disk content as well.
> 
> My point is that you do not have to rely on the local disk contents.
> 
> Moreover, when you happen to manage the local disk also, you want
> to manage as little as possible there, as it is always more expensive
> than using the setup from the global file name space.
> 
> >       Note that the size of this army is dictated
> >       more by the number of applications and the release rate of those
> >       applications, rather than the number of client machines.  This
> >       means it scales well for large organizations, but will be a major
> >       impediment to growth and reliability for anyone else who tries it.
> 
> Worked acceptably for home installation with half a dozen computers :)
> Though of course it is best at large scale deployment.
> 
> >     - You are a dedicated third-party organization, either volunteers
> >       like Ivan or a commercial enterprise, which specializes in
> >       repackaging applications and serving them via the Internet to
> >       individuals and small-to-medium businesses using AFS, Coda, etc.
> 
> That is definitely the best place to use the technology.
> Anyone here with business experience and an ambition to be the first?
> I am pretty sure there is no competing technology like Konvalo yet
> (if we do not consider converting all applications to Java applets...,
> and even then :)
> 
> Of course as soon as anybody begins to realise the potential, he or she
> will implement the same ideas from scratch or just pull the tools from
> Konvalo.org (it is GPL-ed), but it is not trivial to do from scratch, and
> it will cost many man-months.
> 
> >       I think that I could also make a good argument that, if you don't
> >       control local disk too, you are not going to be able to provide
> >       guaranteed app functionality either, which means this might be
> >       confined to volunteer, non-mission-critical communities for at
> >       least the time being.
> 
> I think this explains why nobody did that - nobody believed that it
> would be possible, to split local disk management from applications
> management. 
> 
> Konvalo proves otherwise - thanks to global filesystems it has become possible.
> We can explicitely and exactly describe the constraints on the local disk
> contents and processor type.
> For the current volunteer work the level of guarantees is about
> what the existing free Linux distributions provide.
> For mission-critical work we can do exactly as good as necessary.
> 
> >     Deploying an app for your whole infrastructure should never cost you
> >     more than twice as much human time as installing it once on a single
> >     machine.
> 
> Installing all things globally fits that formula well,
> it takes exactly the same time for one computer or all :)
> 
> Regards,
> --
> Ivan
> 

-- 
Stephen G. Traugott  (KG6HDQ)
UNIX/Linux Infrastructure Architect, TerraLuna LLC
stevegt@TerraLuna.Org 
http://www.stevegt.com -- http://Infrastructures.Org
_______________________________________________
Infrastructures mailing list
Infrastructures@mailman.terraluna.org
http://mailman.terraluna.org/mailman/listinfo/infrastructures
[prev in list] [next in list] [prev in thread] [next in thread]