'[Infrastructures] Training Infrastructure Teams'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       infrastructures
Subject:    [Infrastructures] Training Infrastructure Teams
From:       devick () lanier ! com
Date:       2006-08-18 21:42:28
Message-ID: 20060818174228.B26037 () so ! lanier ! com
[Download RAW message or body]

I am a member of a team of Unix and Oracle administrators at a medium-sized
company; our team is responsible for managing the IT infrastructure for the
company, consisting of some 400 databases, web and application servers,
etc.  We are attempting to define a training process for new team members,
to introduce them to the principles we try to follow in managing the IT
infrastructure.  These people are typically competent administrators, but
may not have heard of Infrastructure Architecture as a discipline.

The attached file is an introducttion to the training process; it attempts
to lay out as concisely as possible the general principles that we try to
live by.  We will attempt to build on this as we develop further training
materials.  Since our philosophy seems close to that of
infrastructures.org, I would be grateful for any feedback the group may
have.

Don Vick
Lanier Worldwide (a division of Ricoh)

["trainingintro.html" (text/html)]

<html>
<head>
<title>
Infrastructure Management Principles
</title>
</head>
<body>

<h3>
The Craft
</h3>

Analogies with sports or war
are popular tools for explaining business principles.
Pithy sayings of coaches are cast into business terms;
or ancient books like "The Art of War" or Machiavelli's "The
Prince" are held up as sources of business wisdom.
One such book is "The Book of Five Rings",
written circa 1643 by an aging samurai, Miyamoto Musashi,
who muses at length on the way of the warrior.
Early in the book, Musashi considers various ways of life as models for
explaining the principles that make a warrior successful.
After considering the farmer, merchant, and nobleman, he settles on
the artisan as the best analogy for a warrior, and gives this summary:

The way of the artisan ... involves skillful construction of all sorts of tools,
knowing how to use each tool skillfully,
drawing up plans correctly ...
making a living by diligent practice of the craft.

The craft that we practice is the management of the corporate IT infrastructure.
Attempts to teach this craft run headlong into the overwhelming complexity
of a modern IT infrastructure,
and elicit analogies such as "drinking from a fire hose".
Here we attempt to set down the basic principles of our craft,
in a form that is general enough to transcend differences in
architectures and applications,
but specific enough to guide newcomers into an understanding of the
team's approach to the task.

<h3>
Nimbleness
</h3>

There is an old saying that good judgement comes from experience,
and experience comes from bad judgement.
The ability of a business to cope with changing conditions
and unexpected situations -- to recover from the sub-optimal decisions
of the past -- depends on nimbleness.
Nimble organizations can mobilize the right resources on short notice
to respond to needs that were only dimly seen in forecasts.
Nimble groups have the right people with the right skills available
at a moment's notice.

But how does one become nimble?
Consider the nimble hands and feet of a shortstop,
the nimble fingers of a concert violinist,
or the smooth coordination of a team of firefighters.
Nimbleness comes from relentlessly practicing the basics.
Nimble groups can spring into action with a minimum of preparation
because they have determined ahead of time how they will work together;
they have practiced the basic skills of their craft so that,
when the time comes to act, they can act with one mind.

<h3>
Idle Time
</h3>

<p>
<i>
War should be the only occupation of a prince.
He should consider peace only as a breathing time
that gives him the leisure to contrive,
and furnishes the ability to execute, military plans.
-- Machiavelli, The  Prince
</i>
</p>

An organization with no spare capacity must tell new requestors
to wait in line;
by the time they can respond, the opportunity may have passed them by.
Managers rightly balk at the idea of idle workers,
so we must be clear about how idle time is to be used.
If time is money, then available time must be invested for future gain.
Idle time is the result of doing one's work quickly and well.
If we use our idle time to hone our skills,
to prepare for the next challenge,
we will be able to do more and more, and still have idle time to reinvest.

<h3>
Body of Knowledge
</h3>

Expertise does not appear out of nowhere.
Practitioners of a craft build up over time a body of knowledge --
the facts, theories, values, and practices which everyone who pursues
the craft is expected to learn.
Proficiency with the body of knowlege is a rite of passage for newcomers,
the continual assignment of every worker,
and the measure of a professional's skill.
The body of knowledge represents the desired state of affairs,
the standard of quality by which we judge our work.
Achieving that state is a never-ending process,
and the responsibility of every member of a team.

<h3>
Convergence
</h3>

The corporate assets over which we preside must be managed in such a way
that every act, every change, takes us closer to the desired state.
<a href=http://www.infrastructures.org>Infrastructures.org</a>
discusses the idea of convergence -- always moving forward,
preserving the present conformance to the body of knowledge,
and managing change in a way that
brings each changed component, process, or procedure more into line with
the standard set by the body of knowledge.
Convergence requires persistence, but it reduces the cost of support,
makes it more scalable, and results in increasing quality over time.

<h3>
Creating New Knowledge
</h3>

Every team member is expected to contribute to the body of knowledge,
but new contributions are not accepted casually.
Peer review is a key element.
Senior members of the team check the quality of new contributions,
and ensure that a harmonious whole is maintained.
New requirements are incorporated into the body of knowledge
by means of abstraction and generalization,
always seeking the simplest solution that meets all needs.
Kernighan and Pike speak of the "standard paradigm" --
a generally accepted best way of doing a particular task.
In creating new knowledge,
we seek to follow the standard paradigm as much as possible,
merging the new knowledge seamlessly into the old.

<h3>
15 Minutes vs 15 Days
</h3>

A problem occurs.
You have 15 minutes to save the world.
When that is done, you have 15 days to make sure it never happens again.

When we encounter exceptions to the existing rules,
we see not only the need for a repair to the existing operation,
but also a development task,
either to find and correct the root cause of the problem,
or to change the existing quality standards, to subsume the anomaly
under a more general rule.
We want to make sure either that the unexpected event does not recur,
or that it is no longer unexpected,
but can be handled as a part of routine operations.

<h3>
Change Control
</h3>

<i>
If I have seen farther than others,
it is because I have stood on the shoulders of giants.
-- Isaac Newton
</i>

Forward movement is only possible by building on the past.
Change control is the management process by which the body of knowledge
is altered.
Quality control asks not only "Does it work?",
but also "Is it consistent with what we have done before?"
Every change not only goes through a submission and approval process,
but also leaves permanent records behind.
Prior states of the body of knowledge are retrievable from archives.
Changes are deployed automatically to ensure that
working environments remain consistent with each other
and with the body of knowledge.

<h3>
Commitment
</h3>

You can learn all the math in the 'verse, but take a boat in the air you
don't love, she'll shake you off just as sure as the turning of the worlds. 
Love keeps her in the air when she oughta fall down, tells you she's
hurting 'fore she keens. Makes her home. -- Malcom Reynolds, Captain of
Firefly

Real systems break.  If we offer a system to our users,
we have an obligation not only to make it work,
but to think carefully about how it will fail.
Failure modes are not created anew with each system;
most kinds of problems are common to all kinds of systems.
Record-keeping, monitoring, quality testing, and support plans
are not optional features to be added if problems occur;
they are the expression of our professional commitment.
And once the system is in operation,
we must pay attention, listen to its voice, and respond to its needs.

<h3>
Unity
</h3>

A successful team is more than a group of people.
Successful teams develop a common language that lends both
richness and economy to their communications;
common tools and techniques,
so that everyone can anticipate what everyone else will be doing;
common values that engender agreement about what is important and what is not.
A team develops a common view of the world,
so that tasks and challenges, even apparently new ones,
take on a familiar air -- the feeling that
"We've been here before, and we know what to do."

<h3>
Consistency
</h3>

Tasks that the team does often should settle into a stable performance pattern.
Estimates of the time required to complete a task become more definite.
Variation and risk are reduced.
Rework should be almost unheard of.
Training programs can be tuned to make sure each team member knows
how a task should be approached in order to meet the expected time target.
Predictable task times lead to more effective triage during incident management,
and more confidence in project planning.
The team is seen as more reliable, and therefore more valuable to the company.

<h3>
Communication
</h3>

Teams must communicate with one another to do their work.
During its formative stages, a team requires a high level of communication.
Much information must be exchanged while establishing team norms.
Asynchronous or low-bandwidth communications like
email, chats, or telephones are not adequate.
But once the team is established,
the quantity and urgency of communication drops dramatically.
Elaborate discussions on how to approach a problem
are reduced to a few sentences.
Sentences are reduced to a word or two.
Much of what needs to be said can be said in email or logs, to be read later.
And much can remain unsaid, being implicit in the team's preparation.
Economy of communication is the mark of a well-prepared team.

But such economy is only for the team;
to newcomers and others, the terseness that works so well for the team
only serves to make their communications inscrutable.
Communications outside the team must be more elaborate,
in plain language and not in the private language of the team.
The team must learn how to keep others informed,
even in the midst of their work.

<h3>
Time and Task Management
</h3>

Time and task management are critical to effectiveness.
In The Effective Executive, Peter Drucker defines effectiveness as
"getting the right things done".
The way we manage our tasks determines what we attempt to get done,
and the way we manage our time determines how long we have to do it.
And if we have decided that a task must be done,
then we must find sufficient time, in sufficient intervals,
to accomplish it.

Task management can benefit from "triage", classifying tasks or problems
into "now", "later" and "never".
"Time boxing" allows a manager to limit the time invested on a task,
to allow for an initial evaluation before commiting fully.
"Time blocking" is the practice of reserving blocks of time for a specific use,
either for individuals or groups,
so that urgent tasks can be set aside temporarily in order to concentrate
on work that cannot be done effectively in small time slices.

<h3>
Tools versus Appliances
</h3>

<p>
<i>
Practice doesn't make perfect.  <b>Perfect practice</b> makes perfect.
-- Vince Lombardi
</i>
</p>

The tools we use to do our jobs can be our friends or our enemies.
If, when using a tool, we are thinking about how it works,
and what it is doing for us,
then each use of the tool becomes a practice session.
Our skills are kept fresh, even enhanced, through the use of the tool.
If the tool ever fails, as tools will,
or we have to work in a new environment where the tool is not available,
then we may work slower,
but we will not have forgotten how to do the job.
On the other hand, if we use a tool as an appliance,
without thinking about what it is doing,
our skills become dull and stale.
Then when, inevitably, the tool fails, we will be lost,
unable to do the job we once knew how to do.
Tools can enable us to work faster,
but they should not be allowed to steal our skills.

<h3>
The Arrow of Time
</h3>

Success is not final. Failure is not fatal.
It is the courage to continue that counts. -- Winston Churchill

All systems, both real and virtual,
have an inexorable tendency to move toward their most likely state.
Ice melts, hot coffee cools, and houses of cards fall
because their final state is more likely than the first.
Physicists speak of "the arrow of time",
the irreversible movement of the physical universe toward its final state.
Human institutions and artifacts change over time.
If we strive to make all changes take us toward the ideal,
to seek out and eliminate inconsistencies and errors,
to keep our goals in view even when we cannot move directly toward them,
then we can make the arrow of time work in our favor.
Well managed systems become better over time,
because the people who tend them refuse to allow the alternative.

</body>
</html>

_______________________________________________
Infrastructures mailing list
Infrastructures@mailman.terraluna.org
http://mailman.terraluna.org/mailman/listinfo/infrastructures

[prev in list] [next in list] [prev in thread] [next in thread] 