[prev in list] [next in list] [prev in thread] [next in thread]
List: gpfsug-discuss
Subject: Re: [gpfsug-discuss] Compatibility question
From: "Steven Daniels" <sadaniel () us ! ibm ! com>
Date: 2019-11-22 17:01:09
Message-ID: OFE09CAEC5.B9B5B0A2-ON872584BA.005AEC92-872584BA.005D7D76 () notes ! na ! collabserv ! com
[Download RAW message or body]
--=_related 005D7CF1872584BA_Content-Type: multipart/alternative; \
boundary="=_alternative 005D7CF1872584BA_="
--=_alternative 005D7CF1872584BA_Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain; charset="US-ASCII"
Felipe,
You are correct. The code is restricted to 4.2.2 or higher. If there is an
ESS storage cluster in the environment then the recommendation I was told
is 4.2.3.7 or higher.
Luke,
If your clients are in the same cluster then you need to be careful with
the setting to minReleaseLevel, Exactly as it sounds, it will restrict
membership to this level or higher.
I have a client running with filesystem version 3.5.0.7 in several
clusters.
We just recently upgraded their ESS systems to the current release so I
can verify that the file system format will carry fine.
If you have ESS at 5.3.3 or higher, then the GNR recommendation is 4.2.3.7
or higher for the clients (my client can't go to v5 due to the RHEL
version requirement) and they must be in a remote cluster.
In my case we upgraded an ESS storage cluster to 4.1.1 to 5.0.3.3 (ESS
v5.3.4.1) and for GNR we had to update this cluster to LATEST for
minReleaseLevel which is 5.0.3.
The client cluster is running 4.2.3.18 which is as Felipe says is N-1 and
minReleaseLevel is set to 4.2.3.7
This client cluster also mounts from another storage cluster which is
constrained to run at 4.1.1-8. This makes the client cluster at N+1 from
its relationship with this cluster.
All three clusters have minReleaseLevel set to LATEST (4.1.0,
4.2.3.7,5.0.3 , respectively).
These changes would not work in a single cluster as all nodes would be
restricted to the 4.1.0 or higher and the ESS would not be able to
co-exist since it requires the 5.0 level.
We have updated the file systems in the 4.1.1-8 cluster from 3.5.0.7 to
4.1.1 and we updated the ESS filesystems from 3.5.0.7 to 5.0.3 (compat
mode)
The changes all went well.
Hopefully this helps you understand the nuance that is whether the clients
and NSDs are in a single cluster or different clusters. The rules are
slightly but importantly different.
Also, when updating the filesystem versions make sure you use the "compat"
flag if you have any remote clusters. If we would have run this with the
"full" flag on the ESS storage cluster, it would have prevented the
4.2.3.18 client cluster from mounting its file systems.
thanks,
Steve
Steven A. Daniels
Cross-brand Client Architect
Senior Certified IT Specialist
National Programs
Fax and Voice: 3038101229
sadaniel@us.ibm.com
http://www.ibm.com
From: "Felipe Knop" <knop@us.ibm.com>
To: gpfsug-discuss@spectrumscale.org
Cc: gpfsug-discuss@spectrumscale.org
Date: 11/21/2019 09:49 PM
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compatibility question
Sent by: gpfsug-discuss-bounces@spectrumscale.org
George,
Coexistence either in the same cluster or across remote clusters is only
supported with versions "N" and "N-1". 5.0.3 and 3.5 are "N" and "N-3",
and this is not supported. If I recall, there is even logic in place that
will prevent 5.x nodes from accepting connections from nodes running below
4.2 .
Felipe
----
Felipe Knop knop@us.ibm.com
GPFS Development and Security
IBM Systems
IBM Building 008
2455 South Rd, Poughkeepsie, NY 12601
(845) 433-9314 T/L 293-9314
----- Original message -----
From: Luke Raimbach <luke.raimbach@googlemail.com>
Sent by: gpfsug-discuss-bounces@spectrumscale.org
To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>
Cc:
Subject: [EXTERNAL] Re: [gpfsug-discuss] Compatibility question
Date: Thu, Nov 21, 2019 8:12 PM
That would make my eyes water and maybe fall out. Yours might too of you
try it.
Be very careful with v3 upgrades, especially going up to v5 directly.
You will certainly run into RPC incompatibility problems with the v3 and
v5 nodes running concurrently. Staying within the supported envelope of
"one major version" difference will ease the raising of support tickets
with IBM if you run into problems.
Why can't you upgrade the clients?
On Thu, 21 Nov 2019, 18:24 George Terry, <gterryc@vmsupport.com> wrote:
Hello,
I have a question about compatibility between two different versions. I
have a cluster with 6 gpfs nodes. 2 of them are NSD servers the other 4
are clientes nodes. I will upgrade the NSD servers from 3.5.0 to 5.0.3.
Due to application's issues I cannot update the client nodes. So, my
question is, can i have a cluster with client nodes with 3.5.0.7 version
and the NSD nodes in 5.0.3 version with de filesystem in 3.5.0.7 (13.23)
version?
Thanks for the support.
George
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug \
-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=6mf8yZ-lDnfsy3mVONFq1RV1ypXT67SthQnq3D6 \
Ym4Q&m=sHhNGNTp4CJIVbXT4t8ZvNT_mETKfXBn7XTvNDAYF7s&s=9ialbzpuZMc8DZzXtU_nqjgQ9BPsHP3DQdpHa0nKQuc&e= \
--=_alternative 005D7CF1872584BA_Content-Type: text/html; charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable
X-Proofpoint-UnRewURL: 5 URL's were un-rewritten
<span style=" font-size:10pt;font-family:sans-serif">Felipe,</span><br><br><span \
style=" font-size:10pt;font-family:sans-serif">You are correct. The code is \
restricted to 4.2.2 or higher. If there is an ESS storage cluster in the environment \
then the recommendation I was told is 4.2.3.7 or higher.</span><br><br><span style=" \
font-size:10pt;font-family:sans-serif">Luke,</span><br><br><span style=" \
font-size:10pt;font-family:sans-serif">If your clients are in the same cluster then \
you need to be careful with the setting to minReleaseLevel, Exactly as it sounds, it \
will restrict membership to this level or higher. </span><br><br><span style=" \
font-size:10pt;font-family:sans-serif">I have a client running with filesystem \
version 3.5.0.7 in several clusters.</span><br><br><span style=" \
font-size:10pt;font-family:sans-serif">We just recently upgraded their ESS systems to \
the current release so I can verify that the file system format will carry fine. \
</span><br><br><span style=" font-size:10pt;font-family:sans-serif">If you have ESS \
at 5.3.3 or higher, then the GNR recommendation is 4.2.3.7 or higher for the clients \
(my client can't go to v5 due to the RHEL version requirement) and they must be in a \
remote cluster.</span><br><br><span style=" font-size:10pt;font-family:sans-serif">In \
my case we upgraded an ESS storage cluster to 4.1.1 to 5.0.3.3 (ESS v5.3.4.1) \
and for GNR we had to update this cluster to LATEST for minReleaseLevel which
is 5.0.3. </span><br><br><span style=" font-size:10pt;font-family:sans-serif">The \
client cluster is running 4.2.3.18 which is as Felipe says is N-1 and minReleaseLevel
is set to 4.2.3.7</span><br><br><span style=" \
font-size:10pt;font-family:sans-serif">This client cluster also mounts from another \
storage cluster which is constrained to run at 4.1.1-8. This makes the client \
cluster at N+1 from its relationship with this cluster.</span><br><br><span style=" \
font-size:10pt;font-family:sans-serif">All three clusters have minReleaseLevel set to \
LATEST (4.1.0, 4.2.3.7,5.0.3 , respectively).</span><br><br><span style=" \
font-size:10pt;font-family:sans-serif">These changes would not work in a single \
cluster as all nodes would be restricted to the 4.1.0 or higher and the ESS would not \
be able to co-exist since it requires the 5.0 level.</span><br><br><span style=" \
font-size:10pt;font-family:sans-serif">We have updated the file systems in the \
4.1.1-8 cluster from 3.5.0.7 to 4.1.1 and we updated the ESS filesystems from 3.5.0.7 \
to 5.0.3 (compat mode)</span><br><br><span style=" \
font-size:10pt;font-family:sans-serif">The changes all went well.</span><br><br><span \
style=" font-size:10pt;font-family:sans-serif">Hopefully this helps you understand \
the nuance that is whether the clients and NSDs are in a single cluster or different \
clusters. The rules are slightly but importantly different.</span><br><br><span \
style=" font-size:10pt;font-family:sans-serif">Also, when updating the filesystem \
versions make sure you use the "compat" flag if you have any remote \
clusters. If we would have run this with the "full" flag on the ESS storage \
cluster, it would have prevented the 4.2.3.18 client cluster from mounting its \
file systems.</span><br><br><span style=" \
font-size:10pt;font-family:sans-serif">thanks,</span><br><span style=" \
font-size:10pt;font-family:sans-serif">Steve</span><br><br><br><span style=" \
font-size:10pt;font-family:Arial">Steven A. Daniels</span><br><span style=" \
font-size:10pt;font-family:Arial">Cross-brand Client Architect</span><br><span \
style=" font-size:10pt;font-family:Arial">Senior Certified IT \
Specialist</span><br><span style=" font-size:10pt;font-family:Arial">National \
Programs</span><br><span style=" font-size:10pt;font-family:Arial">Fax and Voice: \
3038101229</span><br><a href="mailto:sadaniel@us.ibm.com"><span style=" \
font-size:10pt;color:blue;font-family:Arial"><u>sadaniel@us.ibm.com</u></span></a><br><a \
href="http://www.ibm.com/"><span style=" \
font-size:10pt;color:blue;font-family:Arial"><u>http://www.ibm.com</u></span></a><br><img \
src=cid:_2_10815644108153D8005D7CF1872584BA style="border:0px \
solid;"><br><br><br><br><span style=" \
font-size:9pt;color:#5f5f5f;font-family:sans-serif">From: \
</span><span style=" font-size:9pt;font-family:sans-serif">"Felipe \
Knop" <knop@us.ibm.com></span><br><span style=" \
font-size:9pt;color:#5f5f5f;font-family:sans-serif">To: \
</span><span style=" \
font-size:9pt;font-family:sans-serif">gpfsug-discuss@spectrumscale.org</span><br><span \
style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Cc: \
</span><span style=" \
font-size:9pt;font-family:sans-serif">gpfsug-discuss@spectrumscale.org</span><br><span \
style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Date: \
</span><span style=" font-size:9pt;font-family:sans-serif">11/21/2019 \
09:49 PM</span><br><span style=" \
font-size:9pt;color:#5f5f5f;font-family:sans-serif">Subject: \
</span><span style=" \
font-size:9pt;font-family:sans-serif">[EXTERNAL]
Re: [gpfsug-discuss] Compatibility question</span><br><span style=" \
font-size:9pt;color:#5f5f5f;font-family:sans-serif">Sent
by: </span><span style=" \
font-size:9pt;font-family:sans-serif">gpfsug-discuss-bounces@spectrumscale.org</span><br><hr \
noshade><br><br><br><span style=" \
font-size:10pt;font-family:Arial">George,</span><br><span style=" \
font-size:10pt;font-family:Arial"> </span><br><span style=" \
font-size:10pt;font-family:Arial">Coexistence either in the same cluster or across \
remote clusters is only supported with versions "N" and "N-1". \
5.0.3 and 3.5 are "N" and "N-3", and this is not supported. If I \
recall, there is even logic in place that will prevent 5.x nodes from accepting \
connections from nodes running below 4.2 .</span><br><span style=" \
font-size:10pt;font-family:Arial"> </span><br><span style=" \
font-size:10pt;font-family:Arial"> Felipe</span><br><span style=" \
font-size:10pt;font-family:Arial"> </span><br><span style=" \
font-size:10pt;font-family:Arial">----<br>Felipe Knop knop@us.ibm.com<br>GPFS \
Development and Security<br>IBM Systems<br>IBM Building 008<br>2455 South Rd, \
Poughkeepsie, NY 12601<br>(845) 433-9314 T/L 293-9314<br> </span><br><span style=" \
font-size:10pt;font-family:Arial"> </span><br><span style=" \
font-size:10pt;font-family:Arial"> </span><br><span style=" \
font-size:10pt;font-family:Arial">----- Original message
-----<br>From: Luke Raimbach <luke.raimbach@googlemail.com><br>Sent by: \
gpfsug-discuss-bounces@spectrumscale.org<br>To: gpfsug main discussion list \
<gpfsug-discuss@spectrumscale.org><br>Cc:<br>Subject: [EXTERNAL] Re: \
[gpfsug-discuss] Compatibility question<br>Date: Thu, Nov 21, 2019 8:12 PM<br> \
</span><br><span style=" font-size:10pt;font-family:Arial">That would make my \
eyes water and maybe fall out. Yours might too of you try it. </span><br><span \
style=" font-size:10pt;font-family:Arial"> </span><br><span style=" \
font-size:10pt;font-family:Arial">Be very careful with v3 upgrades, especially going \
up to v5 directly.</span><br><span style=" \
font-size:10pt;font-family:Arial"> </span><br><span style=" \
font-size:10pt;font-family:Arial">You will certainly run into RPC incompatibility \
problems with the v3 and v5 nodes running concurrently. Staying within the supported \
envelope of "one major version" difference will ease the raising of support \
tickets with IBM if you run into problems.</span><br><span style=" \
font-size:10pt;font-family:Arial"> </span><br><span style=" \
font-size:10pt;font-family:Arial">Why can't you upgrade the clients?</span><br><span \
style=" font-size:10pt;font-family:Arial"> </span><br><span style=" \
font-size:10pt;font-family:Arial">On Thu, 21 Nov 2019, 18:24 George Terry, \
<</span><a href="mailto:gterryc@vmsupport.com" target="_blank"><span style=" \
font-size:10pt;color:blue;font-family:Arial"><u>gterryc@vmsupport.com</u></span></a><span \
style=" font-size:10pt;font-family:Arial">> wrote:</span><br><span style=" \
font-size:10pt;font-family:Arial">Hello,<br><br>I have a question about compatibility \
between two different versions. I have a cluster with 6 gpfs nodes. 2 of them are NSD \
servers the other 4 are clientes nodes. I will upgrade the NSD servers from 3.5.0 to \
5.0.3. Due to application's issues I cannot update the client nodes.
So, my question is, can i have a cluster with client nodes with 3.5.0.7
version and the NSD nodes in 5.0.3 version with de filesystem in 3.5.0.7
(13.23) version?<br><br>Thanks for the support. </span><br><span style=" \
font-size:10pt;font-family:Arial"> </span><br><span style=" \
font-size:10pt;font-family:Arial">George</span><p \
style="margin-top:0px;margin-Bottom:0px"><span style=" \
font-size:10pt;font-family:Arial">_______________________________________________<br>gpfsug-discuss \
mailing list<br>gpfsug-discuss at </span><a href="http://spectrumscale.org" \
target="_blank"><span style=" \
font-size:10pt;color:blue;font-family:Arial"><u>spectrumscale.org</u></span></a><span \
style=" font-size:10pt;color:blue;font-family:Arial"><u><br></u></span><a \
href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><span \
style=" font-size:10pt;color:blue;font-family:Arial"><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></span></a></p><br><tt><span \
style=" font-size:10pt">_______________________________________________<br>gpfsug-discuss \
mailing list<br>gpfsug-discuss at spectrumscale.org</span></tt><tt><span style=" \
font-size:10pt;color:blue"><u><br></u></span></tt><a \
href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss" target="_blank"><tt><span \
style=" font-size:10pt;color:blue"><u>http://gpfsug.org/mailman/listinfo/gpfsug-discuss</u></span></tt></a><tt><span \
style=" font-size:10pt"></span></tt><br><span style=" \
font-size:10pt;font-family:Arial"> </span><br><tt><span style=" \
font-size:10pt">_______________________________________________<br>gpfsug-discuss \
mailing list<br>gpfsug-discuss at spectrumscale.org<br></span></tt><a \
href="http://gpfsug.org/mailman/listinfo/gpfsug-discuss"><tt><span style=" \
font-size:10pt">http://gpfsug.org/mailman/listinfo/gpfsug-discuss</span></tt></a><tt><span \
style=" font-size:10pt"><br></span></tt><br><br><BR>
--=_alternative 005D7CF1872584BA_=--
--=_related 005D7CF1872584BA_Content-Transfer-Encoding: base64
Content-Type: image/jpeg
Content-ID: <_2_10815644108153D8005D7CF1872584BA>
/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0a
HBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIy
MjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCABUAHgDASIA
AhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQA
AAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3
ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWm
p6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEA
AwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSEx
BhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElK
U1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3
uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD3+imS
yxwQvNNIkcUalndzhVA5JJPQUkdxDMcRzRvyw+VgeVO1vyPB9DQFySorm6t7K3a4up4oIUxuklcK
q5OBknjqa4/xv4+h8NYsrFYrjU2wzI+SkK9fmwQckdBnvk9s854W8HX3ixhrniu4uZrd8m3gdyDI
Cc5/2E9AMZ68DGaUdLs5J4r957KkuaX4L1PQ/wDhJ/D/AP0HdM/8C4/8a0opY54UmhkWSJ1DI6HI
YHkEEdRWeLZtMha30XSbONSwdgXFvGxOQcbFYlhtXqoGCOeMVyWo+JtP8GSyRXGkiwuynmQ2unzK
8F12GRgFDluWKAny8AtjaRRcnZFyqunrU2+Z3ssscELzTSJHFGpZ3c4VQOSST0FV7HUYdR3vbJK0
C42zshVJM/3CeWGMHcBtORgnnHHabo+razD/AG74stHvJCFNrokWFii7b2R2Cl8En5jwCe+AuvY+
NtPvL62tJbW8tHureO4gadEKukjbUOUZtuSQPmxyQOpANOHbUca12nLRP+vkdLRWRN4m0qHS73Uf
tG6CzTzH42lgfulN2AwborD5WPQmoG8W2FuZk1C3vbGaFlVoZYDIxDIzhl8reCuI5OQeNhzip5Jd
jR1YLqb1FZSeJdHl3eVerIoZFV41ZlkLMqDYwGHwzqDtztLAHFRXPiewi0qz1K233ltdsRE0LIow
Ed2LGRlCgBGzkggjFHK+we0h3Nqisa38U6TcwrIk7cwiYhUMgALlAu5MqWLqyhVJLEcZqxDr2m3F
xDBFc7pZsbE2MDkiQ4PHyn9zKCDggqQcHAo5X2BVIPZmjRVPTdVs9XtjcWMxliBwSUZTyAw4YA4I
ZSD3BBHWipatuUmmrolvbSK/sLiznBMNxE0UgBwdrAg/oa53Wm07wZpGoa1Cn+luZPL8x92+SR9w
UAkfKGJbaOg3kck56mvKvjHqX/IM0tJf71xLFt/4Chzj/roMA/XtVRu9Dmxc1SpOp16HHeGtNbxV
4pebUp0EClry/lkO0FAct0IxknGRjAOe1eq6d4403UPFFv4d0SFZLeIOJJ1AEQVF4EYHUZwM9MDj
IORm+AfC6f8ACA3glLxT6zE6u7IQUjIZE4J54JYHjO76GvOfh/rFpoGsX2qXbJ/o9jIYo2ODLIWU
Kg68nPXBwMnoDXRCCmpPsebQ5sNGF/t6t+R6/wCNvG1r4Ssdq7J9TmXMFuTwB03v6L+pPA7kedX/
AIOvl8I6p4q8SSSvq0/ltFE5wYwZFGWHY7TgL0Uds8Ld8CaRceNPE914s1kK0cMwKKhCq0qgbRgc
7VXb168ZJ+at/wCL1+IPDtpYrK6yXNxuKDOHRAcg/wDAmQ4/wpP937kd+v8AkbVn7alKtP4bPlX4
XN7wFrTa54RtJ5pfMuYcwTt82dy9CSepKlST6k/SrS+FdMSW2kCS7re3gto/n/ghkEifjuUZ9a5f
4TSrB4NvpXDlUvJGIRCzECNOgAJJ9hzWX8TNT1fyrSxW9nhuNQYCPS7bblY/mU+ayks7OWUBR8ny
sPmwGMQi3KydjaNZLDRqTV3Y7LTfDHhvTv7R0i3ETPexg3FsZRv8oDao2jBAGcbupJySTzUsnhSJ
4LZF1G986KQs928m64dfKkjVQ/8ADtEhIIHXJ+8xaqkfw88Pjw2mky2Fu0ghCNeLEFmL9S4bkjnn
GSO3Tisz4VeJLnXNDuLS9klmurFxmaQli6PkrliSSQQw7cbfeh3acovY1XKpRpzilfa33m5a+CNC
sbt7i1tfJLNGwVMAKUdH44zgtGhwSRwcY3Nm4vh2xj063s4mniFtM9xBKkhDxyMXJYdj99uGBBBw
QRXP/wBuy+MfETaVo1wyaPYsH1C9gmKPMedscbDnaSvLDGQDggY3Z+nNbX3jqK+8O2UEGhaRDLHd
3NpBtFy7KcooT/WgEIQMdcnnK5fLJ7sXPTVuSOjdv87eSOtTwzp0NvLFD58fmBPnMpdwySvKr5bO
W8x2bnIPcEcUlp4X0+yurW5ha432+T88pfzGPmfMxOSTmaU9R9/2GOC+JN29/wCHdPa8ja3ur+5H
2OzuHMf2eMdXc5C7zlM7zhQ2AAQzHpBFZr4Hjn1S4lOhWlgqRpFIYWukCBQ7YYH5xwsef4vmyxAQ
cZcqbe4KpBzcVHZf8H5G94f0KDQNNjtkkM02xFmnZQDIUQIvA6AKoAHtzkkklcX8HNIltNBu9TkJ
C30gWNeOVj3Dd17szDB/u+9FZ1VabV7m2HlzUlK1j0mvFfi//wAjba/9eKf+jJK9qrgvitocmpeH
4tQgDtJp7MzIOcxtjccAdRhT2AG6lB2ZhmFNzw7t01O1sLOPT9OtbKJmaO3iSJC5BYhQAM478V81
X2jlvGF1ounoSftz20Cse28qMn+tfRuh6vBrujWuowYCzICyA52N/EucDocjpXJ6Z4WkT4satrUl
msVpHEj2zAFQ8joAzDHBPEm7POWBxzmtqFTk5rkYqiq8afJtf8DrtG0m10LSLfTLIMLeBcLvbLEk
kkk+pJJ9OeMV4r8TNbGreLJYInY29iPs6jJwXB+c4PQ5+X32ivUPHPipPDGikxlvt90GS2AAO0gc
uc8YXI47kgdMkeY/DvwkfEGq/bLpSNOs3VmBTImfqE54x3PXjA75rOP8zMMfLnccLS/4Y9Y8Gaa+
k+D9NtJdwlEXmOGTYylyXKkeo3Y/DtXD+HJF8b/FK91tk8zT9NQLbZwRnJEZwQDz87jjIOOeOeq+
I+ttofg26aNSZrs/ZIzjhS4OSeQR8obB9cVm+AdG1rQPCtv9nSwu1vsXZjlleEw7lXA3BG3ZAHZc
HPLZ4uGkHLq9P8zoml7SFFbR1f6G14714eHvCd3cqzLcTD7Pb7SQRIwPIIBxgAt/wHGea4O1juPA
fwpnvGR4dV1eQIpAw0KkHbkhsghQ7A9QWAI4NdsPCB1TV01XxLPFqEkcYWCyWMi3tycbsAk7ySOp
xn04AFjxr4Y/4Szw+1gk/kzxyCaFj90uAQA3fBDHp04POMFwnGNo+eo6tKpNyqLdKyX6nO+CvBbH
wlaJf38psrtBcvZWrNEsm9T/AKxwdzfKUG0FV+Xo3JPcaZdWF3p0UumPE9kuY4jCMJhCVwvbAKkD
HHHHFc9Y6J4murCHT9b1Gxgs4disumK6STooxtZyRsBIGdoGRkAgVsa3pct74autK05oLUywfZ0y
nyIh+UgAdPlyB6cVE3zS1ZrRjyQ92Oy+bPO9Hhk8f/Em61lwx0awUwxHHyyryAh6EhtzMQQeDtPB
FX/iHeTa9rumeCtPm2vcSCW8ZSDsUZIBG4ZwoZyp64TFdb4Q8OJ4W8PxacJFll3NJNKqkB3PsSeg
AH4ZrA0vwLqcXjHV9bvtXdIrtnWNLaQ+Y0RcEIzMMoAqqPk59GGOdOePNfotjB0Z+zUWtZO8v6/A
6/SWsTpkEemsDZwA28eCSAIyUIyeTgqRnvjPNFWooo4IUhhjSOKNQqIgwqgcAADoKK53ud6VlYfU
VzbrdW7RPxnBVtoJRgcqwDAjIIBGQeQKlopDavocJJpkegy3E03h69cylcz+H5pIlkxkDdAsilCB
jpuHJOeafd+Nb6DTxFpXhPXpJ1XYhvLZwoGOGJyzMc464zzzXcUU79zm+ruOkJW+SPKbL4fa94k1
M6j4vu3jwNnloyF2AHGNuVRcnt154Gc16OI7Tw/ohS0s5Ps1rGdlvbRl3b2UdSxPc9zknqav0UOV
9x0cNCldx3fV6s8X8XXOq+KfE2nTS+GNb/sa0Zd0D2bCRwWBkPA4JAAA3fw5yMmvW9L1A6pYrdGy
u7MMSBFdxhJOO5XJx+P8sGrtFXOopJK2wUqLhJycr3OFa3ufEV/4hludev8ASVsJTBDDFN5Swqqg
+c4GNytyRnHGeehD77xy2mTCxgEWqS2ttFJNOPMX7QzIT8gSNlGRtbJIHzYHQkdBf+GNI1K5nuLm
2fzbhBHOYp5IhKo6BwjAN6c54ovPDOk32RLbyKhhW3McM8kSNGudqlUYAgbj1Hepuupm6NZX5Gr9
9dd/uMfUvFV/KviCHS7SEf2ZarN9plnKt88RdWVPLIJGDwSOg9eMVta1F7IkyXH24+FftRYXpEXX
/WbQmRLjnOevy/7Vdy2h6c8mpSNb5fUoxHdne37xQpUDrxwSOMVXj8L6PEpVbViDZtYfNPIx8gkk
pkt054PUDgcUXQp0a0nfm7/np07HGWniZtChtbq9F9cSJ4fjumQ3zSRyM8wUEhlyGJYZbJ2jKgHA
zrxePWlt2UaYftTXkNpAdzrDI0oOG3OisANrZ+T0xnORuSeFtFmULLYK6izFiAzsR5IIIXr1BAIb
rkdaT/hFdGMUqNau7SypM8zzyNLvThGEhbeCBwMHjJ9TRdCjSxEdFJW/ryMzw1e6jc+LvE0N+Snk
m1CW6zmWOPKNnaSB1wCeBRWzpXh/TdFmuprCBo5Lpg07vM8hcjJySxPPzH86KTOmjGUYWlvr+LuW
7y3luYQkN7PaMGz5kIQkj0+dWGPwzxVL+yb3/oYtT/7923/xmqfi1ovJ0yO41ObTraW82y3EVx5B
A8mUgFvQsFrin8V6pp+r286Xc1xDJYfu2u1KoUa8KJMyfIuTHt+ZigORk84LSMK1eEJWlf7z0H+y
b3/oYtT/AO/dt/8AGaP7Jvf+hi1P/v3bf/Ga5qPxDfag8Dm+jjinh024jgWFoZMvchJCMtkrxg9Q
Qy4Pdnr4r1WOytpbl9PQXMNlMbgwSLFarP5mS43HIBjCg5UZcZIFFmHtqfn9/wDwTov7Jvf+hi1P
/v3bf/GaP7Jvf+hi1P8A7923/wAZrm9D8bXepadby3C2aXMwtmWNARkSXbwuQCxOAqqfYnnqBRoP
ie9fU9F0mdkKTWEDmR1LSys0LOWLMw7p1CuPVgTgFmCrUny2b183/mdJ/ZN7/wBDFqf/AH7tv/jN
H9k3v/Qxan/37tv/AIzXDx+ONX07w+qyfZpp7eK2b7RIpO9JLZ5AGDOMvujwW3c7uFzgHZsvEOoa
nr2nxT7bIC7CyWGxvNUGzaT94wbGNxOFKgnbnqpALMUa9KVkr39X/mb/APZN7/0MWp/9+7b/AOM0
f2Te/wDQxan/AN+7b/4zXJ+J2u5dQRY9T1C2VtfgsiLe4ZB5ckERYce4JHoWPrWl4yup9MtrOVNQ
aGG3ilkZWuzA8zoF2gPsYO33vkbhiehAOCw3ViuZtP3fNm1/ZN7/ANDFqf8A37tv/jNH9k3v/Qxa
n/37tv8A4zWM3iq++33SQ28cxiGRp3lkXSr9mEwZsMcAuRH93qRyTwY7fxXfzW0MjraRIXlUTyJl
LoqIyscflu4DNvdRtaQ5jb5ScqCzH7Wl5/e/8zd/sm9/6GLU/wDv3bf/ABmj+yb3/oYtT/7923/x
mub/AOEqurO+ttPR0d5L91cz4YtG95JEAuZFb5VU9FcDjOBVYeItQ02y06YXKv52n2HmPdMGUFku
GZvmkRdxKKCSwz7nAosyfb0/P73/AJnb2VpPa7/O1G5vN2MeesY2fTYi9ffPSis/QtUm1K+1QSyb
UjkhMVrJGFmgV4Uf5+e5LdRwQ3J6AqWdVNpxujbooooLCqt/ptpqcCw3kPmKjrIhDFWRx0ZWBBUj
1BBoooE0mrMmhgitoEggjSOKNQqIgwFA6ACpKKKBhRRRQAUUUUARxwRQyTPGgVpn3yEfxNtC5P4K
B+FSUUUBYKKKKACiiigD/9k--=_related 005D7CF1872584BA_=--
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic