[prev in list] [next in list] [prev in thread] [next in thread]
List: afripv6-discuss
Subject: =?windows-1252?Q?Re=3A_=5Bafripv6-discuss=5D_IPv6_rollout=85?=
From: Andrew Alston <alston.networks () gmail ! com>
Date: 2012-07-31 19:55:22
Message-ID: 4059366D-34BE-4484-8583-CE4A957D7ABC () gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hi Guys,
Ok a bit more info now that I have time to sit down and write, sorry things have been \
rather hectic.
Here is how this came about, and a bit more of the full story.
The university in question was running a network without a real topology, in essence, \
it was a flat network, v4 only, and a massive one at that. This was causing REAL \
issues, and it was the result of years and years of legacy. The decision was taken \
that IPv6 would be required, but, simply put, we had to fix the network first. So \
first step, how do you migrate from a flat network running a single /16 flat, to a \
segmented network, and do it on a live campus environment that has 38 thousand users \
on it every day using the network? The answer... very very carefully, with a lot of \
planning, and some very careful segment and IP planning.
So, the following decisions were made:
A.) We get rid of NAT in entirety - if we were going to do this, we would do it \
properly, and the dual stack would be on live IPs in identical topology v4 and v6 \
network wide. B.) We divide the network into a three tier network, core, \
distribution and edge. C.) Core/Distribution would be routed, would have to be \
scalable, and we'd use SP style protocols to do this. D.) Edge would remain layer 2, \
but we would choose not to span any L2 across distributions. Since Core/Distribution \
in future will be MPLS enabled, if we need L2 between to points, we can EoMPLS it.
So we did our planning, and discovered (to our horror), that changing the topology, \
eliminating NAT, and rolling out the wireless infrastructure we had planned, we'd \
need a LOT more IPv4 space. So, we applied and were granted a second /15 PI space \
from AfriNIC. We then started implementation.
First step: Pick a network segment - we chose the student residences (not because we \
hate the students, but because we wouldn't break any critical research if it all went \
horribly wrong). Then, we moved all the student residences between a single \
distribution, our residence distribution. At this point, Vlan 1 was still spanned to \
the residence distribution, so NOTHING had broken in doing this, all was working. \
Then, we created a point to point link between that distribution and the core. On \
the point to point, we put a /30 v4 and a /126 v6. (Sadly, the gear we are using \
doesn't support either /31 or /127).
Then, we enabled ospf for v4 and ospf3 for v6 across the point to point. There is a \
slight difference in the topologies at this point because the ospf for v4 was \
configured to ONLY carry the point to point and the loopbacks from the distributions, \
the rest is covered by iBGP, where as in v6, the hardware didn't do ipv6 bgp, so we \
had to do full route distribution of v6 in OSPF3.
Note at this point, we had still had zero downtime.
Then, for each residence between the distribution, we created a vlan, and assigned it \
an IP segment and an IPv6 segment. To avoid mass routes in our routing tables, these \
segments were all taken out of supernets we had dedicated to each distribution, and \
the supernets were what we pushed into the routing table on both v4 and v6 level. \
So, the vlans were now created on the distribution, the routing was working. Fixed \
up the DHCP for v4 as well, so that was in place and ready to go with the correct \
scopes. (Note, we are using RA for v6 at this point, we haven't gotten around to \
DHCPv6 yet, so most people are still hitting the DNS servers on v4 addresses, since \
we can't push DNS via RA).
Note: Still ZERO downtime to anyone
Then, we took the created vlan's on the distribution, trunked them down to the edge \
switches, waited till after hours, and moved the edge ports into the correct vlan's. \
(Different vlans for student pcs and wireless aps etc). The actual move into the \
correct vlans was like, a single command on each switch. Then simple forced a port \
flap one very port as we went. The port flap was to force a DHCP reallocation on v4.
Bang, the residences came up on the new topology with v4 and v6 - total downtime to \
the clients - less than 30 seconds.
Then, we did a rinse and repeat job through the various distributions (we're still \
busy doing some of them, 6 outta 11 done so far, and probably around 300 or 400 edge \
switches tagged correctly).
Once we were sure the topology was working, and the IPv6 was working, next step was \
to enable the ipv6 on the proxy servers, so that they could fetch via IPv6. We did \
this, and instantly saw around 30% of the traffic coming in via v6, primarily google, \
youtube, facebook and akamai. Note however, at this point the clients were still \
seeing the proxies via v4, though the proxies were fetching via v6. So next step, \
put in quad-a records for the proxy servers and for the pac file round robin. \
Suddenly, everyone who had a v6 address was fetching from the proxy servers via v6, \
irrespective of if the proxies were fetching v4 or v6.
Suddenly, we had a situation where 50% of the traffic coming in was via IPv6, and we \
infact peaked at well over 100mbit of IPv6 traffic today coming in off the Internet. \
Our next steps of course are to migrate the rest of the distributions and edge to the \
new network, and infact in the next 10 minutes we'll be moving another thousand edge \
ports into this. Once this is done, we'll start looking closely at the server \
infrastructure and how we go about putting the rest of the production servers both \
into the new topology and IPv6 enabling them. We expect this to be the most \
problematic part, since we know there are certain services which have issues with \
IPv6, but we'll work around those when we get there.
In summary - it is entirely possible to take a network with around 15 thousand wired \
network points, a few hundred wireless access points, a few thousand VOIP phones and \
completely redeploy it both on a v4 and a v6 level with almost no downtime if the \
planning is correct. The traffic levels also prove, there is IPv6 content out there, \
lots of it, and we're happy to use it! It just takes some planning, some forethought \
and some people willing to work really hard at strange hours getting it done.
For interests sake, graphs can be seen here:
http://graphs.tenet.ac.za/iris/browser/browse?username=UFS&selectedmnemonicgroup=TSN81
The graph marked vl1081 is the IPv4 interface to TENET (The South African NREN), the \
graph marked vl3081 is the IPv4 interface to TENET. We specifically asked them to \
provide v4 and v6 on separate interfaces as it did allow for us to see the traffic on \
a more individual basis as well, which was useful.
Hope this answers some of the questions I have been sent off list and provides hope \
for those who believe that IPv6 migration is impossible - never forget - we did it on \
both v4 and v6 *at the same time*, on a live network, with no downtime, so if anyone \
doubts it can be done, we're proof that it can.
Thanks
Andrew Alston
Network Consultant
NOTE: I write the above as a private individual and private consultant and have \
gained specific permission from my client (The University of the Free State) to relay \
this story. I would like to say a special thank you to them for allowing me to share \
this with you as well.
On 31 Jul 2012, at 3:58 PM, Maye diop <mayediop@gmail.com> wrote:
> Dear Andrew,
> Congratulations.
> I look forward more details to share with my universities.
> Best Regards.
> Le 31 juil. 2012 07:11, "Andrew Alston" <alston.networks@gmail.com> a écrit :
> >
> > Hi Guys,
> >
> > So, while i'll be sending out a lot more data soon, with a lot more information \
> > on exactly what we did and how we did it etc, I thought I would share some news \
> > that I for one found rather exciting.
> > Yesterday evening starting at around 7pm one of the South African universities \
> > turned up IPv6, in a fairly consistent manner. Now, I'm not talking about \
> > turning up IPv6 on a few servers, I'm talking about integrating it into every \
> > part of their network. By 2:30am this morning it was running on all their proxy \
> > servers, all their residence networks, the wireless networks, all the lab PC's \
> > and a good portion of the staff network. The topology used was identical to that \
> > of the IPv4, and as the rest of the network is migrated to the new IPv4 topology \
> > V6 will be implemented on everything in dual stack along side that as well.
> > Now, here is where things get interesting, another network dual stacked is no \
> > real news, so lets talk about traffic levels.
> > The University in question is now running anywhere between 30 to 50 percent of \
> > its internet traffic on IPv6, and its working flawlessly so far. So flawlessly \
> > infact that even with Apple's default implementation of Happy Eyeballs that tests \
> > RTT and defaults to v4 if the v6 latency is higher, the apples we tested on \
> > running lion and mountain lion were still choosing ipv6 most of the time.
> > I am not going to say this little rollout has been easy though, we had to \
> > rearchitecture the entire network (that had to happen anyway for various \
> > reasons), and we added the v6 as part of that project. It would not have been \
> > possible to do that without first getting our hands on another /15 worth of IPv4 \
> > space though to allow that rearchitecturing to happen properly.
> > As I said though, in coming days we'll write up what we did with a lot more \
> > detail and send through some graphs and other information, I just had to share \
> > the fact that we're seeing at points half the traffic on a standard university \
> > coming in from the internet over ipv6!
> > Thanks
> >
> > Andrew Alston
> > Network Consultant_______________________________________________
> > afripv6-discuss mailing list
> > afripv6-discuss@afrinic.net
> > https://lists.afrinic.net/mailman/listinfo.cgi/afripv6-discuss
> _______________________________________________
> afripv6-discuss mailing list
> afripv6-discuss@afrinic.net
> https://lists.afrinic.net/mailman/listinfo.cgi/afripv6-discuss
[Attachment #5 (unknown)]
<html><head><meta http-equiv="Content-Type" content="text/html \
charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: \
space; -webkit-line-break: after-white-space; ">Hi Guys,<div><br></div><div>Ok a bit \
more info now that I have time to sit down and write, sorry things have been rather \
hectic.</div><div><br></div><div>Here is how this came about, and a bit more of the \
full story.</div><div><br></div><div>The university in question was running a network \
without a real topology, in essence, it was a flat network, v4 only, and a massive \
one at that. This was causing REAL issues, and it was the result of years and \
years of legacy. The decision was taken that IPv6 would be required, \
but, simply put, we had to fix the network first. So first step, how do you \
migrate from a flat network running a single /16 flat, to a segmented network, and do \
it on a live campus environment that has 38 thousand users on it every day using the \
network? The answer... very very carefully, with a lot of planning, and some \
very careful segment and IP planning.</div><div><br></div><div>So, the following \
decisions were made:</div><div><br></div><div>A.) We get rid of NAT in entirety - if \
we were going to do this, we would do it properly, and the dual stack would be on \
live IPs in identical topology v4 and v6 network wide. </div><div>B.) We divide \
the network into a three tier network, core, distribution and \
edge. </div><div>C.) Core/Distribution would be routed, would have to be \
scalable, and we'd use SP style protocols to do this.</div><div>D.) Edge would remain \
layer 2, but we would choose not to span any L2 across distributions. Since \
Core/Distribution in future will be MPLS enabled, if we need L2 between to points, we \
can EoMPLS it.</div><div><br></div><div>So we did our planning, and discovered (to \
our horror), that changing the topology, eliminating NAT, and rolling out the \
wireless infrastructure we had planned, we'd need a LOT more IPv4 space. So, we \
applied and were granted a second /15 PI space from AfriNIC. We then started \
implementation.</div><div><br></div><div>First step: Pick a network segment - \
we chose the student residences (not because we hate the students, but because we \
wouldn't break any critical research if it all went horribly wrong). Then, we \
moved all the student residences between a single distribution, our residence \
distribution. At this point, Vlan 1 was still spanned to the residence \
distribution, so NOTHING had broken in doing this, all was working. Then, we \
created a point to point link between that distribution and the core. On the \
point to point, we put a /30 v4 and a /126 v6. (Sadly, the gear we are using \
doesn't support either /31 or /127).</div><div><br></div><div>Then, we enabled ospf \
for v4 and ospf3 for v6 across the point to point. There is a slight difference \
in the topologies at this point because the ospf for v4 was configured to ONLY carry \
the point to point and the loopbacks from the distributions, the rest is covered by \
iBGP, where as in v6, the hardware didn't do ipv6 bgp, so we had to do full route \
distribution of v6 in OSPF3. </div><div><br></div><div>Note at this point, we \
had still had zero downtime.</div><div><br></div><div>Then, for each residence \
between the distribution, we created a vlan, and assigned it an IP segment and an \
IPv6 segment. To avoid mass routes in our routing tables, these segments were \
all taken out of supernets we had dedicated to each distribution, and the supernets \
were what we pushed into the routing table on both v4 and v6 level. So, the \
vlans were now created on the distribution, the routing was working. Fixed up \
the DHCP for v4 as well, so that was in place and ready to go with the correct \
scopes. (Note, we are using RA for v6 at this point, we haven't gotten around \
to DHCPv6 yet, so most people are still hitting the DNS servers on v4 addresses, \
since we can't push DNS via RA). </div><div><br></div><div>Note: Still \
ZERO downtime to anyone</div><div><br></div><div>Then, we took the created vlan's on \
the distribution, trunked them down to the edge switches, waited till after hours, \
and moved the edge ports into the correct vlan's. (Different vlans for student \
pcs and wireless aps etc). The actual move into the correct vlans was like, a \
single command on each switch. Then simple forced a port flap one very port as \
we went. The port flap was to force a DHCP reallocation on \
v4.</div><div><br></div><div>Bang, the residences came up on the new topology with v4 \
and v6 - total downtime to the clients - less than 30 \
seconds.</div><div><br></div><div>Then, we did a rinse and repeat job through the \
various distributions (we're still busy doing some of them, 6 outta 11 done so far, \
and probably around 300 or 400 edge switches tagged \
correctly).</div><div><br></div><div>Once we were sure the topology was working, and \
the IPv6 was working, next step was to enable the ipv6 on the proxy servers, so that \
they could fetch via IPv6. We did this, and instantly saw around 30% of the \
traffic coming in via v6, primarily google, youtube, facebook and akamai. Note \
however, at this point the clients were still seeing the proxies via v4, though the \
proxies were fetching via v6. So next step, put in quad-a records for the proxy \
servers and for the pac file round robin. Suddenly, everyone who had a v6 \
address was fetching from the proxy servers via v6, irrespective of if the proxies \
were fetching v4 or v6.</div><div><br></div><div>Suddenly, we had a situation where \
50% of the traffic coming in was via IPv6, and we infact peaked at well over 100mbit \
of IPv6 traffic today coming in off the Internet. </div><div><br></div><div>Our \
next steps of course are to migrate the rest of the distributions and edge to the new \
network, and infact in the next 10 minutes we'll be moving another thousand edge \
ports into this. Once this is done, we'll start looking closely at the server \
infrastructure and how we go about putting the rest of the production servers both \
into the new topology and IPv6 enabling them. We expect this to be the most \
problematic part, since we know there are certain services which have issues with \
IPv6, but we'll work around those when we get there.</div><div><br></div><div>In \
summary - it is entirely possible to take a network with around 15 thousand wired \
network points, a few hundred wireless access points, a few thousand VOIP phones and \
completely redeploy it both on a v4 and a v6 level with almost no downtime if the \
planning is correct. The traffic levels also prove, there is IPv6 content out \
there, lots of it, and we're happy to use it! It just takes some planning, some \
forethought and some people willing to work really hard at strange hours getting it \
done.</div><div><br></div><div>For interests sake, graphs can be seen \
here:</div><div><br></div><div><a \
href="http://graphs.tenet.ac.za/iris/browser/browse?username=UFS&selectedmnemonicg \
roup=TSN81">http://graphs.tenet.ac.za/iris/browser/browse?username=UFS&selectedmnemonicgroup=TSN81</a></div><div><br></div><div>The \
graph marked vl1081 is the IPv4 interface to TENET (The South African NREN), the \
graph marked vl3081 is the IPv4 interface to TENET. We specifically asked them \
to provide v4 and v6 on separate interfaces as it did allow for us to see the traffic \
on a more individual basis as well, which was useful.</div><div><br></div><div>Hope \
this answers some of the questions I have been sent off list and provides hope for \
those who believe that IPv6 migration is impossible - never forget - we did it on \
both v4 and v6 *at the same time*, on a live network, with no downtime, so if anyone \
doubts it can be done, we're proof that it \
can.</div><div><br></div><div>Thanks</div><div><br></div><div>Andrew \
Alston</div><div>Network Consultant</div><div><br></div><div>NOTE: I write the above \
as a private individual and private consultant and have gained specific permission \
from my client (The University of the Free State) to relay this story. I would \
like to say a special thank you to them for allowing me to share this with you as \
well.</div><div><br></div><div><br></div><div><div><div>On 31 Jul 2012, at 3:58 PM, \
Maye diop <<a href="mailto:mayediop@gmail.com">mayediop@gmail.com</a>> \
wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><p>Dear \
Andrew,<br> Congratulations.<br>
I look forward more details to share with my universities.<br>
Best Regards.<br>
Le 31 juil. 2012 07:11, "Andrew Alston" <<a \
href="mailto:alston.networks@gmail.com">alston.networks@gmail.com</a>> a \
écrit :<br> ><br>
> Hi Guys,<br>
><br>
> So, while i'll be sending out a lot more data soon, with a lot more information \
on exactly what we did and how we did it etc, I thought I would share some news \
that I for one found rather exciting.<br> ><br>
> Yesterday evening starting at around 7pm one of the South African universities \
turned up IPv6, in a fairly consistent manner. Now, I'm not talking about \
turning up IPv6 on a few servers, I'm talking about integrating it into every part of \
their network. By 2:30am this morning it was running on all their proxy \
servers, all their residence networks, the wireless networks, all the lab PC's and a \
good portion of the staff network. The topology used was identical to that of \
the IPv4, and as the rest of the network is migrated to the new IPv4 topology V6 will \
be implemented on everything in dual stack along side that as well.<br>
><br>
> Now, here is where things get interesting, another network dual stacked is no \
real news, so lets talk about traffic levels.<br> ><br>
> The University in question is now running anywhere between 30 to 50 percent of \
its internet traffic on IPv6, and its working flawlessly so far. So flawlessly \
infact that even with Apple's default implementation of Happy Eyeballs that tests RTT \
and defaults to v4 if the v6 latency is higher, the apples we tested on running lion \
and mountain lion were still choosing ipv6 most of the time.<br>
><br>
> I am not going to say this little rollout has been easy though, we had to \
rearchitecture the entire network (that had to happen anyway for various reasons), \
and we added the v6 as part of that project. It would not have been possible to \
do that without first getting our hands on another /15 worth of IPv4 space though to \
allow that rearchitecturing to happen properly.<br>
><br>
> As I said though, in coming days we'll write up what we did with a lot more \
detail and send through some graphs and other information, I just had to share the \
fact that we're seeing at points half the traffic on a standard university coming in \
from the internet over ipv6!<br>
><br>
> Thanks<br>
><br>
> Andrew Alston<br>
> Network Consultant_______________________________________________<br>
> afripv6-discuss mailing list<br>
> <a href="mailto:afripv6-discuss@afrinic.net">afripv6-discuss@afrinic.net</a><br>
> <a href="https://lists.afrinic.net/mailman/listinfo.cgi/afripv6-discuss">https://lists.afrinic.net/mailman/listinfo.cgi/afripv6-discuss</a><br>
</p>
_______________________________________________<br>afripv6-discuss mailing list<br><a \
href="mailto:afripv6-discuss@afrinic.net">afripv6-discuss@afrinic.net</a><br>https://l \
ists.afrinic.net/mailman/listinfo.cgi/afripv6-discuss<br></blockquote></div><br></div></body></html>
_______________________________________________
afripv6-discuss mailing list
afripv6-discuss@afrinic.net
https://lists.afrinic.net/mailman/listinfo.cgi/afripv6-discuss
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic