User comments to blog posts.
User avatar
Steve
Site Admin
Posts:
10617
Joined:
Mon Dec 06, 2004 6:46 pm
Certs:
CCNA

Upgrading with ISSU

Mon May 14, 2012 11:23 am

Comments for Upgrading with ISSU.

User avatar
cjutting
Post Whore
Posts:
1063
Joined:
Wed Sep 16, 2009 3:16 pm

Re: Upgrading with ISSU

Mon May 14, 2012 2:16 pm

That's a really slick way of doing it.

Just out of curiosity... did you do this during the day or was this done after hours?

User avatar
routerdork
Post Whore
Posts:
1505
Joined:
Tue Feb 22, 2011 9:40 am
Certs:
CCNA, MCDST, MCP, A+

Re: Upgrading with ISSU

Mon May 14, 2012 2:30 pm

Nice job!!! I did some testing on this with the 7600 series a few months back. It was not as disruptive as a sup failover in terms of time but I believe I still did see packets and BGP peering drop.

Also on the note of time. The abort timer is defaulted at 45mins. You can change this with the command "configure issu set rollback timer <secs>" and then verify with "show issu rolback timer"

User avatar
Project2501
Post Whore
Posts:
6158
Joined:
Thu Apr 17, 2008 6:44 pm
Certs:
CCNA

Re: Upgrading with ISSU

Mon May 14, 2012 6:43 pm

This is a very cool blog post. Nice work killabee.

Does this have the same limitation Juniper ISSU has were official you can't upgrade more than 3 versions at a time?
- Pete

Otanx
Post Whore
Posts:
1181
Joined:
Wed Sep 01, 2010 3:37 pm
Certs:
CCNP, CEH

Re: Upgrading with ISSU

Mon May 14, 2012 8:16 pm

Good post. Been reading on this recently as we are going to be doing some upgrades soon.

-Otanx
Stay networked, my friends.

User avatar
baybars
Ultimate Member
Posts:
683
Joined:
Fri Mar 13, 2009 3:53 pm
Certs:
CCNA, CCNA Security, 642-902

Re: Upgrading with ISSU

Mon May 14, 2012 8:31 pm

Saved your blog for future use :) , nice work...

killabee
Post Whore
Posts:
1430
Joined:
Sat Dec 19, 2009 11:52 pm
Certs:
CCNP, CCDA, JNCIA

Re: Upgrading with ISSU

Mon May 14, 2012 10:02 pm

Thanks guys!

Although it's designed to be non-disruptive, I still did it after hours just to be extra cautious. Only part to watch out for is if your config register isn't already set to 0x2102. When you change it, be sure that the 'boot system' config in your running config is set correctly, or else you'll end up in rommon after you reboot.

It has limitations on what IOS pairs you can upgrade/downgrade in terms of features and releases. Minor releases between IOSs should be possible while major releases and different trains between IOSs may not be possible. I don't think there's a limitation on how many times you can upgrade. The only remnant config that remains is the boot system config.

User avatar
ristau5741
Post Whore
Posts:
10239
Joined:
Tue Aug 21, 2007 2:15 pm
Certs:
Instanity

Re: Upgrading with ISSU

Mon Mar 11, 2013 10:54 am

I need to figure this out for a 6500 VSS pair,maybe even get a blog post from it.
gotta do one tomorrow morning.
Tips of the day:
- The human mind is the ultimate creation invention.
- I have so many customers, my customers have customers.
- Sausage time
- POP, stack, and store

User avatar
wirerat
Post Whore
Posts:
5322
Joined:
Tue Mar 31, 2009 4:15 pm
Certs:
More than none

Re: Upgrading with ISSU

Mon Mar 11, 2013 11:17 am

ristau5741 wrote:I need to figure this out for a 6500 VSS pair,maybe even get a blog post from it.
gotta do one tomorrow morning.

There are some limitations. Check out compatibility matrix before doing anything. Our current version to the version being installed is too far apart so we have to do it the old fashion way. Doing my second of three VSS pairs sometime this week.

http://www.cisco.com/en/US/docs/switche ... #wp1172503
"See packet, be packet, you are packet. Ignore all else!" -The Networker
packetsdropped.wordpress.com

User avatar
mlan
Ultimate Member
Posts:
792
Joined:
Thu Nov 17, 2011 6:09 pm

Re: Upgrading with ISSU

Tue Mar 12, 2013 11:03 am

Nice article, thanks for posting it.

User avatar
ristau5741
Post Whore
Posts:
10239
Joined:
Tue Aug 21, 2007 2:15 pm
Certs:
Instanity

Re: Upgrading with ISSU

Tue Mar 12, 2013 2:04 pm

Ran out of time, never got to it, will need to reschedule. :(
Tips of the day:
- The human mind is the ultimate creation invention.
- I have so many customers, my customers have customers.
- Sausage time
- POP, stack, and store

User avatar
williamtyrell78
Post Whore
Posts:
1337
Joined:
Tue Mar 12, 2013 3:58 pm
Certs:
CompTIA Net+, CCENT, CCNA R&S

Re: Upgrading with ISSU

Thu May 30, 2013 12:41 pm

i tried few commands in a few of my 2900 series switches. the commands weren't available. Does anyone know a link for ISSU supported iOS/models?


This is pretty freaking cool
+1+1+1+1 to blog
"""Emergency D-Link. $40........................Boom."""
------------------------------------------------------------
daima ni kosa mtumiaji

User avatar
burnyd
Post Whore
Posts:
3116
Joined:
Fri Nov 13, 2009 5:15 pm
Certs:
CCIE R&S/SP,CCNP-SP,JNCIA,VCP510,VCA-DCV

Re: Upgrading with ISSU

Thu May 30, 2013 1:12 pm

its not unless you have 4500X's I believe is the only one that support ISSU that isnt a NX-OS chasis.
http://danielhertzberg.wordpress.com - I blog about networks!

User avatar
srg
Post Whore
Posts:
1688
Joined:
Thu Dec 30, 2010 2:05 pm
Certs:
CCIE SP, CCNP SP, CCNP, CCDA, CCNA DC/W, HP MASE

Re: Upgrading with ISSU

Fri May 31, 2013 7:21 am

There are a few platforms, but generally with devices running classic monolithic IOS you'll need dual supervisors.
som om sinnet hade svartnat för evigt.

killabee
Post Whore
Posts:
1430
Joined:
Sat Dec 19, 2009 11:52 pm
Certs:
CCNP, CCDA, JNCIA

Re: Upgrading with ISSU

Sat Jun 01, 2013 5:54 pm

williamtyrell78 wrote:Does anyone know a link for ISSU supported iOS/models?


I tried finding one but couldn't. To help me remember, I like to think that Cisco made the feature available mostly for the boxes that need maximum uptime (e.g. ASRs, Nexus, 4500s, 4500Xs, 6500s, 10000s, etc). You're better off using the Cisco Feature Navigator to see if your IOS supports it, though.

User avatar
williamtyrell78
Post Whore
Posts:
1337
Joined:
Tue Mar 12, 2013 3:58 pm
Certs:
CompTIA Net+, CCENT, CCNA R&S

Re: Upgrading with ISSU

Mon Jun 03, 2013 8:14 am

killabee wrote:
williamtyrell78 wrote:Does anyone know a link for ISSU supported iOS/models?


I tried finding one but couldn't. To help me remember, I like to think that Cisco made the feature available mostly for the boxes that need maximum uptime (e.g. ASRs, Nexus, 4500s, 4500Xs, 6500s, 10000s, etc). You're better off using the Cisco Feature Navigator to see if your IOS supports it, though.




thank you!!! Answered my question killa...
"""Emergency D-Link. $40........................Boom."""
------------------------------------------------------------
daima ni kosa mtumiaji

seguse
New Member
Posts:
5
Joined:
Wed Aug 21, 2013 12:32 am
Certs:
A+, Net+, Security+, Server+, CCNA

Re: Upgrading with ISSU

Wed Aug 21, 2013 1:34 am

killabee wrote:Thanks guys!

Although it's designed to be non-disruptive, I still did it after hours just to be extra cautious. Only part to watch out for is if your config register isn't already set to 0x2102. When you change it, be sure that the 'boot system' config in your running config is set correctly, or else you'll end up in rommon after you reboot.

It has limitations on what IOS pairs you can upgrade/downgrade in terms of features and releases. Minor releases between IOSs should be possible while major releases and different trains between IOSs may not be possible. I don't think there's a limitation on how many times you can upgrade. The only remnant config that remains is the boot system config.

I'm in the process of doing the ISSU (in-service software upgrade) on my pair of 4500-X vss switches. The config-register was set to 0x2101 out of the box and one of the issu prerequisites is your config-register has to be set to 0x2102. I set the config-register to 0x2102 then reloaded the standby unit (redundancy reload peer) and after it came back online I did a "show redundancy" and it still showed "Configuration register = 0x2101 (will be 0x2102 at next reload)". I was hoping I would be able to do the same thing on the active unit after I failed the active unit to the stand by unit (redundancy force-switchover). I guess I have to to issue a reload command from the primary unit so both switches reboot for the new config-register to take affect. This didn't work for me so I am guessing it is not possible...which sucks!

Also something I learned while doing this on the 4500-X switches in vss. When you get ready to issue your first command of "issu loadversion" and add the Active RP slot number and Standby RP slot number...

4500X-SW#issu loadversion 1 bootflash:cat4500e-universalk9.SPA.03.04.01.SG.151-2.SG1.bin 2 slavebootflash:cat4500e-universalk9.SPA.03.04.01.SG.151-2.SG1.bin

I got the following error...
%% Standby unit ID [ 2 ] is wrong; expected [ 11 ]

I've found that the slot numbers under the "show redundancy" command are incorrect ones to use. On my active unit it shows the slot number as 1/1 and on my standby unit it shows up as 2/1.

4500X-SW#show redundancy
[text omitted]
Current Processor Information :
------------------------------
Active Location = slot 1/1
Current Software state = ACTIVE
Uptime in current state = 1 day, 23 hours, 43 minutes
Image Version = Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.04.00.SG RELEASE SOFTWARE (fc3)
[text omitted]
BOOT = bootflash:cat4500e-universalk9.SPA.03.04.00.SG.151-2.SG.bin,1;
Configuration register = 0x2101 (will be 0x2102 at next reload)

Peer Processor Information :
------------------------------
Standby Location = slot 2/1
Current Software state = STANDBY HOT
Uptime in current state = 44 minutes
Image Version = Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.04.00.SG RELEASE SOFTWARE (fc3)
[text omitted]
BOOT = bootflash:cat4500e-universalk9.SPA.03.04.00.SG.151-2.SG.bin,1;
Configuration register = 0x2101 (will be 0x2102 at next reload)

The command I had to use was "show issu state" which gave the following...

4500X-SW#show issu state
Slot = 1
RP State = Active
[text omitted]

Slot = 11
RP State = Standby
[text omitted]

When I changed the issu loadversion command to:

4500X-SW#issu loadversion 1 bootflash:cat4500e-universalk9.SPA.03.04.01.SG.151-2.SG1.bin 11 slavebootflash:cat4500e-universalk9.SPA.03.04.01.SG.151-2.SG1.bin

then it would work just fine I bet. I am still getting an error about the config-register so I need to fix that before I can proceed.

The reason I am upgrading from IOS XE cat4500e-universalk9.SPA.03.04.00.SG.151-2.SG.bin to cat4500e-universalk9.SPA.03.04.01.SG.151-2.SG1.bin is because of the previous bug...

http://www.cisco.com/en/US/docs/switches/lan/catalyst4500/release/note/OL_27989-01.html#wp2694682
Resolved Caveats for Cisco IOS XE Release 3.4.1SG
A switch running IOS Release XE 3.4.0SG loses all Layer 3 connectivity to or from the switch IP address.
Switching is unaffected, but routed IP traffic (snmp, ntp, telnet, ssh, etc.) is affected.
Workaround: Once the problem occurs, reboot the switch. Disabling Fa1 prevents the problem. CSCue76243

This happened to me twice in production. The first time it happened, I did not know about the above bug so I proceeded to troubleshoot and everything appeared to be working just fine but all layer 3 connectivity would not work. All interfaces were good, routing table was good, spanning-tree was good, cpu utilization low, etc...the only thing I could think of is to pull the power on the primary switch and everything failed over and started working. The second time it happened I proceed with the same procedure and it fixed the issue once again. I saw there was a new IOS XE version and found that bug listed as being fixed in 03.04.01.SG.151-2.SG1.

Hopefuly this will help someone else out there and thanks for the guide killabee

Update: Well, I issued the "reload" command on the primary unit and I thought both switches were going to reboot at the same time but that did not happen...only the primary unit rebooted. When I went on the standby unit which now turned into the active unit...I issued the "show redundancy" command and now the config-register shows up as 0x2102 and no reboot required. I am not sure if this is because I already reloaded the standby unit? Very strange. I would have thought the standby unit would have to reboot also. Regardless, I issued a redundancy force-switchover on the standby unit which currently was active and that triggered it to reboot itself after the failover happened.

Update 2:

Cisco Catalyst 4500 ISSU Deployment Guide: http://www.cisco.com/en/US/prod/collate ... e6a95.html

I proceeded doing the issu upgrade. I thought I might need to add another boot system flash bootflash:[newer ios image here] before I executed the "issu loadversion" command but Cisco's documentation says nothing about that. The standby unit did load up with the newer ios version without adding the extra command.

The upgrade has been successful. Sometimes it gets confusing when the active becomes standby and standby becomes active and your slots get all moved around. If you ready everything carefully it should work perfectly.

After the update I looked at the boot section and it appears the ISSU process automatically added the newer IOS XE image...
!
boot-start-marker
boot system bootflash:cat4500e-universalk9.SPA.03.04.01.SG.151-2.SG1.bin
boot system flash bootflash:cat4500e-universalk9.SPA.03.04.00.SG.151-2.SG.bin
boot-end-marker
!

I also issued a redundancy force-switchover command just to give my primary unit the active role again.

seguse
New Member
Posts:
5
Joined:
Wed Aug 21, 2013 12:32 am
Certs:
A+, Net+, Security+, Server+, CCNA

Re: Upgrading with ISSU

Sat Aug 24, 2013 9:27 pm

So I decided to do an ISSU upgrade on our 4507R+E's vss pair since I had success on our 4500-X's.

The config-register was set to 0x2101 as well so I decided to do things a little different this time. I issued a "reload" on the active and it only reloaded the active unit and caused vss fail-over to the standby-hot unit. Once the active (now standby) came back online, the standby (now active) still showed 0x2102 upon reboot. So then I reloaded the standby unit (now active) and once it failed over the config-register changed to 0x2102.

One interesting thing happen during the upgrade process. When my standby unit came back online after the first issu command, my WS-X4712-SFP+E line card had a red status light and would not go green. I issued a show module command and it said the line card was "OK". I issued a show int status command and those interfaces were in an Unknown state. As you can imagine, panic mode started to set in. I then reseated the hot-pluggable line card and after the ios went through its cycles the line card went to a green status and all the interfaces started to come up.

Other than that, it was a successful upgrade.

User avatar
burnyd
Post Whore
Posts:
3116
Joined:
Fri Nov 13, 2009 5:15 pm
Certs:
CCIE R&S/SP,CCNP-SP,JNCIA,VCP510,VCA-DCV

Re: Upgrading with ISSU

Sun Aug 25, 2013 9:34 am

Now that NXOS 6.2.2 is out I tried to upgrade my lab 7k from 6.1.2 to 6.2.2.. did a ISSU said everything was fine no upgrades atm. But it killed my ssh session what happened? For some reason I cannot get into the admin VDC only the VDC's I created. So I have to figure out what happened on Monday.
http://danielhertzberg.wordpress.com - I blog about networks!

User avatar
srg
Post Whore
Posts:
1688
Joined:
Thu Dec 30, 2010 2:05 pm
Certs:
CCIE SP, CCNP SP, CCNP, CCDA, CCNA DC/W, HP MASE

Re: Upgrading with ISSU

Sun Aug 25, 2013 11:24 am

On 5k you always loose mgmt when doing ISSU, forwarding is not disrupted though. Something else did probably happen since you cant access the admin VDC though :)
som om sinnet hade svartnat för evigt.

'
Next

Return to Blog Comments

Who is online

Users browsing this forum: No registered users and 5 guests