All other Cisco networking related discussions.
dlploh04
New Member
Posts:
15
Joined:
Wed Apr 18, 2012 7:25 pm

Cisco Catalyst 6509E vs Nexus 7009

Wed Apr 18, 2012 7:53 pm

Hi - our company is moving into a new building. We are moving from current leased space to this new building.

If you are setting up a 500sqft datacenter for 200 employees and have:

(8) Dell 720 servers with 2 dual port 10gbe nics each (all vmware and some hyperv with about 75-100 total virtual servers) [4 ports 10gbe per server]
nics: broadcom 57810S dual port 10gb nics (nic1 port 1 and nic2 port 1 dedicated for 10gbe iscsi mpio. nic1 port 2 and nic 2 port 2 will use the broadcom NPAR feature to break down into 4 vmnics CSV/LM/VM/MGMT - individual 2.5gbps chunks teamed in active/standby with nic1 port 2 being active and nic2 port 2 being standby)

(5) 10gbe Equallogic sans each having active controller with (1) 10gbe NIC and standby controller with (1) 10gbe NIC

All servers/sans are dual homed into two nexus 5548UP's - this is our server/san access layer - I will have six 10gbe VPC link between the two 5548's and will uplink 20gbe from each 5548UP dual homed to our two core switches.

(3) 4507R+E switches w/ sup 7L-E as workstation/PoE access layer with almost every port being used as PoE as the users plug their workstations into the gig switch on back of their 8945 phone The Sup 7L-E's on each 4507R+E are dual homed uplinked to the two core switches via 10gbe.


What core switches do you go with? Two 6509E with quad Sup2T in VSS Design? or Two 7009 with quad Sup1 in VPC Design?

If 6509:
each 6509E will have two power supplies and two Sup2T supervisors running in quad-sup VSS
each 6509E will have a 6800/6900 16 port 10gbe for the downlinks to the nexus 5548's and catalyst 4507's.
each 6509E will have 48port 1gbe for routers/switches/firewalls that are dual homed into the core.
some appliances do not have multiple nics and are single homed into one of the two core switches.

If 7009:
each 7009 will have the redundant supervisors, five fab2 modules, and two power supplies, running in quad-sup VPC
each 7009 will have a f2 48port 10gbe card for downlinks to the nexus 5548's and catalyst 4507's
each 7009 will have a m1 48port 1gbe card for routers/switches/firewalls that are dual homed into the core.
each 7009 will have LAN Enterprise License (L3 Protocols)
some appliances do not have multiple nics and are single homed into one of the two core switches.

The 6509E I'm thinking would be ok as we only have 200 employees and 8 physical servers with 100 virtual servers within...The 7009 seems to be overkill but the 6509E seems to be EOS/EOL in 5/7 years....I can't find any real guidance from cisco sales partner..I'm told they both would work just fine. Maybe we'll hire another 50 employees over next 5 years adding up to 250 employees and couple more servers here and there adding up to maybe 120 virtual servers but..when I read about Nexus 7009 it presents it as a solution for those that are having hundreds to thousands of servers.

The total difference is about 70-80k upfront (35-40k per chassis) and 10k/year smartnet for both chassis extra between the two core solutions. Not enough to completely rule 7009 out but it does take away money that could be spent elsewhere for capex items such as storage, monitoring, disaster recovery, etc... and I'm not quite sure how it will look if we pay the extra money and when we run utilization monitoring it shows 1% utilization rate...at least that's one of my concerns with the 7009.

The environment is 24x7x365 online and downtime would equate to less profits for the company. All Win7/Office environment - company is into financial trading but not wall street type zero second high-frequency trading...more like daily financial trading via the phone and bloomberg terminals.

Thoughts/Opinions?

Thanks,
Matt
Last edited by dlploh04 on Wed Apr 18, 2012 9:55 pm, edited 1 time in total.

User avatar
that1guy15
Post Whore
Posts:
3213
Joined:
Thu Apr 29, 2010 6:12 pm
Certs:
CCNP, CCDP, CCIP

Re: Cisco Catalyst 6509E vs Nexus 7009

Wed Apr 18, 2012 7:57 pm

http://blog.movingonesandzeros.net/

User avatar
Vito_Corleone
Moderator
Posts:
9847
Joined:
Mon Apr 07, 2008 10:38 am
Certs:
CCNP RS, CCNP DC, CCDP, CCIP

Re: Cisco Catalyst 6509E vs Nexus 7009

Wed Apr 18, 2012 8:06 pm

without reading, 7009
http://blog.alwaysthenetwork.com

dlploh04
New Member
Posts:
15
Joined:
Wed Apr 18, 2012 7:25 pm

Re: Cisco Catalyst 6509E vs Nexus 7009

Wed Apr 18, 2012 8:12 pm

that1guy15/Vito_Corleone,

Both say 7009. If the 80k premium for the 7009 is deemed too much, is there a negative to deleting the server/san access layer (two 5548up's) to reduce capex and dual homing the server/san's into the F2 48port line card module on the 7009 - the same line card that the 4507R+E's are dual homed to.. ?

I was trying to keep atleast a 2-tier model but I read on that other thread that there is no netflow on the 5548up's. Network analysis/monitoring is important.
This would leave 5 available line cards on each 7009, for service modules maybe if they are introduced?

Thoughts?

Thanks again,
Matt

User avatar
that1guy15
Post Whore
Posts:
3213
Joined:
Thu Apr 29, 2010 6:12 pm
Certs:
CCNP, CCDP, CCIP

Re: Cisco Catalyst 6509E vs Nexus 7009

Wed Apr 18, 2012 8:44 pm

i don't have any of the technical specs of the 7009 in front of me but the first thing that comes to mind is 7009 ports are expensive. You might save money now but as you scale you will start chewing up a ton of them. Also keep in mind that once you hit the point of needing to move these ports off the 7009 and back to 5548 or whatever at the tie its going to be painfull.

I am very interested to hear what other who have more DC exposure than I do have to say!
http://blog.movingonesandzeros.net/

User avatar
Vito_Corleone
Moderator
Posts:
9847
Joined:
Mon Apr 07, 2008 10:38 am
Certs:
CCNP RS, CCNP DC, CCDP, CCIP

Re: Cisco Catalyst 6509E vs Nexus 7009

Wed Apr 18, 2012 8:56 pm

I got the same 35-40k difference on list price per chassis. With a discount, you should be able to get that down quite a bit.

The argument for the 7K is 550g/slot to the 6500's 80g/slot. When your 10g needs grow, you will max the 6500 very quickly (as soon as you add another 16x10g card, you're more than the 7009 with 48x10g). Plus the multitude of features, plus avoiding VSS, etc, etc. It's worth the difference.
http://blog.alwaysthenetwork.com

zerojunkie
Senior Member
Posts:
368
Joined:
Mon Jan 26, 2009 5:59 pm

Re: Cisco Catalyst 6509E vs Nexus 7009

Wed Apr 18, 2012 10:08 pm

dlploh04 wrote:Both say 7009. If the 80k premium for the 7009 is deemed too much, is there a negative to deleting the server/san access layer (two 5548up's) to reduce capex and dual homing the server/san's into the F2 48port line card module on the 7009


It certainly depends on your traffic patterns, but is running 2232 FEXs directly into your 7k's an option?

dlploh04
New Member
Posts:
15
Joined:
Wed Apr 18, 2012 7:25 pm

Re: Cisco Catalyst 6509E vs Nexus 7009

Wed Apr 18, 2012 11:49 pm

zerojunkie wrote:
dlploh04 wrote:Both say 7009. If the 80k premium for the 7009 is deemed too much, is there a negative to deleting the server/san access layer (two 5548up's) to reduce capex and dual homing the server/san's into the F2 48port line card module on the 7009


It certainly depends on your traffic patterns, but is running 2232 FEXs directly into your 7k's an option?


It is an option if it does not present extra latency (we have all iSCSI 10gbe san's)? If it is an extension of the 7k, what benefits does it provide over plugging directly into the 32port f2 card? the 2232 fex would have our 10gbe server nics and 10gbe san nics plugged dual homed into two 2232fp's. I would think most traffic is east/west on the fex modules - the north/south would primarily be internet traffic and Outlook 2010 / Office (Excel primarily)/ Adobe applications. If we went the fex route then we would want to go all out with fex and not purchase the m1 48port 10/100/1000 line card as well and use two fex's for our switches/routers/firewalls as well?

Many thanks,
Matt

User avatar
Vito_Corleone
Moderator
Posts:
9847
Joined:
Mon Apr 07, 2008 10:38 am
Certs:
CCNP RS, CCNP DC, CCDP, CCIP

Re: Cisco Catalyst 6509E vs Nexus 7009

Thu Apr 19, 2012 6:20 am

Directly connecting FEXes to the 7K is quirky. Also, the F2 is 48 ports not 32. If you're going with FEXes, connect them to your 5Ks. The latency is low, FYI.

Also, you never answered any of my questions.
http://blog.alwaysthenetwork.com

dlploh04
New Member
Posts:
15
Joined:
Wed Apr 18, 2012 7:25 pm

Re: Cisco Catalyst 6509E vs Nexus 7009

Thu Apr 19, 2012 7:44 am

Vito_Corleone wrote:Directly connecting FEXes to the 7K is quirky. Also, the F2 is 48 ports not 32. If you're going with FEXes, connect them to your 5Ks. The latency is low, FYI.

Also, you never answered any of my questions.



I'm sorry - what question? I relooked and could not find anything. Did I not answer the traffic pattern correctly?

Regards,
Matt

User avatar
Vito_Corleone
Moderator
Posts:
9847
Joined:
Mon Apr 07, 2008 10:38 am
Certs:
CCNP RS, CCNP DC, CCDP, CCIP

Re: Cisco Catalyst 6509E vs Nexus 7009

Thu Apr 19, 2012 7:48 am

Shit, it looks like I didn't submit the post before the quote thing.

I was asking why you need the M1 card and if you realize you can't mix M and F2 cards without putting them into separate VDCs. Will you have more than 48 total (1g and 10g) ports in use? If not, you can run 1g on the F2 just fine.
http://blog.alwaysthenetwork.com

dlploh04
New Member
Posts:
15
Joined:
Wed Apr 18, 2012 7:25 pm

Re: Cisco Catalyst 6509E vs Nexus 7009

Thu Apr 19, 2012 8:06 am

I was told I could have one large VDC with M1 and F2 together (sounds like I was told incorrectly!).

I will not have more than 48 total ports in use currently on a single chassis (I have this many ports cumulative but it will be dual homed so approx half to each chassis), it would be more like 20-30ports per chassis in use initially with more added over the years(I am attempting to dual home everything, though some devices can't be dual homed & I will have it documented which cables to move if a chassis fails).

In this scenario you are suggesting deleting the two M1 card which will reduce costs approximately ~$16k and then also deleting the 5548up's (reducing costs approx ~36k) or keeping them? If deleting them I understand you recommend to not use/connect the fex 2232p's to parent 7009. So if deleting the 5548up's, this would keep 7009 as core for all other switches/routers and would also take the role as server/san switch, all on same f2 48port 1/10gbe line card, just to confirm? Does the 5548up not provide value in the scenario? I'm ok with deleting them as it does present the 7009 as stronger switch that can do it all without creating single points of failure, just was not sure what negatives there were with not having a physical two tier model with a server/san switch to the core switches.


The price I have currently for two 5548up's with 32 10gbe ports, no add-on module installed, with 2 PS, 2 Fans, plus 24x7x4 smartnet is ~36k after taxes... and two extra 48port 1/10gbe f2 line card if I make the 7009 continue to be server/san access is ~48k after taxes, so seems to be same cost per port?
Last edited by dlploh04 on Thu Apr 19, 2012 8:15 am, edited 2 times in total.

User avatar
Vito_Corleone
Moderator
Posts:
9847
Joined:
Mon Apr 07, 2008 10:38 am
Certs:
CCNP RS, CCNP DC, CCDP, CCIP

Re: Cisco Catalyst 6509E vs Nexus 7009

Thu Apr 19, 2012 8:13 am

Yeah, you've been misinformed (sounds like your partner sucks). I do suggest pulling the M cards. I don't see the need for them and it definitely makes your design a lot more complex.

You surely COULD pull the 5Ks and hang FEXes off the 7K. Again, this does have some quirks, like no dual-homing your FEXes. That means you HAVE to dual-home your servers (which you should be doing anyway...). If you decide to go that route, I'd also want another F2 in each chassis so you can have some blade redundancy to your FEXes.

If you can keep the 5Ks you'll have a better FEX design, IMO.
http://blog.alwaysthenetwork.com

dlploh04
New Member
Posts:
15
Joined:
Wed Apr 18, 2012 7:25 pm

Re: Cisco Catalyst 6509E vs Nexus 7009

Thu Apr 19, 2012 8:24 am

OK great so we pull the M1! So glad you said something and will look into current partner.

For the two ASA 5520's, 2800/2900/3900 routers and other switches/devices that were going to be dual homed to the core into what we were told would be the M1 card via rj45 - where would they go now? Would we have them go to two 3750x/4948e and then uplink those two 3750x/4948e's via sfp+ dual homed to the two 7009?

To avoid quirks with FEX's - would you say its perfectly OK performance/management wise to dual home server/san directly into 7009 or to definitely not dual home our servers and san's directly into the 7009?

If we keep the 5k's and add fex's, that does sound like perfect two tier design, albiet with more upfront costs. Will we see negative opex or other issues with not doing so?

Really appreciate all your advice btw!

User avatar
Vito_Corleone
Moderator
Posts:
9847
Joined:
Mon Apr 07, 2008 10:38 am
Certs:
CCNP RS, CCNP DC, CCDP, CCIP

Re: Cisco Catalyst 6509E vs Nexus 7009

Thu Apr 19, 2012 8:31 am

I think you can still plug into the 7Ks if you want. You just need the GLC-Ts (keep in mind that you NEED gig on your routers (2851+, I think) here as these aren't going to do 100m in the 7K). I'm not saying you should avoid dual-homing your FEXes to 7K, I'm saying you CAN'T do it.

I don't think you'll see negative opex necessarily. I think it's a less clean design and you're doing access-layer stuff on your core (meaning not too much config separation unless you break into VDCs - IIRC, FEXes can only exist in a single/primary VDC, so keep that in mind).
http://blog.alwaysthenetwork.com

dlploh04
New Member
Posts:
15
Joined:
Wed Apr 18, 2012 7:25 pm

Re: Cisco Catalyst 6509E vs Nexus 7009

Thu Apr 19, 2012 8:42 am

I'm ok with not breaking into VDC's currently based on our company size - If we need that we can allocate capital to purchase the advanced license and possibly dedicated access switches for that capability in the future.

If we do spend the extra ~36k for 5548up's and I'm estimating two 2232p fex's will cost ~$15-20k total, I think we lose netflow on all our servers/sans (5k not support netflow at the moment right?).

so two scenarios then?:

Two 7009's:
each 7009 will have the redundant supervisors, five fab2 modules, and two power supplies, running in quad-sup VPC
each 7009 will have a f2 48port 10gbe card for downlinks to the nexus 5548's and catalyst 4507's
each 7009 will have LAN Enterprise License (L3 Protocols)
some appliances do not have multiple nics and are single homed into one of the two core switches.
each 5548up will have servers/sans dual homed to them (two total) and 5548up's will dual home to both 7009's OR we will have two fex 2232p and each fex 2232p will connect to a single 5548up that is setup in a vpc-design and the 5548up's will be dual homed to both 7009's.


Two 7009's:
each 7009 will have the redundant supervisors, five fab2 modules, and two power supplies, running in quad-sup VPC
each 7009 will have a f2 48port 10gbe card for downlinks to the catalyst 4507's and glc-t for connectivity to routers/switches
each 7009 will have LAN Enterprise License (L3 Protocols)
some appliances do not have multiple nics and are single homed into one of the two core switches.
servers/sans will dual home into the two 7009

Otanx
Post Whore
Posts:
1193
Joined:
Wed Sep 01, 2010 3:37 pm
Certs:
CCNP, CEH

Re: Cisco Catalyst 6509E vs Nexus 7009

Thu Apr 19, 2012 9:19 am

Is there a reason to do quad-sup? I have not used a 7K, but with the 6500s I don't see the benifit considering the cost of adding two sups(is quad-sup supported yet for VSS?).

-Otanx
Stay networked, my friends.

imogthe
New Member
Posts:
19
Joined:
Wed Feb 15, 2006 6:42 am

Re: Cisco Catalyst 6509E vs Nexus 7009

Thu Apr 19, 2012 3:02 pm

I'll put down a few words, so feel free to discard them at will.

We've implemented a similar solution with two 7009 switches with F2 modules, 5596UP switches and 2232PP FEXes, and there are a few things to be aware of with regards to the 7k platform in general and the F2 modules in particular.

1) The F2 mainly a High Troughput line card, whereas the M series is for Features. This means that you need to carefully consider what you want to do and figure out if the F2 will do it for you. For example the data sheet says it does NetFlow whereas the latest software specifically says it _doesn't_ do it on the F2. TAC says to wait for 6.1 release in Q2Y12.

2) IPv6 on NX-OS (caveat: on 7009 chassis with F2 module) is missing a few features such as RA priority and suchlike. Not sure if this is coming or not, but it's a bit of a bummer if you wanted to make one chassis the primary unit for a particular VLAN (both v4 and v6).

3) We're currently engaged in a TAC case concerning some problems we're having with HSRP and DHCP relaying - the DHCP discovery is forwarded to the server, but in some cases the reply doesn't reach the client. In this particular case the servers subnets are routed on both 7009 switches but the primary switch for the client VLAN might not be the primary for the servers, causing a degree of asynchronous routing (long story). At the moment it looks like a ARP or layer 2 issue but we're not sure yet.

4) NX-OS is very much into the new NetConf management way which means many of your old SNMP tools won't work (this goes for both 7k and 5k). DCNM is not a very mature product as far as I'm concerned and leaves a lot to be desired. If you have SNMP tools you need to make sure you can use them with the new platform - the MIB support on the 5k is laughable, for example. If you do everything by CLI and spurn fancy GUIs as toys for children you're golden :)

There have been a few other minor niggles, such as the mgmt0 interface on two 5k switches actually changing MAC addresses after a software upgrade (which meant the NMS thought they were down for a good 30 minutes until we could investigate - the MAC addresses had been incremented by 1 for some reason). They also had a _serious_ bug in the 5.0(something) version of code for the 5k that made the vPC pretty much useless. This was fixed in the 5.0(something + ohcrapIcantbelievewescrewedthisup) release.

I'm not saying to steer clear of the 7k/5k switches, just be aware of what your expectations are and make sure your kit will deliver (ideally with close support from your VAR, preferably with documented deliverables!). The Nexus series appears to be the way forward, but it's not the trusty 6500 platform of yore quite yet. Granted, the 6500 has been allowed to mature for a long time (some would say too long), but there are a few issues on the Nexuses that are not worthy of a Cisco flag ship product no matter how you slice it.

In short, the 7k/F2 combo is a good product with a lot going for it. In a year or two with loving software updates it may even become great. As it stands I feel a tad cheated by Cisco SlideWare and the SE/VAR, but as we've got a sunset/horizon of about 10 years for this kit it might have improved significantly by the time it gets scrapped in favour of something Greater(tm) and we start all over again :D

I apologise if I hijacked the thread, but it's been a full day of smashing my face against the wall of "don't tell me this shiny new Nexus actually fails this badly at something that shouldn't even be an issue" :)

javentre
Post Whore
Posts:
1971
Joined:
Fri Jul 09, 2010 7:38 pm

Re: Cisco Catalyst 6509E vs Nexus 7009

Thu Apr 19, 2012 3:22 pm

Otanx wrote:Is there a reason to do quad-sup?
Yes, dual sups in a single chassis makes upgrades hitless.
http://networking.ventrefamily.com

dlploh04
New Member
Posts:
15
Joined:
Wed Apr 18, 2012 7:25 pm

Re: Cisco Catalyst 6509E vs Nexus 7009

Thu Apr 19, 2012 7:29 pm

imogthe - This is such a large investment it makes it difficult because once chosen its purchased & delivered and there is no turning back. Then it is configured/tested/installed and can't really do a hands on comparison to the other choice. I take your opinion with great weight - thank you.

All of the issues you mention do cause concern as we do not want to be beta testing a core switch! =) sigh...

Otanx,

on the cat 6509e with quad-sup - the failed supervisor dies and the standby comes online after the entire chasis reloads - it provides a deterministic outage window measured in minutes and you are back online. Replacing failed supervisor can be done without any priming or commands.

on the cat 6509e with quad-sup - the plan is later this year to have SSO within the chassis so can do upgrades hitless.

on the nexus 7009 with quad-sup - the failed supervisor dies and standby comes online without any reloading -completely hitless. it also already allows for hitless upgrades.

The bandwidth performance on the Nexus 7009 really does seem to be more than we need or will ever use - but at same time - I do not want to recommend and purchase a 6509e that will be announced end of sale or discontinued within 4-5 years - our partner says the 4500e/6500e should continue to be sold until 2020. I understand Cisco Live is in a couple months and maybe the 7009 will even be upgraded with newer sup's and line cards, and make the 7009 purchase look bad timed...so much to decide on... :|

'
Next

Return to Cisco General

Who is online

Users browsing this forum: No registered users and 16 guests