Application tuning

so yeah, work basically sucked last week. we ran into a huge issue last week with our systems. On a 10,000ft level, the application servers and attached equipment were causing the database servers to max out the CPU's. Now these DB servers are no slouches in toughness. they are equipped with 4 64bit dual core processors backed up with 25 gigs of RAM. 8 cores total were being maxed out by the application servers throwing massive amounts of simple commands at them. at one point I counted up each app server (total of 6 per POP location) dishing out well over 10k requests per 3 minute interval that I was tracing. It's not like the stored procs that were getting called were poorly tuned or missing some critical index, it was just the shear volume of requests that were coming in that was bring down the house. So, for 3 days we dug and dug and dug and dug to try and determine why exactly the application servers decided to all-of-a-sudden start beating down the database servers. Well, after 3 days of intense searching we are still stumped. for the most part it is in Engineerings hands right now, and the only thing I can do is wait and see what they come up with(their mess not mine! :) ). Hopefully Monday will come with a wonderful reward and our talented engineering group triumphantly vanquish this nasty little foe. Until then I just hope we haven't pissed off too many clients.

--Andy