When I started at the company, the server was failing regularly. They called it a “train wreck” as transactions blocked other transactions in a cascade effect, sometimes requiring hundreds of users to have to log out so the system could be rebooted. The application vendor previously had the employer add another volume to the server (for three total), but they still didn’t configure the database files correctly.

A look at the disk I/O in Performance Monitor said it all – the server just couldn’t handle all the concurrent read/writes. I told the network engineers and business analyst acting as a DBA to build a batch processor server, move all background jobs and work files to the batch processor, and clean up the SQL Agent jobs to eliminate redundancy and overlap just for starters.

They ignored me. For 6 months the fiasco continued until the server just wouldn’t respond and the business ground to a halt. They were finally forced to listen, and started by moving the work files off-server. It was like doing CPR and seeing the victim start breathing again. Once all my recommendations were implemented, there wasn’t a single “train wreck” caused by disk I/O or server performance again.

The plan is to full backup weekly w/ nightly diffs. The tlog will be 15 minutes. They get copied off to the network and usb drive, which is taken offsite. I like the convenience of disk, especially for remote sites with non-IT personell.

I'll have a vmware guest to run on the production server, which will (hopefully) retain an identical configuration setup to the production machine. test restores etc can be done on the guest. It shouldn't run 24/7 lest it interfere with the production.

db scripts are from sqlservercentral.com

“A significant part of sql server process memory has been paged out.”

First thing I looked at was the server memory setting and found that Max Memory was not set up. So when OS need the memory it truncated the working set memory of SQL server drastically and hence the error.

This was not a big issue as setting the Max Memory would have solved this problem. But convincing upper management that this is the solution was the big issue. They believed that since I am “New”, I don’t know the system and environment to make this decision. I have to pull many articles from internet and convince them that this is the solution and they accepted it once they knew that its a dynamic change and no reboot of server is required and I have a rollback script ready if anything goes wrong.

Doing this immediately stopped the errors we were getting for a whole week continuously. This also increased the upper management confidence in my abilities. I am at this job 4 months now, still fighting lot of daily small fires here and there, but that particular error is not seen anymore.

I returned to the profiler and after isolating the traffic discovered the source of all the requests this time was from a handful of very similar looking queries. Tracing it back to a service on the Laserfiche server I was able to discover that a PC tech in an effort to reduce the manual workload of some office staff had written a Laserfiche Workflow script. It was a series of reflective triggers that was basically running a handful of queries over and over and over until it got an expected result and moved on to the next task in the purposly infinite loop. About 20 PC’s were all running different versions of the triggering event that was in turn pelting the server incessantly. Luckily for the server it was able to keep most of the requests at bay with cache information but memory levels and disk activity was still low and high respectivly.

After a lengthy discussion with staff regarding this “time and resource saving” feature we were able to convince them to let us convert the Workflow Script to a small .Net app that would only run once – which we then put on a 5 minute schedule with task scheduler as opposed to the “as fast as you can send them” schedule Laserfiche had it on. The office staff still got the automation they were looking for and the average requests per second (on this one database) dropped from 4000 a second to less than 4.

That may not be the biggest mistake ever but the others that come to mind were my fault and I don’t really want to talk about those.

Uncompress, defrag… and voila, the performance improves.

P.S. I can’t top the failed backups, failed hard drive, move before log shipping, secret drive maintenance, drives not screwed in story.

The worst has to be the most simple one I ever came across and the one that only took me a couple of minutes to figure out.

I was called into a situation when an in-house developed application stopped responding in production by refusing to allow any more connections than 10 or so.

The application had been functioning perfectly during development and testing but within a few hours of going live it had responding.

It had the development team completely baffled. Once on the server, the first thing I did was check the version of SQLServer and to my amazement (or should that be horror), discovered that the application had been deployed using SQLServer Developer Edition which of course only allows a limited number of connections.

I simply upgraded the instance to SQLServer Standard Edition and everything started working. Simple, I know, but by far the worst mistake I have come across.

Considering myself a white collar professional, it never dawned on me that I would want to be paid for working more than 40 hours a week.

However, I felt that I would have more flexibility in my day to day work hours: working longer days when the work load required it, and working shorter days when the work was light. Over time ,usually a week, time at work would average 40 hours, although the average usually is more in favor of the employer. This has been my experience as a full-time employee.

My view changed while I worked on contract: I did charge for over time, but only when the time was mandated by the customer and the contract was amended to reflect the additional hours. Here again I was able to be flexible with my work schedule, working longer days to complete the work load, and banking those extra hours to create longer weekends when the workload was light.

If someone decides that we will be paid OT, I will not refuse!

By the way John S, people who have difficulty following instructions from the author have posted comments @ SQL Server Central. That is where I read this blog.