Comments on: March Question: What is the biggest mistake/problem you have ever found on a SQL Server instance, and how did you fix it?

By: bradmcgehee

bradmcgehee — Thu, 01 Apr 2010 18:37:29 +0000

The March 2010 contest is closed, and a winner will be announced soon.

By: Bisjom

Bisjom — Mon, 29 Mar 2010 00:14:40 +0000

In one of my job, the developers were using the same production box for developing, testing and production and with FULL GOD permissions to the server and complaining about the performance and security. But they were demanding that they need full access for programming. They were running around 75 dts packages for testing and live etc..and more 500 individual users which has full access to the server.
My manager helped me to present a session to the team to make them realise the need of different platform and security changes and that helped me to do a remediation project.
Now it changed to three environments:
Development server which has full access to developers, Staging with partial access to developers and production server.
The production server has only 17 packages now and applications running really smooth and fast.

By: Alex Rosa

Alex Rosa — Sat, 27 Mar 2010 16:03:58 +0000

Scenario:
In the beginning of my career as DBA, I was working in a product release, and database schema update was one of the steps required.
We had 4 hours to deploy the new version, but one of the scripts to update the database schema was wrong.
The script was adding a new table column into a huge table, and the NOT NULL constraint was used.

Impact:
As you know, including a new table column with NOT NULL constraint into a huge table take a long, long time.
So, I needed to abort the operation, but as I know today, this situation can become a big problem too.

Cause
I got the script from Development team, and I didn’t validate the script properly, because we didn’t have enough time and the Project Manager was pushing us.

Solution
Only to restart the SQL Server instance was enough, because the database started the recovery process.
So, I dropped the database and recovered the backup file made in the beginning of the deployment, and redo the schema updates, off course changing the table column to accept NULL.

Lessons learned
Firstly, always perform an instance/database backup before any production deployment.
Next, test the backup at least with RESTORE VERIFYONLY.
Finally, follow the best practices and your checklist to avoid mistakes and save your job.

P.S.: The deployment took the double of hours planned because of this.

That’s all folks

By: Shawn Johnson

Shawn Johnson — Mon, 22 Mar 2010 23:45:41 +0000

I was asked by a friend who was a SQL DBA for a financial services company who had a tool that integrated with the SWIFT financial network to help figure out an issue they were having. The application was written in VB.Net and had a SQL Express database that communicated with a larger SQL Server that did the SWIFT processing. Anyhow, they had one system whose nightly batch jobs just wouldn’t run at night but would during the day if she ran them manually or even if they were scheduled. The problem persisted for about 6 months and had everyone just baffled. After all, the job would run just fine. Problem was they needed to be processed for batch processing after 5pm. Finally, in complete frustration they had her document her entire day. What step by step things she was doing in hopes of finding something, anything. She was doing everything perfectly and went above and beyond in the detail which included her coffee breaks and everything…it was kinda funny. They finally focused in on what she was doing between 5:00 and 5:05pm and started moving the job forward a bit and one day they had it work but it was before her deadline so they could only do it once and then move it back past that cutoff (a financial requirement). As soon as they moved it back past the 5pm cutoff, it would happen again. Unbelievable!

This is where I entered the picture over a beer one night playing darts. I asked Tucker for the SQL logs and noticed that the server was shut down at almost exactly 5pm each night and was restarted almost exactly at 8am the next day. Thinking we had a smoking gun, we asked her about shutting off her PC that processed all of this. She swore up and down that she was not turning it off as she knew that it needed to batch process things and her very detailed logs proved she didn’t. Her detailed step by steps showed that around 5pm each day she would lock the workstation, organize her things, get her coat, turn out the lights, lock the office and run to catch the bus. Then she would arrive in the morning and do the reverse and after getting situated grab a cup of coffee and then sign into the PC–that asked her username and password.

So how was the workstation being shut down? Well, come to find out, the workstation was plugged into the upper outlet of a wall socket that was tied to the light switch. So when she turned out the lights at night, it killed the power to the workstation. When she came in the morning and turned on the lights, it started it up. By the time she got situated and got her morning cup of brew, it was at the login screen and the SQL Server service had already started up and appearing as it was the night before…a locked workstation asking for a login.

Moral of the story: Make sure the computer is plugged into a hot electrical outlet and check your SQL logs!

By: Marianne

Marianne — Mon, 22 Mar 2010 22:42:37 +0000

I was working as a System Admin/Business Analyst. One particular day, I started receiving phone calls and emails stating that a lot of information was missing off of reports that were sent out at the end of the month to various clients. I gathered the missing info and when I checked the database tables I could not locate the information. I looked at the month before reports and verified that the info had been in the table at one time. Low and behold, it appeared that our DBA team had loaded an update to the application that they had received from the application’s developers. They didn’t bother to look at it before loading it. If they had they would have seen right at the top a DROP TABLE command. Lesson here is to read anything that could mess up your data before loading it.

By: Holger Schmeling

Holger Schmeling — Thu, 18 Mar 2010 16:13:08 +0000

Wow, really interesting topics here! The comments made so far seem to fall into two categories: configuration- and design issues. From my experience, I’d say bad application or database design as well as insufficient requirements analysis are main sources for kind of misbehavior or unexpected outcome. SQL Server itself is very modest and tolerating – well almost.
Ok, here’s my favorite, falling in the “bad design” category:
I was hired on contract basis for a very large and international operating company, where I had two heads on. First, I was kind of advisor/consultant for database administration, as the main existing experience was based on Oracle. In particular however I had to develop reports by the use of SSRS. As it soon turned out the decision of implementing an SSRS infrastructure was quite arbitrary taken. All the end users (the report-readers, if you like) wanted to see were Excel sheets. No more. The wanted to have all of their data in large Excel documents, so they could perform the reporting mostly by themselves. Moreover we had a lack of specification, that is, nobody made any particular requests, what data to present in those excel sheets. The statement was just this: “We want to have all available data in Excel, so we can do the filtering, sorting, and whatever by ourselves.” I have to say that those users where very well trained and quite smart in working with Excel, so I yould understand their issue. I tried to explain that SSRS is not the best option for generating Excel documents and also that Excel sheets, containing 2,000,000 rows and 100 columns may be difficult and resource intensive to create and process. But they insisted and I – as a freelancer – had no chance in achieving any noticeable acceptance. Ok, I got paid for this and everybody (except me and a colleague of mine) was quite happy.
Actually they even upgraded to the latest Excel version to overcome the 64,000 rows barrier (not sure, but was it with Office 2007)? and constantly complained about how bad SSRS’ Excel rendering functionality was.
We had some large databases (TB range) and the load we put on SQL Server was huge. Guess what? As often this problem was soved simply by adding more hardware…
So, I didn’t solve the issue, since nobody wanted a more customized solution. Why not? Because this could only been achieved by going back to the requirements process and nobody was in the “mood” for doing so.
So I grabbed my money and left – assignment done. I was neither quite happy nor satisfied but at least a little wealthier than before.

By: Tim

Tim — Thu, 18 Mar 2010 15:13:01 +0000

My first programming job was a clothing order entry system that used Delphi/SQL server for order management and an AS 400 for inventory and financials. We used a 3rd party extended proc to interface the orders to the AS 400. This had been working well for a couple years. Then the production server starts blue screening on us. After a couple weeks, a pattern develops. The server (single processor 3gb memory) was blue screening every Tuesday around lunch time. After a ton of traces, analyzing query plans, trying the 3gb and AWE switches, learning about performance counters, it was still going down every Tuesday. We even contacted the power company to see if we were getting spikes or anything unusual on Tuesdays.
We finally broke down and contacted MS and opened a support ticket. After a month of analyzing dumps and using scripts to capture performance counters every 2 minutes, we finally figured out that the the 3rd party extended proc had a memory leak. Because extended procs live in the MemToLeave buffer, the memory is not managed by SQL Server and would eventually end up using all available memory on the server, causing the blue screens.
We ended up adding 3gb of memory and setting the max memory limit. That worked until we upgraded to Server 2003 DataCenter.

By: Greg E

Greg E — Thu, 18 Mar 2010 13:56:46 +0000

A few years ago, we were going to apply a service pack to SQL server. This one had a note about removal – rebuild the server, it couldn’t be rolled back. We had no real dev environment to test in – dev was virtual and 32 bit. Testing looked good. We had backups – someone else was in charge of them. Just for insurance, we made backups of our db’s.
Friday afternoon was determined to be a good time as we went a BI server with no transactional db’s, and if we ran into any issues, support was available.
We applied the service pack to the x64 bit production server. Everything worked good at first. We ran our ETL to the SQL base warehouse – worked fine. Then we tried to build our cube, and the problems started.
After making very little progress fixing the problem, we made the decision to roll back to where we were at the start. We started from the ground up with the OS, and then SQL Server, etc.
There was a new adminstrator who was in charge of the backups, and a 3rd party tool was used that compressed them. He ran into issues and was unable to get the product installed. No 24 hour support contract with the vendor, so we were on our own. What this meant, is our backups were OK – the redundant uncompressed ones we made ended up needed. But the core system dbs could not be restored. So we ended up late into the night manually recreating logins, permissions, etc. from memory.
We managed to get it up and running, but it was almost sun up on Saturday morning before we had a cube building.
By Tuesday afternoon, the dba in charge of the backups had worked with the vendor and got the 3rd party software installed properly.
And we ended up a bit later finding out our service pack issue was a bug related specifically to x64 bit, and also to partitioned cubes (and Enterprise only feature).
We now have dev environments that are the same platform as production and script out everything ‘just in case’.
Greg E