While I might expect to see results like this on a test box, I would be mortified if I saw it on a production box. As a small challenge to your SQL Server administration skills, just by looking at the above data, what can you deduce it? What do you think is going on, and why, and how might it affect the databases’ performance? In addition, what might you do to prevent the problem in the first place? Let’s see who can come up with the best explanation. Please post your responses below.

PS: If you are not familiar with this report, I suggest you learn more about it. It is available from SSMS by right-clicking on a database, and then selecting Reports | Standard Reports | Disk Usage. Such a simple report can make it easy to identify some specific database configuration problems, as demonstrated in the example above.

In SQL Server 2005 and later, there is a database option called “forced parameterization”. When it is turned on, it overrides the default “simple parameterization” normally used by SQL Server. It does this by forcing most SELECT, INSERT, UPDATE, and DELETE statements to be parameterized, potentially boosting performance of databases by reducing the number of query compiles and recompiles that need to occur.

As with many “options” in SQL Server, whether this option should be turned on for a particular database is not a simple decision. So I was curious to see how many DBAs actually employ this option. I ran the above poll on my website, and had two surprises. The first surprise is that about 19% of responders said they use it, which I thought was high. The second surprise is that about 44% of responders have not even heard of this option.

If you are one of the 44% who aren’t familiar with forced parameterization, I suggest you read the following to learn more about it. While this database option can be very handy in specific cases, it can also hurt performance in other databases. If you try this option, be sure to perform a benchmark before implementing it, then another benchmark after implementing it, and seeing if you get the behavior you expect. Of course, perform this experiment in a test environment, not in production.

http://technet.microsoft.com/en-us/library/ms175037.aspx 

http://www.sqlmag.com/blog/sql-server-questions-answered-28/sql-server/forced-parameterizationwhen-should-i-use-it-137149

Two years ago when SQL Server MVP Deep Dives: Volume 1 was released, I had the opportunity to contribute two chapters. In SQL Server MVP Deep Dives Volume 2, I not only had the opportunity to contribute a chapter, but the privilege of being the section editor of the “Performance Tuning and Optimization” section of the book. I would like to thank all of the following writers in my section for all their hard work, and for getting me their drafts on schedule. I also want to thank Grant Fritchey, who helped me coordinating the technical editing of the chapters.

The writers who I worked with in the Performance Tuning and Optimization section of the book included:

• Glenn Berry
• Grant Fritchey
• Jason Strate
• Robert Pearl
• Linchi Shea
• Jennifer McCown
• Tim Chapman
• Jungsun Kim

The book is available for pre-order at the Manning website, and hard copies will be available for purchase at the PASS Summit. If you are attending PASS and want to purchase a book, you can also get it signed by many of the authors at two different autograph parties at these times:

• Wednesday, 1:00 – 1:30 PM, at the Expo & Dining Hall
• Friday, 7:15 – 8:00 AM, in Room 211

If you can’t attend either autograph party, each of the book’s authors attending PASS will be wearing ribbons that say “MVP Deep Dives Author”, so all you have to do is to chase them down to get their autographs.

All author royalties will be donated to Operation Smile, an international children’s medical charity.

• I revoked permissions as sysadmin for myself.
• Instead of logging out of the SQL Server, I shut it down instead, right in the middle of the day when it was heavily being used.
• I was intending to drop a database from a test server, but didn’t realize I was actually on the production server, and as you have probably guessed, I deleted a production database.
• I setup a SQL Server Agent job to delete a single record each night. But I forgot to include the WHERE clause and all the data in the table was deleted when the job ran.
• I had noticed that a SQL Server Agent job running a SSIS package failed. Even though I was not familiar with what the SSIS package did, I manually re-ran the job, and it worked this time. Unfortunately, the SSIS package dropped a critical production database.
• I didn’t bother backing up some seemingly unimportant databases. When one of these databases got corrupted, there was no backup. I had to recreate the database and manually repopulate it myself.
• I neglected to pay attention to how fast our databases were growing and ran out of space without knowing it was going to happen.
• I ran a script given by me by our auditors to generate some reports, which caused the database to go into a suspect mode. Now I don’t run any scripts unless I review them first.
• I accidentally restored an older backup over a production database. I had intended to restore the backup over a test database.
• I setup log shipping backwards.
• I restarted the SQL Server service during the middle of a schema change that resulted in dropping of a production table.
• A user accidently deleted some rows on a table and I was asked to restore the database from the most recent backup. Unfortunately, I had never tested any of the backups, and none of them would restore.
• I restarted a SQL Server instance. When it came back online, one of the databases was in suspect mode. Not knowing what to do, I detached and then tried to reattach it, which made matters worse. Now I know that detaching a suspect database is about the worst thing you can do.
• A large server was experiencing some disk-related errors, so it was decided that the fix to the problem we needed to essentially rebuild the server. Of course, before doing so, I backed up all of the databases on the local machine. So we performed the hardware repairs, reformatted all the arrays, reinstalled the OS and SQL Server. It was only then that I realized that I had forgotten to move the backups off the server we had just reformatted. to another location. Fortunately, older backups were available, but not the most recent backups.
• My biggest mistake happened when I needed to take down a server to upgrade the disks that held the SQL Server data files. The plan was simple: take the SQL server down, move all the files off the RAID 1 array  that held the .mdf files, replace the disks in the array (I think we were going from 73Gb HDDs to 146Gb HDDs), let the RAID 1 mirror rebuild, then put the files back, and go home.  Very quickly we hit a snag (which turned out later to be a firmware bug) in that the new larger disks weren’t recognized by the RAID controller. We then put the old drives back, and they weren’t recognized as well. My biggest mistake was that I hadn’t backed up the databases or detached them. I’d simply moved the files after shutting the SQL Server down – so when SQL Server was started it would be looking for data files which simply wasn’t there! The only option was to get the server fixed, and luckily we had 4 hour support on the server, however it was rapidly approaching midnight and I was due to attend my brother’s wedding the following morning!  I quickly returned home to get enough things packed in a bag so that I could go straight to the wedding once the work was complete, while we waited for the engineer and replacement RAID controller card to arrive. Four hours later we had the server back up and running – and watching that partition initialize was excruciating!  I managed to get home and catch a couple hours of sleep, even though I was still buzzing from adrenaline and caffeine!  The lessons were hard learned from this, and I have never since planned ANY upgrade be it hardware or software, major or minor, without an escape plan after every step.
• While upgrading a large database from SQL Server 2000 to SQL Server 2005, one that held important financial, it was decided by the business users that a large portion of the data in the database would be archived and then deleted from the production database. The end users worked with the vendor to get a script to handle this operation, and in testing it, the process took over 30 hours, which was way beyond out maintenance window.  So after some review, and discussion with the vendor’s support and development staff, I realized that it would be faster to create a new copy of the table, insert the rows to be kept into it, drop the old table, rename the new one, and rebuild all the constraints and indexes.  This took the process down to about 1/2 hours in testing. The problem came when I loaded the script on production to run the purge process.  In testing, I had performed the operation as an explicit transaction, and initially I had left the SET IDENTITY INSERT statements out of the batch by mistake.  It wasn’t a problem on development because I had blocked the code in an explicit transaction, so when the inserts failed because I didn’t SET IDENTITY_INSERT ON, I issued a rollback, waited, and then fixed the problem.  For some reason, I never saved the script.  When it came time for the scheduled production downtime to perform the upgrade, I loaded the script, ran it, and the table disappeared in mere seconds. The first thing I did when I realized what I had just done was to notify my manager of the problem and let him know that the system wasn’t going to be available as fast as expected, allowing him to decide how best to notify the end users if necessary. Then I took a walk around the building to think about what was going to be required to fix this, and in the shortest time possible. This two minute walk gave me the time to relax for a minute and plan what I was going to do, rather than trying to shoot from the hip. The good thing was I had backups and they had been validated numerous times during testing of the upgrade, and the database was fully logged, so I was able to backup the transaction log from production, and restore the database side-by-side, applying log backups to roll the database forward to the point in time after the downtime began, and before I executed the script.  Then a few changes to the original script allowed me to move the data that was supposed to be retained from the restored copy into the production database, and the system was able to come back online with zero data loss.

If you have any mistakes you would like to share, anonymously or not, please do so below.

Every since the first file was written to a floppy disk drive using DOS, physical file fragmentation has been a problem. Essentially, when the OS writes a file to a disk subsystem, and if contiguous clusters are not available, they are written elsewhere on disk. So when a file is stored on disk in a non-contiguous manner on a disk subsystem, the file is considered to be physically fragmented. Physical file fragmentation can contribute to an additional load on your I/O subsystem and reduce performance because the disks have to work harder to read and write data. This is because the heads on disk drive have to randomly jump around to different physical locations on the disk to find all of the data, instead of reading and writing the data sequentially, as when the data is contiguous.

Physical file fragmentation can be slight, say when a single file is divided into only a handful of locations on a subsystem. In other cases, a single physical file might be divided into thousands of fragments, resulting in heavy physical file fragmentation. The greater the physical file fragmentation, the harder disks have to work, and the greater overhead that is incurred. Ideally, for best I/O performance, files should be written as contiguously as practical.

How physical file fragmentation affects SQL Server’s performance varies. For example, if a single random read or write needs to be performed, the degree of physical file fragmentation will have little or no effect on I/O performance, as the amount of work to perform a single read or write is not significantly affected by physical file fragmentation. On the other hand, if SQL Server needs to read or write a large quantity of sequential data to or from disk, and the file is heavily fragmented, then the disks will have to randomly read or write the data, even though the data is logically sequenced, resulting in extra work for the disk drives, hurting I/O performance.

Another factor that can affect how SQL Server and the disk subsystem interact is if there is a large cache sitting between SQL Server and this subsystem, or if the disk subsystem in made up of SSD or similar drives. Both of these can significantly mask the negative effects of physical file fragmentation, although it is not eliminated.

So what does all of this mean to DBAs? Because of the complexities of measuring the negative impact of physical file fragmentation on SQL Server’s performance, I make the assumption that it is always bad and do my best to prevent it in the first place. As a DBA, I always assume the worse and plan for it.

Here’s what I like to do. Before I create any new database (MDF, NDF, or LDF files), I first use the Windows defrag.exe tool to see if there is any physical file fragmentation on the disk array where I want to create my database files. If there is, then I use defrag.exe, or a third-party tool, to defrag the physical files before creating my new database. Then, I pre-size my database files to a size that I think will suffice for the next year or so. For example, if I estimate that my MDF file will be 500 GB in size in the next 12 months or so, then that is the size I create the database now. The same with the other database files.  By pre-sizing my database files, auto growth doesn’t have to kick in, which is one of the biggest causes of disk fragmentation. In other words, by creating a large file on a disk array that is defragmented, the file is created contiguously and I can prevent physical disk fragmentation from occurring in the first place.

Now, you may have two questions. First, you may be wondering about how you go about defragging a disk array that is currently being used with production databases. The problem is, is that the defrag.exe program will only defrag closed files, not open files, such as any SQL Server MDF, NDF, or LDF files that are in production. So the dilemma is, you have identified physical file fragmentation on an existing disk array where you want to create a new database, and you want to defragment it first before creating the new database to eliminate any physical file fragmentation. Assuming that you are using defrag.exe, your only option is to shut down the production databases during a maintenance window, perform the defrag, then restart SQL Server. Yes, this is a pain, but this one time pain will not only remove any existing physical file fragmentation in your current databases, it will help prevent it from future databases. As the DBA, you have to decide if the trade-off in taking the server down for a while, and potential performance gains it can provide, is worth it.

The second question you may have is how do you know what size you should pre-size your database files when you first create them? This requires experience and an educated guess. The odds are that your guess will not be exact. You will either underestimate or overestimate the size of the database 12 months in the future. There is no way to know for sure. But if you make an attempt using your best guess, your guess will be much better than letting auto growth grow the database for you, which will most likely result in physical file fragmentation. As time passes, and you realize that your guess was wrong, you can always manually grow the database as needed to make more room. Or, if you estimated too much, I would leave the extra room (and not shrinking the file sizes), unless you are in the immediate need of additional disk space. But won’t this also lead to physical fragmentation? Yes, to a limited degree. But most likely, a slight amount of physical file fragmentation will never be noticed.

The times where I have seen physical file fragmentation become a real problem is when an initial database is created with the default settings, where the MDB is set to a 1MB auto growth and the LDF is set to the 10% auto growth. If such a database is not pre-sized, but grows to multiple gigabytes over time, physical file fragmentation can become a substantial problem, especially if there are multiple databases, all using the default setting, on the same SQL Server instance.

In other words, a little physical file fragmentation is nothing to worry about. But a large amount of physical file fragmentation, often due to poor database management, can end up hurting your SQL Server’s I/O performance more than you might expect.

Most of the time when I run a poll on my blog, I run it for at least a month so that I can get enough results in order for the data to be more or less representative of the SQL Server community. What has been odd about this poll, is that even though I have run it longer than any other poll on the website, it has had one of the fewest response rates.

When I look at the results, my best guess is that people who have looked at the poll weren’t familiar with the question, and because of that, ignored responding to it, instead of admitting that they didn’t know what it was. Or, perhaps it is because this feature is only available in SQL Server 2008 and higher, and not everyone is running this version yet.

When you look at the results, just over 36% of the respondents said that they use the “optimize for ad hoc workloads” SQL Server configuration option. Of the 15 or percent that said no, it is hard to know if they are not using it because they don’t know what it is, because they have made a deliberate choice not to use it, or because they don’t have any SQL Server 2008 instances. (That’s one of the limitations of such a simple poll.) About another 10% said “it depends” if they use it or not, and about 38% of the responders admitted not knowing what the feature does. I am guessing that the 38% is understated, assuming my belief that people didn’t respond to this question because they didn’t understand the question. It’s too bad that so many DBAs aren’t familiar with this option, as it is a very simple option that can return big benefits for many SQL Server instances.

For those of you who don’t know, when “optimize for ad hoc workloads” is turned on (it is off by default), SQL Server will only store a small compiled plan stub, not the entire execution plan, the very first time an ad hoc query is compiled for the first time. Should the same batch be executed again, then this time around, the full execution plan will be cached in the plan cache, which is what happens to ad hoc queries when this option is not turned on.

So how is this option useful? If your SQL Server instance executes many ad hoc queries, it is very possible that once the query is executed, it may never be executed again. This can produce what is sometimes called “plan cache bloat”, which means that much of your plan cache is wasted by execution plans that will never be used again, reducing the amount of buffer pool memory available to SQL Server. The “optimize for ad hoc workloads” option helps to prevent “plan cache bloat” by only caching the full execution plan of queries that are known to be used more than once. If “plan cache bloat” is a problem that you have, then turning on this feature can boost the performance of your SQL Server instance.

How do you know if you have the “plan cache bloat” and should turn on “optimize for ad hoc workloads”? The best way is to query sys.dm_exec_cached_plans, using one of the many queries available on the web, such as the one written by Kimberly Tripp. By using the querying this DMV, you will be able to quickly tell if your instance has a “plan cache bloat” problem. If it does, then turn this option on, as there is virtually no downside to doing to, and a great potential upside.

If you have any experience using this option, and would like to share your experiences, please do so below.

This past week, about 1,200 developers, IT professionals, and DBAs spent three days attending DevConnections at the J.W. Marriott Resort and Conference Center in Orlando, FL. While the above photo depicts a perfect Florida spring day, most of the time the weather was terrible with storms and rain. It really didn’t matter much for the attendees, as we stayed inside and focusing on learning and networking.

DevConnections is a combination of many smaller conferences, including:

• Visual Studio Connections
• ASP.net Connections
• Silverlight Connections
• SharePoint Connections
• Exchange Connections
• Windows Connections
• SQL Server Connections

The SQL Server Connections part of DevConnections includes 3-4 tracks each day, focusing on a wide range of topics of interest to DBA administrators and developers. One of the SQL Server Connections tracks was sponsored by SQLServerCentral.com, which I helped to manage. The speakers and sessions for the SQLServerCentral.com track included the following.

Jonathan Kehayias spoke on “Learn SQL Server Internals Using Extended Events” and “Wait Statistics: A Troubleshooting Methodology”.

Brian Kelley spoke on “Windows Operating System Internals for Data Professionals” and “Knowing What to Look for in SQL Server Security”.

Joe Webb spoke on “Tips & Tricks for Writing Better Queries” and “Locking & Blocking Made Simple”.

I make three presentations, including “Introduction to SQL Trace”, “How to Optimize TEMPDB Performance”, and “Database Maintenance Essentials”.

While Steve Jones did not speak (above left), he Tweeted and blogged the event. Overall, 373 people attended the SQLServerCentral.com track this year. Currently, I am working on the SQLServerCentral.com track for the DevConnections events in Germany and London in June, and in Las Vegas this November.