While we can’t stop growth, SQL Server 2008 and later, gives us some tools to help us better deal with all this data, and that is the promise of compression. Given the right circumstances, DBAs can use data compression (Enterprise Edition only, or alternatively take a look at Red Gate SQL Storage Compress) to reduce the size of our MDFS, and backup compression can help us reduce the amount of space our backups take. Not only does compression reduce physical file sizes, it reduces disk I/O, which can greatly enhance the performance of many database applications, along with database backups.

When we discuss SQL Server compression, we need to think of it two different ways. First, there is data compression, which includes row-level and page-level compression that occurs within the MDF files of our databases. Second, there is backup compression, which occurs only when data is backed up. While both of these are forms of compression, they are architected differently. Because of this, it is important to treat them separately. In this blog post, I will mainly focus on data compression.

Data Compression comes in two different forms:

• Row-level Data Compression: Row-level data compression is essentially turning fixed length data types into variable length data types, freeing up empty space. It also has the ability to ignore zero and null values, saving additional space. In turn, more rows can fit into a single data page.
• Page-level Data Compression: Page-level data compression starts with row-level data compression, then adds two additional compression features: prefix and dictionary compression. We will take a look at what this means a little later in this chapter. As you can imagine, page-level compression offers increased data compression over row-level compression alone.

Backup Compression comes in a single form:

• Backup Compression: Backup compression does not use row-level or page-level data compression. Instead, backup compression occurs only at the time of a backup, and it uses its own proprietary compression technique. Backup compression can be used when using, or not using, data compression, although using backup compression on a database that is already compressed using data compression may not offer additional benefits.

In the next section, we will take a high-level overview of data compression, and then we will drill down into the detail of the different types of compression available with SQL Server 2008 and later.

Data Compression Overview

Data compression has been around for years. For example, who hasn’t zipped a file at some point in their career? While compression isn’t a new technology, it was new to SQL Server 2008. Unlike zip compression, SQL Server’s data compression does not automatically compress an entire database; instead, data compression can only be used for these database objects:

• A table stored as a heap
• A table stored as a clustered index
• A non-clustered index
• An indexed view
• Partitioned tables and indexes

In other words, as the DBA, you must evaluate each of the above objects in your database, decide if you want to compress it, then decide whether you want to compress it using either row-level or page level compression. Once you have completed this evaluation, then you must turn on compression for that object. There is no single switch you can flip that will turn compression on or off for all the objects listed above, although you could write a Transact-SQL script to accomplish this task.

Fortunately, other than turning compression on or off for the above objects, you don’t have to do anything else to enable compression. You don’t have to re-architect your database or your application, as data compression is entirely handled under the covers by the SQL Server Storage Engine. When data is passed to the Storage Engine, it is compressed and stored in the designated compressed format (on disk and in the Buffer Cache). When the Storage Engine passes the information to another component of SQL Server, then the Storage Engine has to uncompress it. In other words, every time data has to be passed to or from the Storage Engine, it has to be compressed or uncompressed. While this does take extra CPU overhead to accomplish, in many cases, the amount of disk I/O saved by compression more than makes up for the CPU costs, boosting the overall performance of SQL Server.

Here’s a simplified example. Let’s say that we want to update a row in a table, and that the row we want to update is currently stored on disk in a table that is using row-level data compression. When we execute the UPDATE statement, the Relational Engine (Query Processor) parses, compiles, and optimizes the UPDATE statement, ready to execute it. Before the statement can be executed, the Relational Engine needs the row of data that is currently stored on disk in the compressed format, so the Relational Engine requests the data by asking the Storage Engine to go get it. The Storage Engine (with the help of the SQLOS) goes and gets the compressed data from disk and brings it into the Data Cache, where the data continues to remain in its compressed format.

Once the data is in the Data Cache, the row is handed off to the Relational Engine from the Storage Engine. During this pass off, the compressed row is uncompressed and given to the Relational Engine to UPDATE. Once the row has been updated, it is then passed back to the Storage Engine, where is it again compressed and stored in the Data Cache. At some point, the row will be flushed to disk, where it is stored on disk in its compressed format.

Data compression offers many benefits. Besides the obvious one of reducing the amount of physical disk space required to store data—and the disk I/O needed to write and read it—it also reduces the amount of Data Cache memory needed to store data in the Data Cache. This in turn allows more data to be stored in the Data Cache, reducing the need for SQL Server to access the disk to get data, as the data is now more likely to be in memory than disk, further reducing disk I/O.

Just as data compression offers benefits, so it has some disadvantages. Using compression uses up additional CPU cycles. If your server has plenty to spare, then you have no problem. But if your server is already experiencing a CPU bottleneck, then perhaps compression is better left turned off.

Row‐Level Data Compression

The simplest method of data compression, row-level compression, works by:

• Reducing the amount of metadata used to store a row.
• Storing fixed length numeric data types as if they were variable-length data types. For example, if you store the value 1 in a bigint data type, storage will only take 1 byte, not 8 bytes, which the bigint data types normally takes.
• Storing CHAR data types as variable-length data types. For example, if you have a CHAR (100) data type, and only store 10 characters in it, blank characters are not stored, thus reducing the space needed to the store data.
• Not storing NULL or 0 values

Row-level data compression offers less compression than page-level data compression, but it also incurs less overhead, reducing the amount of CPU resources required to implement it.

Page Level Data Compression

Page-level data compression offers greater compression, but at the expense of greater CPU utilization. It works using these techniques:

• It starts out by using row-level data compression to get as many rows as it can on a single page.
• Next, prefix compression is run. Essentially, repeating patterns of data at the beginning of the values of a given column are removed and substituted with an abbreviated reference that is stored in the compression information (CI) structure that immediately follows the page header of a data page.
• And last, dictionary compression is used. Dictionary compression searches for repeated values anywhere on a page and stores them in the CI. One of the major differences between prefix and dictionary compression is that prefix compression is restricted to one column, while dictionary compression works anywhere on a data page.

The amount of compression provided by page-level data compression is highly dependent on the data stored in a table or index. If a lot of the data repeats itself, then compression is more efficient. If the data is more random, then little benefits can be gained using page-level compression.

Data Compression Demo

Data compression can be performed using either SQL Server Management Studio (SSMS) or by using Transact-SQL. For this demo, we will take a look at how you can compress a table that uses a clustered index, using SSMS.

Let’s say that we want to compress the Sales.Customer table (which has a clustered index) in the AdventureWorks database. The first step is to right-click on the table in SSMS, select “Storage,” and then select “Manage Compression.”

Figure 1: SSMS can be used to manage compression.

This brings up the Data Compression Wizard, displayed below.

Figure 2: The Data Compression Wizard, or Transact-SQL commands, can be used to manage data compression.

After clicking “Next,” the wizard displays the following screen, which allows you not only to select the compression type, but it also allows you to calculate how much space you will save once compression has been turned on.

Figure 3: Use this screen to select the compression type, and to calculate how much space will be saved.

First, let’s see how much space we will save if we use row-level compression on this table. To find out, click on the drop-down box below “Compression Type,” select “Row,” and then click “Calculate.”

Figure 4: For this table, row-level compression doesn’t offer much compression.

After clicking “Calculate,” the wizard runs and calculates how much space is currently being used, and how much space would be used after row-level compression. As we can see, very little space will be saved, about 1.6%.

Now, let’s see how much compression savings page-level compression offers us for this particular table. Again, I go to the drop-down menu under “Compression Type,” select “Page,” then press “Calculate.”

Figure 5: Page-level compression is higher than row-level compression.

After pressing “Calculate,” we see that compression has improved greatly, now saving about 20% space. At this point, if you should decide to turn on page-level compression for this table, click on the “Next” button.

Figure 6: The wizard allows you several options in which to turn on compression.

At the above screen, you can choose to perform the compression now (not a good idea during busy production times because the initial compression process can be very CPU and disk I/O intensive), schedule it to run later, or just to script the Transact-SQL code so you can run it at your convenience.

Once you have compressed this table (a clustered index), keep in mind that any non-clustered indexes that this table may have are not automatically compressed for you. Remember, compression is based on a per object basis. If you want to compress the non-clustered indexes for this table, you will have to compress each one, one at a time.

While this wizard helps you to see how much compression either method offers, it does not suggest which compression method should be used, nor does it recommend whether compression should be used at all for this object. As the DBA, it will be your job to evaluate each compressible object to determine if it should have compression enabled or not. In other words, you must decide if the benefits of compression outweigh the negatives.

Summary

In this article, we have learned about the two forms of data compression, and about backup compression. While data compression might seem like a seductive new feature of SQL Server, I highly recommend that it is only used by experienced DBAs. While it offers lots of promise for increased performance, it can just as easily cause performance problems if misused.

This post was updated on 3-13-2012. The original posting can be found here.

I was very surprised to see the results of my latest poll, which asked “When was the last time you tuned your transaction logs?”

According to those DBAs who responded to the poll, about 60% of them said they didn’t know transaction logs could be tuned. And just over 19% said they have never tuned their transaction logs. Sadly, only about 21% said they have tuned their transaction logs in the past.

If you are one of those DBAs who don’t know transaction logs can be tuned, you need to take the time as soon as possible to learn about this topic.Why? First, every data modification made in SQL Server must be logged. Second, no transaction can complete until all of the related data modification are written to the transaction log. And third, writing to a transaction log is a single thread process. On a busy SQL Server instance, writing to the transaction log can become a bottleneck, preventing transactions from being completed on a timely basis, slowing down the entire instance.

There are many ways to tune a transaction log, some very simple, and others slightly more complicated, but it is not beyond the ability of most DBAs. To learn more about transaction log performance tuning, download my presentation slides on “Inside the SQL Server Transaction Log”, which will give you a good start on learning the basics of this important topic.

In SQL Server 2005 and later, there is a database option called “forced parameterization”. When it is turned on, it overrides the default “simple parameterization” normally used by SQL Server. It does this by forcing most SELECT, INSERT, UPDATE, and DELETE statements to be parameterized, potentially boosting performance of databases by reducing the number of query compiles and recompiles that need to occur.

As with many “options” in SQL Server, whether this option should be turned on for a particular database is not a simple decision. So I was curious to see how many DBAs actually employ this option. I ran the above poll on my website, and had two surprises. The first surprise is that about 19% of responders said they use it, which I thought was high. The second surprise is that about 44% of responders have not even heard of this option.

If you are one of the 44% who aren’t familiar with forced parameterization, I suggest you read the following to learn more about it. While this database option can be very handy in specific cases, it can also hurt performance in other databases. If you try this option, be sure to perform a benchmark before implementing it, then another benchmark after implementing it, and seeing if you get the behavior you expect. Of course, perform this experiment in a test environment, not in production.

http://technet.microsoft.com/en-us/library/ms175037.aspx 

http://www.sqlmag.com/blog/sql-server-questions-answered-28/sql-server/forced-parameterizationwhen-should-i-use-it-137149

Indexes can be great for boosting the performance of a query, but if an index is never used, it can drag down an instance’s performance. This is because unused indexes must be maintained just like any index. For example, anytime a row is inserted, updated, or deleted, indexes must be maintained. If a lot of records are updated or inserted into a table, an index may experience page splitting. During index rebuilding or reorganizing, indexes are defragmented. During DBCC CHECK DB, indexes have to be checked for their integrity. During all of these processes, all data modifications have to be logged in the transaction log. And of course, indexes take up space in the data cache and on disk. All of these resources are wasted on unused indexes, and can hurt the overall performance of an instance of SQL Server.

Now imagine for a moment that a database has lots of unused indexes, and some of the indexes have many multiple columns, and thus are very wide? I have seen databases where virtually every column in every table had an index, just because the developer thought the “more indexes, the better”. I have also seen many indexes with 4, 5 6, or even more columns that are part of the index key. In situations like this, unused indexes can be a huge hidden performance problem.

Of course, your databases or not like that? Or are they? When was the last time you evaluated the indexes in your databases to see if they were being used or not? According to a recent poll of visitors on my website, nearly 45% of responders said that they have never dropped an index on a database. This could be for two reasons. First, because they have never looked for unused indexes, or because they have looked, but didn’t find any unused indexes to drop. I am guessing that most of them have never looked. While this seems like a large number, about 55% of the responders have dropped an index in the past year, which means that most DBAs are familiar with the issue of unused indexes.

I think it is important to periodically review all of the indexes in your databases in order to identify unused indexes. With the introduction of the sys.dm_db_index_usage_stats DMV in SQL Server 2005, this has become relatively easy. You can choose to run SELECT * on the DMV to return all the results, although the raw results are hard to read. Instead, you might want to use a more useful script such as the one written by Glenn Berry, or one of the many dozens of similar scripts available for identifying unused indexes by searching the Internet.

When you are evaluating which indexes are not used or not, you must keep in mind that this DMV only tracks data on index usage since the instance was last restarted. For example, if you ran Glenn’s query just after restarting your server, then most of the indexes will have not been used. Instead, you must wait for your server to be up long enough to be representative of how indexes are used in your environment. This might mean that your server needs to be running for a week, a month, even a quarter if you have many reports that are only run at the end of a quarter. If your time frame is not representative, you may end up dropping indexes that are used, but less often. Another option to consider when collecting index statistics usage over time is to run a periodic job that collects the data over any time period you like, and store it in a database for later analysis.

And that brings up another area to discuss. Should you drop indexes that are only being used occasionally? For example, perhaps a report is only run once a month, once a quarter, or even once a year. Should you keep this indexes, or drop them? This is a question I can’t answer for you, as each situation is different. For example, if an index that is used for a once a month report takes 5 seconds with the index, and 30 seconds without the index, is the benefit of that index, saving 25 seconds once a month, worth the overhead of maintaining that index? This is only a question you can answer based on the business needs of your environment.

So if you are one of the DBAs who have dropped an index in the past year, good for you. If you have never dropped an index, then I suggest you run Glenn’s query on your databases and then decide for yourself if you need to drop some unused indexes.

Every since the first file was written to a floppy disk drive using DOS, physical file fragmentation has been a problem. Essentially, when the OS writes a file to a disk subsystem, and if contiguous clusters are not available, they are written elsewhere on disk. So when a file is stored on disk in a non-contiguous manner on a disk subsystem, the file is considered to be physically fragmented. Physical file fragmentation can contribute to an additional load on your I/O subsystem and reduce performance because the disks have to work harder to read and write data. This is because the heads on disk drive have to randomly jump around to different physical locations on the disk to find all of the data, instead of reading and writing the data sequentially, as when the data is contiguous.

Physical file fragmentation can be slight, say when a single file is divided into only a handful of locations on a subsystem. In other cases, a single physical file might be divided into thousands of fragments, resulting in heavy physical file fragmentation. The greater the physical file fragmentation, the harder disks have to work, and the greater overhead that is incurred. Ideally, for best I/O performance, files should be written as contiguously as practical.

How physical file fragmentation affects SQL Server’s performance varies. For example, if a single random read or write needs to be performed, the degree of physical file fragmentation will have little or no effect on I/O performance, as the amount of work to perform a single read or write is not significantly affected by physical file fragmentation. On the other hand, if SQL Server needs to read or write a large quantity of sequential data to or from disk, and the file is heavily fragmented, then the disks will have to randomly read or write the data, even though the data is logically sequenced, resulting in extra work for the disk drives, hurting I/O performance.

Another factor that can affect how SQL Server and the disk subsystem interact is if there is a large cache sitting between SQL Server and this subsystem, or if the disk subsystem in made up of SSD or similar drives. Both of these can significantly mask the negative effects of physical file fragmentation, although it is not eliminated.

So what does all of this mean to DBAs? Because of the complexities of measuring the negative impact of physical file fragmentation on SQL Server’s performance, I make the assumption that it is always bad and do my best to prevent it in the first place. As a DBA, I always assume the worse and plan for it.

Here’s what I like to do. Before I create any new database (MDF, NDF, or LDF files), I first use the Windows defrag.exe tool to see if there is any physical file fragmentation on the disk array where I want to create my database files. If there is, then I use defrag.exe, or a third-party tool, to defrag the physical files before creating my new database. Then, I pre-size my database files to a size that I think will suffice for the next year or so. For example, if I estimate that my MDF file will be 500 GB in size in the next 12 months or so, then that is the size I create the database now. The same with the other database files.  By pre-sizing my database files, auto growth doesn’t have to kick in, which is one of the biggest causes of disk fragmentation. In other words, by creating a large file on a disk array that is defragmented, the file is created contiguously and I can prevent physical disk fragmentation from occurring in the first place.

Now, you may have two questions. First, you may be wondering about how you go about defragging a disk array that is currently being used with production databases. The problem is, is that the defrag.exe program will only defrag closed files, not open files, such as any SQL Server MDF, NDF, or LDF files that are in production. So the dilemma is, you have identified physical file fragmentation on an existing disk array where you want to create a new database, and you want to defragment it first before creating the new database to eliminate any physical file fragmentation. Assuming that you are using defrag.exe, your only option is to shut down the production databases during a maintenance window, perform the defrag, then restart SQL Server. Yes, this is a pain, but this one time pain will not only remove any existing physical file fragmentation in your current databases, it will help prevent it from future databases. As the DBA, you have to decide if the trade-off in taking the server down for a while, and potential performance gains it can provide, is worth it.

The second question you may have is how do you know what size you should pre-size your database files when you first create them? This requires experience and an educated guess. The odds are that your guess will not be exact. You will either underestimate or overestimate the size of the database 12 months in the future. There is no way to know for sure. But if you make an attempt using your best guess, your guess will be much better than letting auto growth grow the database for you, which will most likely result in physical file fragmentation. As time passes, and you realize that your guess was wrong, you can always manually grow the database as needed to make more room. Or, if you estimated too much, I would leave the extra room (and not shrinking the file sizes), unless you are in the immediate need of additional disk space. But won’t this also lead to physical fragmentation? Yes, to a limited degree. But most likely, a slight amount of physical file fragmentation will never be noticed.

The times where I have seen physical file fragmentation become a real problem is when an initial database is created with the default settings, where the MDB is set to a 1MB auto growth and the LDF is set to the 10% auto growth. If such a database is not pre-sized, but grows to multiple gigabytes over time, physical file fragmentation can become a substantial problem, especially if there are multiple databases, all using the default setting, on the same SQL Server instance.

In other words, a little physical file fragmentation is nothing to worry about. But a large amount of physical file fragmentation, often due to poor database management, can end up hurting your SQL Server’s I/O performance more than you might expect.

Most of the time when I run a poll on my blog, I run it for at least a month so that I can get enough results in order for the data to be more or less representative of the SQL Server community. What has been odd about this poll, is that even though I have run it longer than any other poll on the website, it has had one of the fewest response rates.

When I look at the results, my best guess is that people who have looked at the poll weren’t familiar with the question, and because of that, ignored responding to it, instead of admitting that they didn’t know what it was. Or, perhaps it is because this feature is only available in SQL Server 2008 and higher, and not everyone is running this version yet.

When you look at the results, just over 36% of the respondents said that they use the “optimize for ad hoc workloads” SQL Server configuration option. Of the 15 or percent that said no, it is hard to know if they are not using it because they don’t know what it is, because they have made a deliberate choice not to use it, or because they don’t have any SQL Server 2008 instances. (That’s one of the limitations of such a simple poll.) About another 10% said “it depends” if they use it or not, and about 38% of the responders admitted not knowing what the feature does. I am guessing that the 38% is understated, assuming my belief that people didn’t respond to this question because they didn’t understand the question. It’s too bad that so many DBAs aren’t familiar with this option, as it is a very simple option that can return big benefits for many SQL Server instances.

For those of you who don’t know, when “optimize for ad hoc workloads” is turned on (it is off by default), SQL Server will only store a small compiled plan stub, not the entire execution plan, the very first time an ad hoc query is compiled for the first time. Should the same batch be executed again, then this time around, the full execution plan will be cached in the plan cache, which is what happens to ad hoc queries when this option is not turned on.

So how is this option useful? If your SQL Server instance executes many ad hoc queries, it is very possible that once the query is executed, it may never be executed again. This can produce what is sometimes called “plan cache bloat”, which means that much of your plan cache is wasted by execution plans that will never be used again, reducing the amount of buffer pool memory available to SQL Server. The “optimize for ad hoc workloads” option helps to prevent “plan cache bloat” by only caching the full execution plan of queries that are known to be used more than once. If “plan cache bloat” is a problem that you have, then turning on this feature can boost the performance of your SQL Server instance.

How do you know if you have the “plan cache bloat” and should turn on “optimize for ad hoc workloads”? The best way is to query sys.dm_exec_cached_plans, using one of the many queries available on the web, such as the one written by Kimberly Tripp. By using the querying this DMV, you will be able to quickly tell if your instance has a “plan cache bloat” problem. If it does, then turn this option on, as there is virtually no downside to doing to, and a great potential upside.

If you have any experience using this option, and would like to share your experiences, please do so below.

Lock Pages in Memory is a setting that can be set on 64-bit operating systems that essentially tell Windows not to swap out SQL Server memory to disk. By default, this setting is turned off on 64-bit systems, and depending on the circumstances, the following conditions could occur when this setting is not turned on:

1) Performance of SQL Server could suddenly decreases for no apparent reason.

2) The hardware running SQL Server could suddenly appear to stop responding for a short period of time.

3) Applications connected to SQL Server could time out.

4) Among many others.

This sudden performance degradation can occur when Windows tells SQL Server that it needs more physical memory, and SQL Server, in response to the request, begins to swap the data out of RAM (such as the data cache) to disk, killing performance. Obviously, this is something that you don’t want to have happen on your production server.

Microsoft, and many SQL Server experts, recommend that in most cases, the Lock Pages in Memory setting be turned on for 64-bit SQL Server instances when running under Windows 2003 and earlier. With this option turned on, SQL Server will refuse Window’s request for giving up its memory, this preventing the problems described above. This problem occurs much less in SQL Server instances running under SQL Server 2008, and because of this, turning on Lock Pages in Memory is most likely not needed, and many SQL Server experts are now recommending that it not be turned on when using Windows 2008 unless you actually experience this problem.

According to the poll, 40% of those responders who have 64-bit systems turn this setting on, while another 30% don’t. I don’t know if the 30% who don’t turn it on are deliberately not turning on for a specific reason (perhaps because they are running under Windows 2008), or they are not turning it on because they don’t know about the potential benefits it offers.

Another 30% haven’t heard of this option, which is a shame, as this setting can have significant performance effects on a server, and DBAs need to know more about it. If you want to learn more about this option, check out these resources.

• How to Reduce Paging of Buffer Pool Memory in the 64-Bit Version of SQL Server
• Lock Pages in Memory: Do You Really Need It?
• How to Enable the Lock Pages in Memory Option

Before turning Lock Pages in Memory on for your SQL Server instance, be sure to carefully read the article “How to Reduce Paging of Buffer Pool Memory in the 64-Bit Version of SQL Server” listed above. It will provide you the information you need to know before you make the decision as to whether or not you want to turn Lock Pages in Memory on for a particular SQL Server instance. Before making any changes in SQL Server, it is very important that you test them before implementing them in production.

When I manage a SQL Server instance, I always make an extra effort to optimize the performance of TEMPDB, so I decided to do a poll to see if other DBAs did the same.

When the results came in, I was a little surprised, as about 91% of those who responded to the poll said that they do take steps to optimize TEMPDB. When you look at the results, you can see that about 9% of the respondents said that they did not take any action to boost the performance of TEMPDB.

The remaining 91% of the respondents had the option to select one or more TEMPDB tuning options. Of the options they selected, three of the options were very close in the number of responses (the top three in the graph), with only the use of solid state storage coming in last. Because the top three options are so close, I am guessing that most of the people who responded to the poll probably perform all three, which is something that I do myself.

Of course, there are many other things you can do to help optimize TEMPDB, the ones listed here are only the tip of the iceberg. If you are one of those DBAs who still haven’t taken any action to optimize TEMPDB on your SQL Servers, you might want to download my presentation (a PDF file) on “How to Optimize TEMPDB Performance”.