Strate SQL » Deadlocks

Index Those Foreign Keys

jstrate — Fri, 18 Jun 2010 21:00:59 +0000

Today started with some quality time getting to know a deadlock that had occurred. While working through the deadlock, I noticed that there were a number of foreign key relationships that weren’t indexed on the parent side of the relationship.

I am going to skip over the why to index foreign keys and save that for a later point when I have more time to go through it with some really pretty pictures. Today though, I want to share the scripts that I put together to look for these situations and help prevent issues related to them.

Brute Force Indexing

This first script is a brute force attack on this need. If you set the output for text results in SQL Server Management Studio (SSMS) you’ll get a script with all of the indexes you’ll need to cover all of your foreign key relationships.


SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED
SET NOCOUNT ON

;WITH cIndexes
AS (
SELECT i.object_id
,i.name
,(SELECT QUOTENAME(ic.column_id,'(')
FROM sys.index_columns ic
WHERE i.object_id = ic.object_id
AND i.index_id = ic.index_id
AND is_included_column = 0
ORDER BY key_ordinal ASC
FOR XML PATH('')) AS indexed_compare
FROM sys.indexes i
), cForeignKeys
AS (
SELECT fk.name AS foreign_key_name
,'PARENT' as foreign_key_type
,fkc.parent_object_id AS object_id
,STUFF((SELECT ', ' + QUOTENAME(c.name)
FROM sys.foreign_key_columns ifkc
INNER JOIN sys.columns c ON ifkc.parent_object_id = c.object_id AND ifkc.parent_column_id = c.column_id
WHERE fk.object_id = ifkc.constraint_object_id
ORDER BY ifkc.constraint_column_id
FOR XML PATH('')), 1, 2, '') AS fk_columns
,(SELECT QUOTENAME(ifkc.parent_column_id,'(')
FROM sys.foreign_key_columns ifkc
WHERE fk.object_id = ifkc.constraint_object_id
ORDER BY ifkc.constraint_column_id
FOR XML PATH('')) AS fk_columns_compare
FROM sys.foreign_keys fk
INNER JOIN sys.foreign_key_columns fkc ON fk.object_id = fkc.constraint_object_id
WHERE fkc.constraint_column_id = 1
UNION ALL
SELECT fk.name AS foreign_key_name
,'REFERENCED' as foreign_key_type
,fkc.referenced_object_id AS object_id
,STUFF((SELECT ', ' + QUOTENAME(c.name)
FROM sys.foreign_key_columns ifkc
INNER JOIN sys.columns c ON ifkc.referenced_object_id = c.object_id AND ifkc.referenced_column_id = c.column_id
WHERE fk.object_id = ifkc.constraint_object_id
ORDER BY ifkc.constraint_column_id
FOR XML PATH('')), 1, 2, '') AS fk_columns
,(SELECT QUOTENAME(ifkc.referenced_column_id,'(')
FROM sys.foreign_key_columns ifkc
WHERE fk.object_id = ifkc.constraint_object_id
ORDER BY ifkc.constraint_column_id
FOR XML PATH('')) AS fk_columns_compare
FROM sys.foreign_keys fk
INNER JOIN sys.foreign_key_columns fkc ON fk.object_id = fkc.constraint_object_id
WHERE fkc.constraint_column_id = 1
), cRowCount
AS (
SELECT object_id
,SUM(row_count) AS row_count
FROM sys.dm_db_partition_stats ps
WHERE index_id IN (1,0)
GROUP BY object_id
)
SELECT
'--Missing foreign key index for '+fk.foreign_key_name+CHAR(13)+CHAR(10)+'GO'+CHAR(13)+CHAR(10)+
+'CREATE NONCLUSTERED INDEX FKIX_'+OBJECT_NAME(fk.object_id)+'_'+REPLACE(REPLACE(REPLACE(REPLACE(fk.fk_columns,',',''),'[',''),']',''),' ','')
+CHAR(13)+CHAR(10)+
+'ON [dbo].['+OBJECT_NAME(fk.object_id)+'] ('+fk.fk_columns+')'+CHAR(13)+CHAR(10)+
+'GO'+CHAR(13)+CHAR(10)+CHAR(13)+CHAR(10)
FROM cForeignKeys fk
INNER JOIN cRowCount rc ON fk.object_id = rc.object_id
LEFT OUTER JOIN cIndexes i ON fk.object_id = i.object_id AND i.indexed_compare LIKE fk.fk_columns_compare + '%'
WHERE i.name IS NULL
ORDER BY OBJECT_NAME(fk.object_id), fk.fk_columns

Foreign Key Monitoring

This second script accommodates for those situations when you may not want to just index every foreign key that is out there. Maybe there’s a really old table in the database with a foreign key relationship that just doesn’t matter any more. Is it worth indexing along a vector that won’t lead to any performance impact – either negative or positive? Most likely not.

For this script, the results output a list of foreign keys relationships that are not fully indexed. Included in the result script is a column with XML data in it that contains a script for creating an index. You may notice that the format for this is very similar to the schema created when outputtingmmissing Indexes from execution plans.


SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED

;WITH cIndexes
AS (
SELECT i.object_id
,i.name
,(SELECT QUOTENAME(ic.column_id,'(')
FROM sys.index_columns ic
WHERE i.object_id = ic.object_id
AND i.index_id = ic.index_id
AND is_included_column = 0
ORDER BY key_ordinal ASC
FOR XML PATH('')) AS indexed_compare
FROM sys.indexes i
), cForeignKeys
AS (
SELECT fk.name AS foreign_key_name
,'PARENT' as foreign_key_type
,fkc.parent_object_id AS object_id
,STUFF((SELECT ', ' + QUOTENAME(c.name)
FROM sys.foreign_key_columns ifkc
INNER JOIN sys.columns c ON ifkc.parent_object_id = c.object_id AND ifkc.parent_column_id = c.column_id
WHERE fk.object_id = ifkc.constraint_object_id
ORDER BY ifkc.constraint_column_id
FOR XML PATH('')), 1, 2, '') AS fk_columns
,(SELECT QUOTENAME(ifkc.parent_column_id,'(')
FROM sys.foreign_key_columns ifkc
WHERE fk.object_id = ifkc.constraint_object_id
ORDER BY ifkc.constraint_column_id
FOR XML PATH('')) AS fk_columns_compare
FROM sys.foreign_keys fk
INNER JOIN sys.foreign_key_columns fkc ON fk.object_id = fkc.constraint_object_id
WHERE fkc.constraint_column_id = 1
UNION ALL
SELECT fk.name AS foreign_key_name
,'REFERENCED' as foreign_key_type
,fkc.referenced_object_id AS object_id
,STUFF((SELECT ', ' + QUOTENAME(c.name)
FROM sys.foreign_key_columns ifkc
INNER JOIN sys.columns c ON ifkc.referenced_object_id = c.object_id AND ifkc.referenced_column_id = c.column_id
WHERE fk.object_id = ifkc.constraint_object_id
ORDER BY ifkc.constraint_column_id
FOR XML PATH('')), 1, 2, '') AS fk_columns
,(SELECT QUOTENAME(ifkc.referenced_column_id,'(')
FROM sys.foreign_key_columns ifkc
WHERE fk.object_id = ifkc.constraint_object_id
ORDER BY ifkc.constraint_column_id
FOR XML PATH('')) AS fk_columns_compare
FROM sys.foreign_keys fk
INNER JOIN sys.foreign_key_columns fkc ON fk.object_id = fkc.constraint_object_id
WHERE fkc.constraint_column_id = 1
), cRowCount
AS (
SELECT object_id
,SUM(row_count) AS row_count
FROM sys.dm_db_partition_stats ps
WHERE index_id IN (1,0)
GROUP BY object_id
)
SELECT
fk.foreign_key_name
,OBJECT_NAME(fk.object_id) AS fk_table_name
,fk.fk_columns
,rc.row_count AS row_count
,CAST(' +CHAR(13)+CHAR(10)+'GO'+CHAR(13)+CHAR(10)+
+'CREATE NONCLUSTERED INDEX []'+CHAR(13)+CHAR(10)+
+'ON [dbo].['+OBJECT_NAME(fk.object_id)+'] ('+fk.fk_columns+')'+CHAR(13)+CHAR(10)+
+'GO'+CHAR(13)+CHAR(10)+'--?>' AS xml) foreign_key_index_schema
FROM cForeignKeys fk
INNER JOIN cRowCount rc ON fk.object_id = rc.object_id
LEFT OUTER JOIN cIndexes i ON fk.object_id = i.object_id AND i.indexed_compare LIKE fk.fk_columns_compare + '%'
WHERE i.name IS NULL
ORDER BY OBJECT_NAME(fk.object_id), fk.fk_columns

Closing Up

The DDL schema output in these scripts is very basic. It doesn’t account for potentially important things like partitions and filegroups. Obviously, you’ll need to modify this for your own environment and don’t just run this on production.

I see a lot of potential in these scripts and am planning to include them as part of preparing for releases when I am clients. A good way to dot the i’s and cross the t’s.

Individual results may vary. No Legos were harmed in the writing on this post.

Does Your Stored Procedure Grant Itself Permissions?

jstrate — Thu, 03 Dec 2009 13:00:00 +0000

It’s a very good question. One that might not seem to insidious. Nothing that should be able to bring down the system and cause failures. Or will it?

I’ve been to a number of clients and done it myself before where I start to check out a stored procedure with some performance issues and sitting all pretty at the bottom is a GRANT EXEC statement. When I script out the stored procedure I get something similar to the following:

CREATE PROCEDURE dbo.FooGetTableA
    (
    @Parameter varchar(4)
    )
AS
 
SELECT Column1 
FROM dbo.TableA
WHERE Column2 = @Parameter
 
GRANT EXEC ON dbo.FooGetTableA TO ApplicationRole
GO

But if you look carefully, there is something missing, or one could say included that shouldn’t be. Look again if you don’t see it. It’s hidden in plain sight. The permissions for the procedure are included in the body of the stored procedure. When the procedure was written, someone thought ahead to add permissions to the script but forgot the GO statement between the stored procedure

In a better world this script would have looked like this:

CREATE PROCEDURE dbo.FooGetTableA
    (
    @Parameter varchar(4)
    )
AS
 
SELECT Column1 
FROM dbo.TableA
WHERE Column2 = @Parameter
GO
 
GRANT EXEC ON dbo.FooGetTableA TO ApplicationRole
GO

It’s Just a Permission Statement

Who cares, right? So you are assigning some permissions every time that procedure executes. What harm could possibly come of it. I’ve seen this so many times and usually it’s one of things I’ll point out and say, “oops, you should take care of that”. When I should be saying, “yeah, fellas. You’ve got a time bomb there waiting for your business to take off.”

And the time bomb is deadlocks. Completely preventable deadlocks.

If you have procedures that grant themselves permissions, then as the volume of activity in your database increases you may start to see deadlock graphs similar to the following:

deadlock-list deadlock victim=process30108bac8  process-list   process id=processec55dd68 taskpriority=0 logused=0 waitresource=METADATA: database_id = 10 PERMISSIONS(class = 1, major_id = 219199881) waittime=15000 ownerId=746424569 transactionname=Load Permission Object Cache lasttranstarted=2009-10-22T23:06:59.287 XDES=0x3712a8e98 lockMode=Sch-S schedulerid=1 kpid=5832 status=suspended spid=157 sbid=2 ecid=0 priority=0 transcount=1 lastbatchstarted=2009-10-22T23:06:59.287 lastbatchcompleted=2009-10-22T23:06:59.280 clientapp=.Net SqlClient Data Provider hostname=PRDWB0111 hostpid=5640 loginname=portaluser isolationlevel=serializable (4) xactid=746424394 currentdb=10 lockTimeout=4294967295 clientoption1=671088672 clientoption2=128056    executionStack     frame procname=AdventureWorks2008.dbo.FooGetTableA line=1 sqlhandle=0x03000a0089b9100d0e527800669c00000100000000000000CREATE PROCEDURE dbo.FooGetTableA    (    @Parameter varchar(4)    )AS

SELECT Column1 FROM dbo.TableAWHERE Column2 = @Parameter

GRANT EXEC ON dbo.FooGetTableA TO ApplicationRole    inputbufProc [Database Id = 10 Object Id = 219199881]       process id=process30108bac8 taskpriority=0 logused=0 waitresource=METADATA: database_id = 10 PERMISSIONS(class = 1, major_id = 1746157316) waittime=2125 ownerId=746479249 transactionname=Load Permission Object Cache lasttranstarted=2009-10-22T23:07:12.180 XDES=0x3786c61c8 lockMode=Sch-S schedulerid=3 kpid=4048 status=suspended spid=69 sbid=2 ecid=0 priority=0 transcount=1 lastbatchstarted=2009-10-22T23:07:12.180 lastbatchcompleted=2009-10-22T23:07:12.167 clientapp=.Net SqlClient Data Provider hostname=AMBER hostpid=568 loginname=portaluser isolationlevel=serializable (4) xactid=746372404 currentdb=10 lockTimeout=4294967295 clientoption1=671088672 clientoption2=128056    executionStack     frame procname=AdventureWorks2008.dbo.FooGetTableB line=1 sqlhandle=0x03000a00043f146882564201a09b00000100000000000000CREATE PROCEDURE dbo.FooGetTableB    (    @Parameter varchar(4)    )AS

SELECT Column1 FROM dbo.TableBWHERE Column2 = @Parameter

GRANT EXEC ON dbo.FooGetTableB TO ApplicationRole    inputbufProc [Database Id = 10 Object Id = 1746157316]      resource-list   metadatalock subresource=PERMISSIONS classid=class = 1, major_id = 219199881 dbid=10 id=lock4153ec880 mode=Sch-M    owner-list     owner id=process30108bac8 mode=Sch-M    waiter-list     waiter id=processec55dd68 mode=Sch-S requestType=wait   metadatalock subresource=PERMISSIONS classid=class = 1, major_id = 1746157316 dbid=10 id=lock415451780 mode=Sch-M    owner-list     owner id=processec55dd68 mode=Sch-M    waiter-list     waiter id=process30108bac8 mode=Sch-S requestType=wait

Breaking It Down

When I first started looking at these there are a few things I noted right away:

The procedures were access completely different tables with no common objects between them. In the sample above there is TableA and TableB and no relationship.
Looking at each of the processes in the deadlock both of them have the following attributes

waitresource=METADATA: database_id = 10 PERMISSIONS
transactionname=Load Permission Object Cache

So nothing in common and a deadlock on a metadata resource for permissions. This made me start to re-think how the two procedures were related. With a metadata resource wait, there seems to be an issue above the data in the table. Since both procedures point to the Load Permission Object Cache, maybe there is an issue there.

If you take a look, each of the procedures has a GRANT EXEC permission statement in it. This is the area of commonality and where the two executions deadlocked. Removing the GRANT EXEC permissions statements stop this deadlock from occurring.

After going through and removing these permission statements from a number of procedures that had this issue, all of the deadlocks with these types of issues disappeared. And it is smooth sailing once again.

Cautionary Tale

Hopefully this is a scenario that only I’ve run into. But if it’s not then this should serve as a reminder that little details that seem like a little non-issue, could be the crack that breaks the damn when there’s enough water behind it. The thing that gets you on this issue is that it isn’t until execution start to really grow before it pops out and it will only hit when you’re the busiest. This is something I’ll be keeping an eye out for in the future and I’d recommend the same for others as well.

Deadlocks on exchangeEvent and threadpool

jstrate — Wed, 02 Dec 2009 13:00:00 +0000

I got to work with deadlocks quite a bit recently. There were quite a few interesting ones that came up that I had the chance to research. Since I like easy, I’ll start with the one that I forgot to grab the deadlock details for.

Well, maybe not all of the details… in this case as the title states I was looking at deadlocks with the events exchangeEvent and threadpool. I managed to come across a post from Bart Duncan that went through and deciphered this deadlock. The long and the short of it… parallelism deadlocks.

Bart does a better job explaining this than I can do here, especially since I didn’t take the time to grab the deadlock details for review. Maybe I’ll have that one the next one…

Fortunately, a large part of the issue that I was reviewing for the client had to do with parallelism and so solving this issue actually occurred as a side effect of dealing with parallelism issues. But I will share my little secret that I used to resolve this and most of the parallelism…

Indexes!!

There I said it. True, you can have too many indexes. But no indexes is too few. No clustered indexes can lead to too many scans. I could pulpit here on indexes and making sure that you have them, but I’ll save that for another time.

Overall, I used Bart’s Workaround #1. Hopefully this helps… direct you to a post that is more prescriptive.

Deadlock Resources

jstrate — Mon, 17 Sep 2007 17:02:00 +0000

Deadlocks are a not so wonderful unnaturally occurring event that all DBAs will eventually have the pleasure to take a look at. Since deadlocks are time senisitive it is important that at the time of the deadlock the correct mechanisms are in place to capture the detail of the deadlock in the SQL Server error log.

For SQL Server 2000 the trace flags to use are 1204 and 3605. And for SQL Server 2005, you can use the same flags or one up them with 1222 which produces similar results but in a much cleaner output.

Since there are different trace flags between 2000 and 2005 there are of course different attack plans for resolving deadlocks. A couple of the better links for SQL Server 2000 are:

Some SQL Server 2005 resources:

But really the key to deadlocks is really not having them in the first place:

Reducing SQL Server Deadlocks

That’s all for now…

At least I figured out the deadlocks I was working on this morning…