8. Mixing Metaphors – We’re Going Down a Rabbit Hole Pam: And then outta that cake pops another stripper holding a smaller cake and then an even smaller stripper pops outta that one. Michael: What is that smaller stripper holding? Pam: A cupcake! It's cupcakes and strippers all the way down.
14. INFORMATION_SCHEMA contains tables and views (ANSI 92, kind of generic RDBMS model, doesn’t expose a lot of implementation details) sys schema contains tables and views (more proprietary, particularly non-portable things like indexing features) SQL Server Basic Metadata Services
15. You could make tables of additional information about database objects DON’T Built-in extended properties work great for this! MS_Description: Microsoft’s standard “description” property sp_addextendedproperty, sp_updateextendedproperty, sp_dropextendedproperty fn_listextendedproperty sys.extended_properties I’ll improve this in the DBMeta schema SQL Server Extended Properties
16. Layer this view on top of sys.extended_propertiesto: Make it easier to use Give a standard naming convention Easier than CROSS APPLY fn_listextendedproperty DBMeta.Properties
17. Easier to use than sp_addextendedproperty, sp_updateextendedproperty, sp_dropextendedproperty The builtinprocs have some awkward hierarchy aspects: e.g. a column or trigger is defined at level 2 where the table is level 1 DBMeta.AddXP, UpdateXP, DropXP
18. With some simple tools we now have an easier way of dealing with the metadata in the extended properties This can be expanded at will to join regular base metadata information, PIVOT with the property information and combine them in interesting ways For instance, say your question was “I want to see NULLability (base metadata) of all money (base metadata) columns in tables in the Accounts subsystem (extended property) which have not been marked as reviewed. (extended property)” You can do that. DBMeta
19. Some questions about the health of the database system will be very standard By health, I mean outside of integrity you can realistically enforce with constraints and beyond what you might use DDL triggers for Can we use the metadata to do something generally useful? Can we organize the health system using metadata so that it is self-maintaining as much a possible? DBHealth
20. Let’s create some data structures in our Demo schema – some tables and indexes Demo Data
21. Arbitrary, but useful and chosen for variety Rule #1: All tables should be identified with a SUBSYSTEM (Organizational Rule) Rule #2: No unique indexes with columns which allow NULLs (Indexing Rule) Rule #3: No varchar(N) columns where N <= 2 (Table Design Rule) Rule #0: We only want to enforce these on schemas under management Let’s Make Up Some DBA Rules
22. We simply make a view DBHealth.MonitoredSchemas We’ll need to remember to use this view You could make a whole layer of views which filter through this, of course Rule #0
23. Look for tables without extended property ‘SUBSYSTEM’ Rule #1
24. Look for unique indexes where any of the columns in the index are NULLable. Rule #2
26. Let’s tag things for exclusion with extended properties! What about exclusions?
27. How do we organize them? Could we put them in individual views or procedures to make them easy to use? Then make a master procedure which runs them all. But we’d need to make a list of all the procedures But what did we say? Let’s use the metadata to tag the procedures! We’ll mark the procedures We’ll categorize the procedures We’ll be able to look for all the marked procedures and run them in an automated fashion – no new tables, limited maintenance! Now we have a bunch of rules
28. All procs in DBHealth should have HEALTH_CHECK_PROC and HEALTH_CHECK_SET properties Show me any unindexed tables (heap tables with no non-clustered indexes) Show me any items tagged with a ‘TODO’ extended property Show me all columns in the database which aren’t actually used (yes, you can do this!) Ideas For Rules
30. Materials http://bitly.com/bundles/caderoux/3 4 Part article by Adam Aspin on SQL Server Central Article by Brent Shaub on MSSQLTips.com Google Code Repository of presentation and code/scripts
First of all, thank you for coming. I’m Cade Roux, I’ve been designing and building software for over 17 years and hopefully I’m going to show you some techniques which can save you time and effort and make your systems better in a lot of different ways.
Archimedes is often associated with work with simple machines and this quote about the lever is attributed to him. With an appropriate tool, you can magnify your strengths and free up time to concentrate on less tractable problems. The analogy with the lever does break down – with a lever, you use a smaller force through a larger distance – with the techniques in this talk, you should use less effort and less time!
My Goal is to demonstrate concrete metadata techniques for SQL Server, simplified from actual production code that: Make for maintainable systems, Are flexible for a variety of problems andExtend your productivity for more leverage – taking simple building blocks and concepts to build larger structures.
My Goal is to demonstrate concrete metadata techniques for SQL Server, simplified from actual production code that: Make for maintainable systems, Are flexible for a variety of problems andExtend your productivity for more leverage – taking simple building blocks and concepts to build larger structures.
My Goal is to demonstrate concrete metadata techniques for SQL Server, simplified from actual production code that: Make for maintainable systems, Are flexible for a variety of problems andExtend your productivity for more leverage – taking simple building blocks and concepts to build larger structures.
My Goal is to demonstrate concrete metadata techniques for SQL Server, simplified from actual production code that: Make for maintainable systems, Are flexible for a variety of problems andExtend your productivity for more leverage – taking simple building blocks and concepts to build larger structures.
As you can imagine, this talk is going to be a bit recursive. Metadata is data about data. In this talk I’ll be showing tactics that use metadata to manage metadata… We normally build software in layers. I’m going to be talking about using metadata from one layer to support the next layer. We’ll be using metadata to do code generation, to inspect the database, and to generally automate tasks using the built-in features of the datatabase.
Although using metadata and code generation can be faster and more efficient than manual coding, using the proper lifting technique can also be safer as well by eliminating defects through consistently generated and re-generated code, exception reports and self-maintaining subsystems.
Most what I am going to be talking about is just effective use of features which have been around a long time to solve problems which have been around for a long time in a consistent, simple and coherent fashion.
Most what I am going to be talking about is just effective use of features which have been around a long time to solve problems which have been around for a long time in a consistent, simple and coherent fashion.
I’d like to lay out some simple database ground rules for this demo. Schemas are simply a container within a database, a feature added in SQL Server 2005. I’m simply going to have one schema for “user” data – Demo. I’m going to have a schema called DBMeta for some general metadata utilities and a schema called DBHealth which will contain things which depend upon DBMeta. So if you were to do something like this in your own database, DBMeta is standalone and DBHealth would require DBMeta to exist and provide those services. You want to avoid building anything with circular dependencies – that’s certainly possible in SQL Server, so you need to keep a clear conceptual model and manage this yourself. It will only be apparent when you go to deploy and script and the object give warnings that their dependencies don’t exist. You could probably use metadata techniques to detect cycles in a directed dependency graph you could extract from the metadata.Run Script LT-01.sql
I’m going to assume a pretty basic SQL Server platform for this demo: tables (with indexes) in schemas, views on top of tables, and stored procedures using those tables and views. I’m not really going to address user-defined functions, but these techniques can also be used there.
So SQL Server provides basic metadata services required by the relational model. This is the active online catalog required by Codd. You have the INFORMATION_SCHEMA which has been standardized by ANSI, and there are also the proprietary versions which have shifted around a little over the years with the different versions.
When we talk about systems, the built-in metadata is kind of limited. Say you need to categorized every table into different types, or you need to make a list of stored procs or some kind of dependency tracking which isn’t really part of a traditional database schema element like a constraint or something.You could make tables and design whole systems for tracking that. One problem with that is you either have to use DDL triggers or manual maintenance to ensure it stays updated when a table is added or removed.So in addition to the built-in metadata, you’ve also got very powerful ad-hoc extended properties which can be attached to most objects in the database. I am only going to be using character strings (varchar) in my extended properties. In actual fact, you can use ANYTHING (sql_variant) in extended properties (up to 7500 bytes of data).
sys.extended_properties is a convenient view, but it doesn’t expose everything in really useful form. This view on top helps. It might seem to be a lot of special cases, but that’s the same way you have to use sys.extended_properties anyway.Run Script LT-02.sql
These SPs simplify calling the underlying SPs. I built this on top of a table-valued function to eliminate repeating some code.Run Script LT-03.sql
With some simple tools we now have an easier way of dealing with the metadata in the extended propertiesThis can be expanded at will to join regular base metadata information, PIVOT with the property information and combine them in interesting waysFor instance, say your question was “I want to see NULLability (base metadata) of all money (base metadata) columns in tables in the Accounts subsystem (extended property) which have not been marked as reviewed. (extended property)” You can do that.
So let’s move on to trying to enable some way of looking for problems in the database – mainly design and implementation problems. I’m assuming that the database is doing everything you have told it – so it’s enforcing referential integrity where you’ve specified. But we’re talking meta – so, for instance, can it tell you all the tables you’ve forgotten to enforce referential integrity even though you have a primary key?
Run Script LT-Demo-01.sql
So let’s make up some rules and try to start checking the database using the metadata. These are arbitrarily chosen as examples, but they’re real-world examples.Rule #1 is that we’re going to specify that the database objects are documented beyond just having the standard MS_Description. We want every table to be identified with a SUBSYSTEM. Say you have a largish system, and you need an HR subsystem and an ACCOUNTING subsystem within the same database. You might use schemas for this or a prefix naming system, so it’s kind of self-documenting, but let’s say you didn’t and you need to retro fit this to help organize and inventory your database.Rule #2 is that say that if you have a unique index, you want to know if any of those columns are nullable. This should be a pretty rare possibility, but SQL Server allows it.Rule #3 is that we want to look for short character columns which don’t need the overhead of varchar.Finally, for Rule #0, we want to go back and say, let’s only check these rules on schemas that we want to manage.
We simply make a view DBHealth.MonitoredSchemasWe’ll need to remember to use this view when ever we are looking at objects to make sure they are in a monitored schemaYou could make a whole layer of views which filter through this, of courseSo let’s look at the view.Run Script LT-04.sql
We’ll write the query in ad hoc form. Then productionize it into an SP.Run Script LT-05.sql
We’ll write the query in ad hoc form. Then productionize it into an SP.Run Script LT-06.sql
We’ll write the query in ad hoc form. Then productionize it into an SP.Run Script LT-07.sql
Now let’s change that one a little bit, since it’s finding something we don’t want to change and there’s nothing like false positives to make a warning system easily ignorable.Run Script LT-08.sql
So now we have a bunch of rules. Great! How do we organize them?Could we put them in individual views or procedures to make them easy to use? Then make a master procedure which runs them all.But we’d need to make a list of all the procedures. But what did we say (don’t make more tables to keep track of tables!)? Let’s use the metadata to tag the procedures! We’ll mark the procedures We’ll categorize the procedures.We’ll be able to look for all the marked procedures and run them in an automated fashion – no new tables, limited maintenance!Let’s see how to do this.Run Script LT-09.sql
Now you can get all kinds of ideas for rules.
Health Monitoring in .NET – HealthMonitorFormCode Generation in T-SQL – LT-Demo-02.sql, LT-Demo-03.sql, LT-Demo-04.sqlCode Generation in T4 – DemoApp – GenerateLookups.tt, DumpLookups.tt
