Systematically Tackling Performance Issues in Your Code

“I just got this report from this monitoring service and the new page takes 20 seconds to load!”

Almost all of us in software have had someone complain that a web page or part of a program takes too long. The information thrown at us is not always clear, and someone wants it fixed. And they want it fixed now. Resist the temptation to create a knee-jerk solution and remember to tackle this problem systematically.

Verify The Current State

First, get your hands on the performance report they’re talking about. Continue reading

Tips for Good Entity Framework Code First Models

One of the things in programming that you should get right the first time is code first models. Refactoring a database is way more work than normal code, and there will be cases where it’s so much work that it won’t be worth it.

So here’s a few tips from my recent (sometimes painful) refactoring work.

  1. Don’t think about how your model will look as a table, too much. Continue reading

My Entity Framework Code First Checklist

How many times have we all worked on some technology and then wanted to use it again on a different project but couldn’t figure out how to get started? We’ve all done it, and Entity Framework with Code First is definitely one of those technologies that is hard to get started.

…unless you have a checklist. That’s why I’m writing this. It’s my checklist. It helps me start new projects and hopefully it helps you, too. Continue reading

Using Entity Framework 6 with an existing database

DungeonMart’s starter data set is designed for SQL Server, so it’s time to leave DocumentDB behind and start the SQL work. First, I need to create the SQL database and load the seed data. The schema files that I’m using for the initial load can be found here: https://github.com/qanwi1970/dnd-srd-35/tree/master/V1. If you want to follow along, just run them against a new database and you’re good to go.

Since I already have a database, I’m going to use Entity Framework’s database first model to get talking to it quickly. Continue reading

Creating Automated Integration Tests for a DocumentDB Repository

You can unit test all your layers, but when you get to the repository it becomes an exercise in mocking that provides little real benefit. That’s why we have to give up on the unit test idea for repositories and embrace integration tests. There’s nothing wrong with integration tests, as I explained a while back. So let’s review the criteria of effective automated tests and see if this is worth while.

  1. Automated – Yes, my tests will be standard NUnit tests that can be run just about anywhere by just about any automated process. Continue reading

Refactoring My Repository Methods to be Asynchronous

Now I’ve written a DocumentDB based repository for Equipment data in DungeonMart, but I didn’t try to make it perfect, I just tried to make it work. Here it is, if you’re catching up: Part 1 and Part 2.

My next step in getting it closer to perfect is to make all the methods aynchronous. Face it, we’re going out to the cloud to get or write data and we shouldn’t make the whole service wait while that happens. DocumentDB gives you Async methods at least for writing, and we can figure out how to make the read methods async. If we were in Entity Framework, we would already have async methods for both reading and writing.
Continue reading

Creating a DocumentDB Repository Layer (Part 2 – Writing)

“Last time, on Bill DeLude dot com…”

Hmm, it kind of loses its effect without a good narrator voice. Does someone have Morgan Freeman’s number? Because he could say that last time I talked about creating a data repository class using DocumentDB as the data store and covered the setup of the class and the read methods. This time, I’m going to go over the write methods.

But first a disclaimer before we get started. Not everything in this class is complete. As with normal programming, I like to take a highly iterative approach. First, I’ll make it work. Then I’ll improve some part of it. And then I’ll improve some other part, and so on. There’s two things conspicuously missing from today’s code. First, the write methods, as they are offered up, are synchronous. Second, it doesn’t allow for the Equipment type to change. I’ll address both of these in a later post.

First, I’m going to work on adding Equipment to the collection. Here’s the implementation of the AddEquipment method:

public Equipment AddEquipment(Equipment equipment)
{
    dynamic doc = Client.CreateDocumentAsync(CollectionLink, equipment)
        .Result.Resource;
    Equipment result = doc;
    return result;
}

Adding a document to a collection in DocumentDB is pretty straight-forward. All you really have to do is call CreateDocumentAsync with the link to the collection and the document you’re adding.

I made it a little more complicated because I like to return the added document back to the caller, so they can have any changes such as the id or a timestamp. To accomplish that, I need to get the Resource of the CreateDocumentAsync result as a dynamic type. Then I put that into an Equipment type and return the equipment.

Now that we can add documents, let’s make sure we can update them. Updating in DocumentDB is a two step process. You first have to get a document link by reading the collection, and then you replace the document with your changed version. Gone are the days of “UPDATE Equipment SET …” because DocumentDB, as with most NoSQL databases, just doesn’t work that way. Here’s the code for updating:

public Equipment UpdateEquipment(string id, Equipment equipment)
{
    var doc = Client.CreateDocumentQuery(CollectionLink)
        .AsEnumerable().First(d => d.Id == id);
    dynamic updatedDoc = Client.ReplaceDocumentAsync(doc.SelfLink,
        equipment).Result.Resource;
    Equipment result = updatedDoc;
    return result;
}

Notice that when I get the document this time, I’m not specifying a type to return. By omitting the return type, I get it as a Document and not as an Equipment. I want to do that because Equipment doesn’t have the self link I need to do the replace on the next line. Once you have the document, replacing it is pretty simple. Just call ReplaceDocumentAsync with the document’s self link and the new document. Returning the updated equipment requires some extra work, but it’s just like the Create.

Finally, we need to delete documents from the collection. This works a lot like updating: get a self link, then call a delete method. It looks like this:

 public void DeleteEquipment(string id)
{
    var doc = Client.CreateDocumentQuery(CollectionLink)
        .AsEnumerable().First(d => d.Id == id);
    Client.DeleteDocumentAsync(doc.SelfLink);
}

And that’s it. Now you have a full blown Equipment repository using DocumentDB as the data store.

The code that was used at the time of this writing is in my github at this commit.

Creating a DocumentDB Repository Layer (Part 1 – Reading)

Even with NoSQL, having a repository layer is a good idea. In the world of DocumentDB, writing this layer is really simple. First, we’ll start with an interface, of course. In your application, this interface might already exist.

public interface IEquipmentRepository
{
    IEnumerable<Equipment> GetEquipments();
    Equipment GetEquipmentById(string id);
    Equipment AddEquipment(Equipment equipment);
    Equipment UpdateEquipment(string id, Equipment equipment);
    void DeleteEquipment(string id);
}

As, you can see, this is a pretty straight-forward interface with CRUD (Create-Read-Update-Delete) operations. Notice that the identifier is a string, not an integer like you might be used to seeing. DocumentDB, like other document databases, likes the identifier to be a GUID. Don’t try to make it the .NET GUID type. Just go with string, and then the database will handle it just like SQL does with integer identifiers.

If you’re following along in your own code, you’ll need the Equipment model. Here’s a snapshot of what it looked like when this post was written: Equipment.cs.

The next thing would be to implement the repository, but before you do that you’ll need a couple of extension methods to make your life easier. You can find them in my previous post on getting existing databases and collections. Do that, and then come back. I’ll be right here. You can also look at the source for the whole project, using the link at the bottom.

You’re going to need two fields in your repository: the DocumentClient and the Document Link. This is how I’m getting them at the time I write this. It isn’t optimal, but I have plans for more posts about making that better and those topics are beyond the scope of this post. So here it is:

private static readonly DocumentClient Client =
    new DocumentClient(new Uri(<Your Url>), <Your Key>);

private static readonly string DocumentLink =
    Client.GetOrCreateDocumentCollectionAsync("dmart", "equipment")
    .Result.DocumentsLink;

Now that we have those, let’s implement the first method, GetEquipments. There’s not much to it, but there’s a lot to talk about:

public IEnumerable<Equipment> GetEquipments()
{
    return Client.CreateDocumentQuery<Equipment>(DocumentLink)
        .AsEnumerable();
}

This method uses the DocumentClient we created as a field. It calls the CreateDocumentQuery<T> method to query the document collection referenced by the DocumentLink (our other field).

Notice that the method returns an IEnumerable<T> and not a List<T>. What this means is that when the method returns, we haven’t actually queried the collection. We’re putting that off until later to allow the caller to filter on the IEnumerable. This makes the usage of our get list method more flexible, and possibly saves us from unnecessarily passing a huge collection across the wire.

However, in some cases there will be a way of filtering that is so common we’re just going to provide the users of our class a built in method for that. The seminal example would be wanting to get an Equipment by it’s identifier. For that, we’ll implement the next method.

public Equipment GetEquipmentById(string id)
{
    return Client.CreateDocumentQuery<Equipment>(DocumentLink)
        .AsEnumerable().First(d => d.id == id);
}

It looks a lot like the last method, except that it filters and returns a single Equipment object. Notice that I’m using First instead of FirstOrDefault, which will throw an exception if the Equipment with that guid doesn’t exist. Since a guid is a really hard key to fake, I don’t expect that will happen much and an exception is probably appropriate. I might regret that decision later.

That’s it for reading from your DocumentDB collection. In my next post, I’ll go over the remaining write methods.

The code used for this post lives here:
https://github.com/qanwi1970/dungeon-mart/tree/3ad6081b36b5fc804b0c6dce03daa1a2699edd87

Continue with the writing methods in Part 2.

Update: The code when I published was missing the .AsEnumerable() in the GetEquipmentById method. Sorry about that. By the time I found and fixed it, the code had moved on and would just be confusing to point to, now.

Really, How You Start a Project in Visual Studio Can Mean So Much

You know what I hate? Being well into a project and realizing that you started the wrong type or picked the wrong scaffolding options. Some things can be very difficult to change later. Also, one of my pet peeves is bloated projects. It’s crazy that a new WebAPI project will have styles, javascript files, areas, and a we-only-supported-it-for-one-release help page. In my drive to avoid these issues, I pretty much have project creation down to a science.

You might have your favorite way of creating projects, and I’d love to hear about them. Give me ideas in the comments. But for right now, this is how I normally go about a web site. The DungeonMart website it going to use Web API 2 for data services, AngularJS for UI, and Azure’s DocumentDB for the database (for now, at least).

Start with a web application.

Create a new Web Application

This one looks simple, but there’s so much you can mess up here. Name and Location are important. You can move the whole solution if you don’t like the location, but changing the name is really annoying. I also like to uncheck “Create directory for solution” so I get a flatter folder structure, and I’m already in a source controlled folder so I’m not going to add source control again.

Remember, Templates are bad!

Select Template

These templates are where all the bloat that I hate so much comes from. So I always pick empty, and then add the core references I want using the checkboxes below the template list. Since the UI will be AngularJS, and not MVC with Razor, I did not choose the MVC checkbox.

I did check the unit tests box, but won’t be doing that again. In VS2012, it would put the unit test project folder in the same sub-folder as the main project folder, but VS2013 is placing the unit test project in the MyDocuments project path. So I had to remove the unit test project and create a new one to get it in the right folder.

If you choose to host in the cloud…

dmart3

… then make sure you pick the right name here. It’s one of those things that is really, really hard to change later.

What’s next?

Well, now you have an empty project waiting for you to add stuff. If you’ve used templates before, you’ll notice that a lot of things are missing: Areas, Scripts, Content, HomeController, four of the App_Start files, billions of Nuget dependencies, and so on. You have to add the things you’re going to want yourself, as you need them. Probably the first thing would be a start page, and then your first ApiController, but I’ll get into that next time.

Welcome to DungeonMart! Can I Interest You in a Longsword?

I need a web app, something with all kinds of webappy features that can serve as my go to project when I’m writing about different technologies. I’ve actually needed this for a while, but have been struggling to come up with the concept.

I tried for a content management system. Then I thought about a customer management system. I thought about building out that online retailer system that I dreamed up back when I was an Amazon/Ebay seller. The main thing lacking in all these was a mature domain model, due to my lack of experience with the subjects.

Then, in a completely unrelated-to-this-topic search, I stumbled upon a complete database of the D20 System Reference Document (SRD). You would probably know it as something else, but I’m not allowed to use that term. Anyway, it’s a database of character types, monsters, and items and equipment that can be used in various table-top gaming systems, like the kind where you poke around in dungeons and fight mighty dragons. This works out great for me, for a couple of reasons.

  1. The data set is very mature, having evolved from a system that was created in the 70’s.
  2. The SRD is open content. The license is full of all kinds of legalese, but I know from other sites that I can do what I need to do with it.
  3. The data is highly relational, but is represented in the database as unrelated tables. This gives me opportunity to model how one would refactor a site or a data model after the site has gone live, the whole changing-the-engines-while-in-flight problem.

Thus DungeonMart was born, a modern day e-commerce site where an adventurer could go to buy all the known adventuring equipment and items. It could be anything from a pound of wheat to a magical suit of chainmail. It’s nothing special yet, just a page that says Welcome. But as I evolve it over time, it should be pretty cool.

Technology wise, I’m first going to play around with getting the data into DocumentDB, while it’s still in preview on Azure and because I’m possibly going to be doing a session on DocumentDB at an upcoming event. Once the data is in place, I’ll probably progress to a Web API 2 RESTful service and then finally the actual web site part.

I’m sure I’ll get distracted many times on the way as I need to talk about one thing or another, but it will slowly come together. The source will be on my github the whole time, and the site will live on Azure probably for a while, here.