Making sure your Umbraco site performs on Azure

Azure Web Apps (previously known as Azure Websites) is an excellent service that can be used to host your Umbraco website in a cost effective way. I will not go into deploying your site to Azure, this is something that can be done in different ways and works exactly the same for Umbraco as it would work for any other ASP.NET website. Google it.

Instead, this post focuses on making sure your performance on Azure is optimized.

Sync Examine indexes

First and foremost there's something you need to understand about how the server structure is set up behind Azure Web Apps (AWA). The two most important servers in the setup are the Web Worker (the machine that has IIS on it which is hosting your site) and the File Server. Whenever a new website on AWA spins up, it gets assigned to a Web Worker which has enough capacity to run another web site on it, there may be dozens / hundreds of other sites running on it. Sometimes the Web Worker needs maintenance at which point your site will be re-allocated to another available Web Worker and your site will move there. I say "move" but there's one important thing to understand here: the files used in your website live on the File Server and not on the Web Worker. All that the Web Worker does is it sets up a network path (UNC path) and tells IIS: there you go, my files are here. This points to the File Server, not to a path on the local machine, even though a UNC path very much behaves like a local path.

Why is this important? One word: latency. Every time the Web Worker gets instructed to change a file on disk it has to reach out over a network connection to a different server and make that change. This is a relatively slow process. You will not notice this latency when doing "normal" operations like, for example, saving a template in Umbraco.

However, where you will notice this latency is when it comes to disk operations that need to be highly performant. One of the best examples here in both read and write latency are the Examine indexes. Umbraco comes with a Lucene.Net implementation called Examine and it is fast.. really fast. It can be this fast because it relies on speedy access to it's data, the stored indexes on disk. Each time you ask for a media item, a query gets sent off to Examine to find the media item quickly. Each time you save a document in Umbraco, the saved information gets sent into Examine to be indexed. Each time you build a custom Examine searcher and use it.. you get the idea.

So, long story short: you want your Examine indexes to live on the local disk of the Web Worker instead of on the remote File Server. This can be done as of Umbraco 7.2.8 by adding the following attribute to all of your indexers and searchers: useTempStorage="Sync".

Update: it has come to my attention that, for now, it's best not to include the machine name in your ExamineIndex.config, so if you've previously added that you might want to consider removing that for the time being. Note that your indexes will be rebuilt when you do this, so if this is a long process make sure to plan for some downtime. More information can be found in this forum topic.

The 'Sync' setting will store your indexes in ASP.NET's temporary file folder which is on the local file system (so: on the Web Worker). Any time the index is updated, this setting will ensure that both the locally created indexes and the normal indexes are written to. This will ensure that when the app is restarted or the local (Web Worker) temp files are cleared out that the index files can be restored from the centrally stored index files (on the File Server). If you see issues with this syncing process (in your logs), you could change this value to be 'LocalOnly' which will only persist the index files to the local file system in ASP.Net temp files.

Setting the indexes to "Sync" will ensure that your websites starts as fast as it can after it's been moved to a new Web Worker, the indexes will be copied from the File Server to the Web Worker during application startup. If you set the setting to "LocalOnly" your indexes will only ever exist on the Web Workers and thus need to be rebuilt every time your site is moved to a new Web Worker. This could work just fine for most sites but there's no performance to be gained from it and if you have a large index, this may slow down your website on startup significantly as it needs to rebuild all of the indexes from scratch.

FcnMode

Shannon has written an in-depth blog post about File Change Notifications already so I'm not going to repeat that. We have shipped with `fcnMode="Single"` since Umbraco 7.3.5 and you should also set that attribute on your system.web/httpRuntime element in your web.config.

We have made this change to all of our Web Workers on Umbraco as a Service too and to quote Shannon from that blog post above:  

To solve this issue we changed fcnMode=”Single” in the machine.config so that all sites would effectively use “Single”… and the result was instant: No more constant app restarts, file server performance was instantly back to normal. And as far as I can tell, there has been no downside to running FCNMode in Single.

That blog post is now 6 months old and with ~2000 sites still happily running at great performance we're now confident that fcnMode="Single" is the only way to run any ASP.NET website. If you're on Umbraco versions older than 7.3.5, you can just add this attribute yourself, it will work on any version of Umbraco.

Logs

Umbraco uses Log4Net for logging debug information and errors. A few people with very active sites have found that the logger started slowing down the website so much it was easily noticeable and the performance hit could be tracked back to the logger. This occurred even though the logger was asynchronous and "shouldn't" have affected website performance. Since version 7.3.0 we've been shipping with a new logger that should take more of the stress off of your website. Even so, we do log a lot of information and all that logging can quickly start taking up a lot of disk space. 

We recommend at least for your live website to set the Log Priority to "WARN" instead of the default "INFO" and when the site has been running well for a while and you're comfortable you could even swith to "ERROR".

Debug

It's your live environment. So what do you do in your web.config? Yes, indeed, you set debug mode to "false" and customErrors mode to "RemoteOnly". Right? Right! You also built your dlls in Release mode before deploying them, right? Right!

What else

Far be it for me to say that you might have produced some badly performing code. You're a professional programmer who has been at it for years and you know exactly what you're doing, right? Well on the off chance that you had a minor glitch in your brain, maybe you should check to make sure.

The biggest offense we see these days is people using the ContentService and other Services in their frontend queries. Please, please don't do this, it goes straight to the database and kills your performance, especially if you make queries as crazy as: find the root node and then query all of it's descendants (that's ALL nodes in Umbraco!) to find the property "thisIsAnAmazingArticle" and then ToList() it and show only the first article with this property set to true. Stop. :)

Another thing that seems to be a pattern (and it's certainly an anti-pattern) is that people make the exact same mistake that I once made for Our Umbraco's download section: I needed a download counter.. so, hey I know: let's add a property "counter" on each Download Document Type and save & publish the Download page each time someone downloads this version, increase the counter with 1. Trust me, this doesn't scale. Ever. Please do not put counters on your document types and increase them programmatically. It's just as much work to create a simple table with a `nodeId` and `count` column and update that one with a bit of PetaPoco.

Caching can be super useful if you know that you need to do something that you can't optimize the performance of right now (or ever). Use Umbraco's Macro caching, or Umbraco's ApplicationCache, or implement Donut Caching, etc. Even a very short cache of a few minutes can make a world of difference if you have 10 requests per second doing the exact same work 10 times. Note, however, that caching really should be a last resort after you've tried to optimize as much as you can. With caching comes cache invalidation and that can be hard or even dangerous.

Also, I won't JUST blame you: we have had some issues with Umbraco performance and some, let's say "interesting" bugs in older versions. Make sure to upgrade to the latest version of Umbraco that you can upgrade to. If you have to stick with patch upgrades then 7.2.8 and 7.3.8 are both considered stable, but if you can do minor upgrades, definitely consider going to at least 7.3.8. Of course if you like receiving regular update, then going to version 7.4.2 and keeping up with the latest versions would be much preferred (remember, I blogged about upgrading Umbraco recently as well).

Microsoft also makes mistakes, this one is a pretty huge bug and you need to be aware of it and manually install the patch if you're using Windows Server 2012R2.

Conclusion

This post is partly the result of questions and answers gathered over the last few months in this forum post, it might be good to read through that to find some more specific examples that might apply to your situation. I hope this helps you optimize you Azure hosted Umbraco site (pro-tip: a lot of these tips also help on any other hosting provider).