Download Source Code
Introduction
Sitemaps can be a useful tool to help search engines like Google find all the content on your website.
Normally, search engines find pages by scanning the links in the pages they already know about. So if, for example, your home page is indexed by Google and you add a new page on your site, Google will find the new page if your home page contains a link to it.
For the most part, this works well. But sometimes your content is not that readily linked. For example, I recently developed a website that was region-based. I could've make a complex hierarchy of folders for each region. But in the end I decided it provided a better user experience if I just had the user select their region on the first visit, and then track their selection in a browser cookie on future visits.
While I'm happy with this design, it doesn't play nice with search engines. When a search engine crawls the web, it doesn't make selections from lists or process cookie requests and so it would not be able to find my regional content. So I needed a sitemap that I could submit to the major search engines so they could easily find all my pages.
While I could've manually created a sitemap file and placed it on my site, this is a tedious process. Moreover, I'm constantly adding new content to my site. A manually created sitemap would be a headache to keep current and would be prone to typing and other errors.
Fortunately, most of my content is related to data in a database. So the obvious answer is to automatically generate the sitemap on the fly. I could then submit the sitemap URL to major search engines, and I would generate the file any time they (or anyone else) requested that URL. This way, the sitemap would always be current and would not be subject to typing errors or omissions.
Sitemaps
All major search engines appear to support the XML schema for the Sitemap protocol described at http://sitemaps.org/protocol.php.
Basically, a sitemap is an XML file. It uses a simple format: the root node <urlset> contains any number of <url> nodes.
Each <url> node describes a single page with several sub-elements. <loc> specifies the URL location of the page. <lastmod> specifies the last modification date. And then, optionally, you can include <changefreq>, which provides a hint about how often the page might be updated, and <priority>, which provides a hint of how important the page is in relation to other pages on your site.
I should take a moment to mention that ASP.NET projects can also use another type of sitemap. This type of sitemap is used for ASP.NET navigation. ASP.NET sitemaps are in a different format than search engine sitemaps. Although you may be able to write code that combines the two types somehow, this article will deal only with search engine sitemaps.
Looking at the Code
Listing 1 shows my Sitemap.aspx page. In the attached sample project, I use ASP.NET 4 page routing to map www.domain.com/Sitemap to www.domain.com/Sitemap.aspx. Remember, a sitemap is an XML file. You can't name the file XML unless you map handlers for that file type. For my purposes, Sitemap works just fine, although Sitemap.aspx seemed a little questionable.
The page's Load event handler starts by declaring an array of sample data, which is read in a loop. Normally, you would replace the loop with code that reads data from your database. I used the sample data so I could demonstrate my techniques.
The code needs to produce the content of an XML file so it starts by calling Response.Clear() and sets Response.ContentType. This clears any non-XML data that may have been emitted before the Load event was called.
My code then establishes an instance of XmlTextWriter and associates it with the response output stream.
Listing 1: Sitemap.aspx
<%@ Page Language="C#" %>
<%@ Import Namespace="System.Xml" %>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<script runat="server">
// Class for sample data (replace with actual data)
class DataItem
{
public string Slug { get; set; }
public DateTime LastUpdate { get; set; }
}
protected void Page_Load(object sender, EventArgs e)
{
// Populate sample data
List<DataItem> items = new List<DataItem>();
items.Add(new DataItem() {
Slug = "sample-page-slug",
LastUpdate = new DateTime(2011, 1, 7) });
items.Add(new DataItem() {
Slug = "happy-new-year-2011",
LastUpdate = new DateTime(2011, 1, 1) });
items.Add(new DataItem() {
Slug = "merry-christmas",
LastUpdate = new DateTime(2010, 12, 25) });
items.Add(new DataItem() {
Slug = "happy-thanksgiving",
LastUpdate = new DateTime(2010, 11, 25) });
items.Add(new DataItem() {
Slug = "happy-birthday-washington",
LastUpdate = new DateTime(2010, 2, 15) });
items.Add(new DataItem() {
Slug = "happy-new-year",
LastUpdate = new DateTime(2010, 9, 11) });
items.Add(new DataItem() {
Slug = "another-article",
LastUpdate = new DateTime(2010, 7, 14) });
items.Add(new DataItem() {
Slug = "heres-another-article",
LastUpdate = new DateTime(2010, 3, 8) });
items.Add(new DataItem() {
Slug = "april-fools-day",
LastUpdate = new DateTime(2009, 4, 1) });
items.Add(new DataItem() {
Slug = "heres-an-article",
LastUpdate = new DateTime(2009, 2, 12) });
items.Add(new DataItem() {
Slug = "sample-article",
LastUpdate = new DateTime(2009, 1, 7) });
// Clear any previous response
Response.Clear();
Response.ContentType = "text/xml; charset=utf-8";
//
XmlTextWriter writer = new XmlTextWriter(Response.OutputStream, Encoding.UTF8);
writer.Formatting = Formatting.Indented;
writer.WriteStartDocument();
// Set list of URLs (urlset)
writer.WriteStartElement("urlset");
writer.WriteAttributeString("xmlns", "http://www.sitemaps.org/schemas/sitemap/0.9");
// Home page
writer.WriteStartElement("url");
writer.WriteElementString("loc", "http://www.domain.com");
writer.WriteElementString("lastmod", DateTime.Today.ToString("yyyy-MM-dd"));
writer.WriteElementString("changefreq", "always");
writer.WriteElementString("priority", "1.0");
writer.WriteEndElement();
foreach (DataItem item in items)
{
writer.WriteStartElement("url");
writer.WriteElementString("loc",
String.Format("http://www.domain.com/Articles/{0}", item.Slug));
writer.WriteElementString("lastmod", item.LastUpdate.ToString("yyyy-MM-dd"));
writer.WriteElementString("changefreq", "monthly");
writer.WriteElementString("priority", "0.8");
writer.WriteEndElement();
}
// Close urlset
writer.WriteEndElement();
// Close document
writer.WriteEndDocument();
writer.Flush();
writer.Close();
// Terminate response
Response.End();
}
</script>
<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
<title></title>
</head>
<body>
<form id="form1" runat="server">
<div>
</div>
</form>
</body>
</html>
The code above adds the home page, and then loops through the data to add each page associated with records in my database. Of course, you would customize this for your site. In my production code, I ended up not listing all my physical pages mostly because not all my physical pages were important. You'll need to decide which pages to include this way.
Conclusion
That's about all there is to it. Like my production code, the sample project imlements page routing so that the sitemap is at www.domain.com/Sitemap (with no extension). If you download the project, you can see how it's set up before deciding how you would customize it for your own site.
End-User License
Use of this article and any related source code or other files is governed
by the terms and conditions of
.
Author Information
Jonathan Wood
I'm a software/website developer working out of the greater Salt Lake City area in Utah. I've developed many websites including Black Belt Coder, Insider Articles, and others.