Track Nuget Downloads using OData, RSS and Ifttt.com

28. November 2011 11:24 by Matt Wrock in   //  Tags:   //   Comments

In this post I am going to show you how you can be notified of new downloads of any Nuget package via email from a service that will poll Nuget every 15 minutes. If email sounds overly intrusive, there are other options. So If this sounds interesting, read on.

If you have open source projects hosted on Nuget and you are a bit on the OCD (obsessive compulsive disorder) side, you are frequently tempted to wander over to Nuget.org and check out your download stats. Well, I have finally started to notice that I spend a fair amount of time every day, checking the Nuget site as well as other sites that may provide key indicators of my projects’ health. For instance I like to check for new followers or twitter mentions. Unfortunately, a lot of this time is spent simply waiting for a web page to load and reveal to me that there is no new information. So not only is this unproductive but it can lead to negative thoughts and emotions.

There are many pharmaceutical options available here, but I am not a Medical doctor and it would be unwise for me to give advise of a psychiatric nature. However I do have some technical solutions that simply require a computer with access to the world wide web. If you lack either of these, I have nothing to offer and you should now leave this page.

Ok. good. It’s just you and me now….hmm…this is uncomfortably intimate. No matter…

Switch from a Pull to a Push model

What I found myself craving was a way to let all of this information come to me and announce to me that there is new data rather than me having to spend time pinging several sources for what is likely to be no new information. In my case, I really wanted my phone to beep or vibrate when I get a new download, follower or mention. For me, this would not be a nuisance given the small amount of data. If you owned jQuery, you may want a more unobtrusive notification. Fortunately the solution I am about to propose can channel notifications through a variety of mediums.

Enter If-this-then-that ifttt.com

A few months ago I noticed a new referring link on my blog from a domain called ifttt.com. I visited the link and perused the site and discovered that it provided a way of creating sort of mash ups of various social media. ifttt stands for If This Then That. And the site simply allows you to create rules of If something occurs (new tweet, RSS feed item, DropBox item, etc.) Then some other thing should be triggered such as an email sent or a tweet or facebook update, etc. I have to admit my initial impression was “That’s dumb.” Then about a week later Scott Hanselman blogged about this service having been duly impressed by its offerings. I still didn’t really get it.

Not sure why I didn’t see the value right away but I see it now. Last week I setup a number of tasks that have freed me of the constant compulsion to check these various web sites for new data. I have a rule that will send me an email whenever my project has a new Github follower or a new mention on twitter. I have tasks that tell me when I have new stack overflow comments or new stack overflow points. All of these tasks were relatively easy to set up using ifttt.com. ifttt’s very friendly GUI provides an extremely simple way to send an email to yourself triggered by a new tweet or RSS feed item.

Here is an example of the task that sends me an email when my project RequestReduce is mentioned on twitter:

image

It is honestly trivial to set this up.

But Nuget Has no RSS Feed with items representing downloads

Currently Nuget provides no RSS feed or any notification option for subscribing to download stats beyond what is displayed on the project search results and individual project details pages. I don’t know if there are plans to implement this by the Nuget team in the near future, but I wanted something up and running soon that didn’t need to be polished.

All Nuget data iavailable from the website is exposed through an OData feed

I knew that the data I was interested in was available via OData. There are a few posts out there that talk about this. I found that David Ebbo’s post had the detail I deeded to get started. With the name of any Nuget package Id, you can get its total download count via the public Nuget OData endpoint at http://packages.nuget.org/v1/FeedService.svc.

Here is an example query using LinqPad:

image

 

Creating a custom RSS Feed to broadcast new downloads

Currently as far as I can tell, there is no facility built into ifttt to consume this OData format. Yes, you can expose OData as an ATOM feed but given the Nuget schema, this would only be useful if you wanted to be notified of new versions. Essentially each version is a child entity of the master Packages entity. DownloadCount is simply a property associated with each version. Note that a version has both a VersionDownloadCount and a DownloadCount. The first is simply the count for a single version and the latter is the aggregate count of all combined versions released in a single package.

At first I tried playing with Yahoo Pipes and some other online RSS builder apps but none of these was going to work. At least not simply. I didn’t want to spend a lot of time on this since what I wanted was really quite simple and could be coded up fairly trivially. So I ended up just writing my own feed generator and I took the opportunity to create my first Azure application. I plan to blog more specifically on the azure specific details later and how they differed from my work with an AppHarhor application.

Here is the RSS Generator code:

public class FeedHandler : IHttpHandler
{
private const string NugetServiceUri = "http://packages.nuget.org/v1/FeedService.svc";
private readonly IDictionary<string, IList<SyndicationItem>>
packageDownloadCounts = new ConcurrentDictionary<string, IList<SyndicationItem>>();

public bool IsReusable
{
get { return true; }
}

public void ProcessRequest(HttpContext context)
{
var packageName = context.Request.QueryString["packageId"];

var nugetContext = new Nuget.GalleryFeedContext(new Uri(NugetServiceUri));
var last = (
from x in nugetContext.Packages
where x.Id == packageName && x.IsLatestVersion
select new { x.DownloadCount, x.Version }).First();

var items = GetSyndicationItems(packageName, last.DownloadCount);
var nugetUrl = string.Format(
"{0}/Packages(Id='{1}',Version='{2}')", NugetServiceUri, packageName, last.Version);

var feed = new SyndicationFeed("Nuget Download Count Feed",
"Provides the current total download count for a Nuget Package",
new Uri(nugetUrl), nugetUrl, items.Last().LastUpdatedTime,
items);
using (var xmlWriter = XmlWriter.Create(context.Response.OutputStream))
{
feed.SaveAsRss20(xmlWriter);
xmlWriter.Flush();
xmlWriter.Close();
}

context.Response.ContentType = "text/xml";
context.Response.End();
}

private IList<SyndicationItem> GetSyndicationItems(string packageName, int count)
{
IList<SyndicationItem> items;
lock (packageName)
{
if (packageDownloadCounts.ContainsKey(packageName))
items = packageDownloadCounts[packageName];
else
{
items = new List<SyndicationItem>();
packageDownloadCounts.Add(packageName, items);
}
var title = string.Format("{0} has {1} total downloads", packageName, count);

if (!items.Any(x => x.Title.Text == title))
items.Add(new SyndicationItem(
title,
"",
new Uri(string.Format("http://nuget.org/packages/{0}",
packageName)), Guid.NewGuid().ToString(),
new DateTimeOffset(DateTime.UtcNow)));
while (items.Count > 20)
items.RemoveAt(0);
}

return items;
}
}


You can grab the full Visual Studio Solution from https://github.com/mwrock/NugetDownloadFeed. Not much happening here. Its just a handler that takes a packageId in the query string and then checks the odata feed to see if there are more downloads than there were since the last time it checked. If there are, it creates a new feed item.

ifttt.com will poll this feed every 15 minutes. I currently have this feed up and running at http://wrock.cloudapp.net/downloadFeed.axd. Anyone is free to use it but I provide no guarantee for stability or longevity. That said, I have no plan to change the endpoint or bring it down. However, I may clean the code up a bit and offer it as a Nuget package so that anyone can host their own feed.

Consuming the feed from an ifttt.com Recipe

Beyond the creation of “one off” tasks. ifttt provides a means of encapsulating common task logic into a reusable “Recipe.” These are handy if you find yourself creating the same task again and again with the only difference being a single variable. In my case here, I wanted to create three tasks. One for each of my Nuget projects. It also seemed reasonable that others may want to make use of this as well. So I created a recipe that anyone can use in order to create their own Nuget Download Notification task. Simply create an ifttt account (Super fast and easy to do) and go here: http://ifttt.com/recipes/9302.

image

As the directions state, simply replace my Package Id RequestReduce with the Package Id that you are interested in.

If you do not want to be notified by email, you have several different options. You could have it tweet from a specific account, send an SMS message or create an Evernote entry. And there are many more options than that.

I’d really like to hand it to the folks at @ifttt for creating this ingenious service and wish them the best of success!

blog comments powered by Disqus

About Me

Hey thats me!

I'm Matt Wrock with over fifteen years of experience architecting scalable, distributed, high traffic web applications as well as environment and deployment automation. I currently live in Woodinville, WA with my wife, two daughters, four dogs and two cats. Until just recently I worked for Microsoft as a Sr. Software Engineer and now work for CenturyLink Cloud focusing on data center automation. I'm also project founder of http://Boxstarter.org and a committer to http://chocolatey.org.

Month List