Wednesday, February 20, 2013

Generating and Sorting on a Transient Property in a Django Model Class

I ran into an interesting little issue in a Django application today that led to what I thought was some pretty powerful stuff in Python and Django that I hadn't had to use before, so I thought I'd share.

I'm working on an application that to keep it generic I'll simply say deals with requests from users, and these requests require various levels of approval depending on the type and severity of the request.

Here's a basic Django model class that contains the fields relative to this example:


Basic stuff so far.

Where the wrinkle comes in is there is a page in the application that lists the requests ordered by date, but the date by which each request will be displayed and sorted depends upon the impact of the request as follows:
  • minor: use datetime_supervisor_approved
  • major: use datetime_admin_approved
  • severe: use datetime_president_approved
To state the issues succinctly:
  • The date on which I want to sort varies for each request
  • I don't want to store an additional, redundant field in the database simply to have a consistent name to use for sorting
  • Because the sort date isn't stored in the database I can't use order_by on a QuerySet
That last bullet was the killer. In Python it's simple enough to add attributes to a class on the fly, so I could loop over the QuerySet and use conditional logic around the change impact to add a new calendar_date field to each instance of the UserRequest class, but that's kind of ugly because the business logic winds up in a view function as opposed to being in the model, and still doesn't solve the inability to use order_by in the QuerySet.

Because I want to keep the business logic in the model (fat models FTW!), I looked into using a Python property to add a transient attribute to my UserRequest class that is the result of a function call.

Basically what this means in concrete terms is I'm adding a calendar_date attribute to my model class and the value of calendar_date is set by calling a function in the class itself that contains the aforementioned conditional logic around the request impact.

Here's the modified model class:


The big addition here is the _get_calendar_date function that returns a calendar date based on the impact of the request. This is added as an attribute to the class, but it's not saved to the database which is exactly what I wanted in this instance.

That solves one piece of the puzzle, but the other piece is sorting the requests based on this new transient property, which again can't be done by using order_by on the QuerySet since the field on which we want to order isn't in the database.

This is where Python's sorted() function comes in. sorted() can take any iterable (which the Django QuerySet is) and sort it based on a provided comparator function, and this is where I can leverage the transient calendar_date property to sort the requests.

Putting all the pieces together in the view function, here's how it looks:


Python's sorted() function is pretty powerful and straight-forward -- throw it an iterable and what to compare on, and it does the heavy lifting for you.

The only potentially tricky part of this is setting the key since it uses Python's lambda statement (great explanation here), which is Python's way of kinda sorta dipping a toe into the functional programming waters by letting you define an anonymous inline function.

In this case how the lambda plays out is to sort the QuerySet based on two attributes: first the transient calendar_date property that isn't stored in the database, and subsequently the request_impact attribute that is stored in the database. Perfect!

I'm sure none of this is rocket science to seasoned Python and Django veterans but since I hadn't run into the need to do this before I thought I'd document what I came up with both for my own future reference and hopefully for the benefit of others who made need to so something similar.

Sunday, February 17, 2013

Installing and Configuring NextPVR as a Replacement for Windows Media Center

If you follow me on Google+ you'll know I had a recent rant about Windows Media Center, which after running fine for about a year suddenly decided as of January 29 it was done downloading the program guide and by extension was therefore done recording any TV shows.

I'll spare you more ranting and simply say that none of the suggestions I got (which I appreciate!) worked, and rather than spending more time figuring out why, I decided to try something different.

NextPVR is an awesome free (as in beer, not as in freedom unfortunately ...) PVR application for Windows that with a little bit of tweaking handily replaced Windows Media Center. It can even download guide data, which is apparently something WMC no longer feels like doing.

Background

I wound up going down this road in a rather circuitous way. My initial goal for the weekend project was to get Raspbmc running on one of my Raspberry Pis. The latest version of XBMC has PVR functionality so I was anxious to try that out as a potential replacement for WMC.

Installing Raspbmc was easy enough and the base install is working fine, but based on what I've read it may not (yet) be up to functioning as a full-blown PVR like I'm used to, and will likely serve better (as some folks on Google+ suggested) as a front-end to something else.

That led me to searching around for a WMC replacement that will run on Windows 7.

Given my huge preference for free (as in freedom) software why continue to use Windows and not move to something like MythTV? Honestly I'll probably go the MythTV route eventually, but in the near term I was just looking for something that would quickly allow me to schedule and record shows again. I'm running WMC on a very nice Acer Revo, which has a DVD player form factor and is designed to function specifically as a media center, complete with a nice pop-out dual-function mouse/keyboard pad. I'll go the full free software route another weekend.

As I looked around I came across this article on Lifehacker about turning a Raspberry Pi into a media center, and it too was focused on the Pi as a front-end to another machine that does the heavy lifting. Honestly I like this idea anyway -- sticking a quiet, low-power Raspberry Pi behind every TV other than the one with the NextPVR box seems like a slick solution.

Installing NextPVR

Installing NextPVR is very straight-forward -- I mostly just did what's outlined in the Lifehacker article.
  1. Download the latest version
  2. Download the patches and copy them into the directory in which you installed NextPVR
  3. Start NextPVR and adjust the settings
The specifics of that last step will be dependent upon your hardware. Note that the default buffer directory of C:\Temp may not exists so if you get an error related to that when you fire up NextPVR, just create that directory and restart NextPVR. I also did enable the background recording service as recommended.

In my case I'm using a SiliconDust HDHomeRun as my tuner, which means I do not have a tuner card in the media center PC. The HDHomeRun makes three tuners available over ethernet, and NextPVR sees them out of the box, though there is some configuration and installation of codecs necessary to get things working properly.

Watching Live TV on NextPVR

Next you'll want to configure NextPVR to be able to watch live TV, and the first step in this process is to have NextPVR scan your tuner devices to get a channel listing.

With the HDHomeRun I simply had to go into devices, choose each tuner, and click the "Scan" button. I have digital cable so this pulled back a little over 500 channels. Note that you have to do this for each tuner. You can tell it to copy a configuration from one tuner to another, but I found that to be much slower than performing the scan on each one individually.

The first problem I ran into after the channel scan is that although NextPVR sees the HDHomeRun fine, when I tried to watch live TV I got no picture or sound. I could see that it was talking to the HDHomeRun since I saw the tuner light come on and it was going through the motions of changing channels, and even had the channel names correct, but there was no picture or sound.

If you have this issue don't panic; it's very likely you just need to go into the NextPVR settings and choose the correct decoder. If you don't have valid MPEG2 or AC3 codecs installed, which are the video and audio formats the HDHomeRun uses, you'll have to do that separately.

In my case to get the video working I had to enable the MPEG2 and H.264 video decoders, and for those I selected the "Microsoft DTV-DVD Video Decoder" that was already installed on my machine.

As you change these settings, make sure and restart NextPVR. I'm not sure that's supposed to be required but with both audio and video I had to restart NextPVR for the changes to take effect.

To get the audio working, I had to install additional codecs since I didn't have any on the machine that could handle AC3 audio. For MPEG1 audio the Microsoft DTV-DVD Audio Decoder should work (though I can't confirm since I'm not sure I tested any MPEG1 audio specifically), but it definitely didn't work for me for any of the other audio types.

Some suggestions for fixing this issue were to install Windows Media Center since it comes with codecs that won't otherwise be on the machine, but from what I could tell anything WMC installs is pretty MS-specific, so you're better off installing more agnostic codecs.

Luckily there are some excellent free (as in beer) codecs available. What I ended up installing was the Windows 7 Codec Pack. This comes with codecs for just about every audio and video format you can imagine and gave me what I needed to fix the lack of audio.

One the Windows 7 Codec Pack was installed, I selected "ffdshow Audio Decoder" for all the audio types other than MPEG1, and after restarting NextPVR I was getting audio. Again I think the only audio type I was really verifying was AC3 since that's what the HDHomeRun uses, so if you have a different setup that requires different decoders, you may need to try some different options.

Downloading Program Guide Data (EPG) and Mapping Channels

With video and audio both working there was one last problem to solve, namely program guide data. Without guide data you won't know what's on and, as I discovered the hard way with WMC, you won't be able to record anything.

In the case of the guide data I decided to throw a little money at this problem. You don't necessarily have to pay for program guide data but from the little bit of research I did it seems like the free solutions would require some ongoing interaction on my part (or some scripting), and I don't want to turn recording TV into another job for myself, so I decided to pay for it. If you want to investigate solutions like XMLTV and others, the NextPVR wiki page on program data (EPG) is a great place to start.

Specifically, I decided to pay $25/year for a Schedules Direct account. It integrates completely seamlessly with NextPVR and based on what I read was well worth the money compared to dealing with this yourself. (I'm not averse to a little scripting and hacking but I have enough side jobs at the moment!)

After getting a Schedules Direct account, you then create a listing in Schedules Direct, which is done by providing your zip code and choosing your cable provider.

Once the listing is created in Schedules Direct you then go into the Channels section of NextPVR, enter your Schedules Direct login information, pick a listing, and then program guide data will be continually updated for you.

That's not quite the end of the story though. You still have to map the channels so NextPVR knows which channel in NextPVR goes with which channel in the lineup.

This is where I think NextPVR could use a little bit of work. You can do a bulk mapping, which works pretty well overall (though verify because it did mismatch some in my case), but if you do the bulk mapping and manually change any specific mappings it asks you every time you change one if you want to have it scan and try to do a bulk match even after it already did that. Minor annoyance but seems like it'd be an easy fix as well.

With all that in place, in the Channels section of the NextPVR setting I clicked the Update EPG button and a couple of minutes later, program guide data was available. (Take that, Windows Media Center!) Then I could finally go in and start recreating the season passes I lost when I tried reinstalling WMC.

As for the Raspberry Pi ...

Since I basically took a huge detour from my original goal of getting XBMC up and running on my Raspberry Pi, that piece isn't quite done yet. Raspbmc itself is running fine, and I installed the PVR Client for NextPVR, but it's not working properly for some reason so I still need to dig into that.

Once I get that figured out then I can have the Raspberry Pi hooked up to a TV in another room and watch either live TV or recordings from the NextPVR box, which will be very slick. I'll do another post on that once I get it working.

Conclusion

Overall I'm quite impressed with NextPVR so far. The installation was quite simple, it saw the HDHomeRun without any problems, and the codec issues I ran into were very easy to solve.

The UI may not be quite as slick, strictly speaking, as WMC, but it's also a lot cleaner and zippier, and I'll take that any day. With the guide data in place I'm very happy with the functionality and don't think I'll miss anything at all from WMC.

Longer term I'll definitely look into MythTV so I have a free as in freedom solution, but this was a quick and easy project that at least got me up and running again after the WMC meltdown.

The first real test of my new setup will be when NextPVR records The Walking Dead tonight. As WMC found out if it screws that up, it's history.