I'm putting the finishing touches on my latest Django site (a registration site for a dog rescue charity walk) and part of the functionality is the ability for users to upload photos. Since most people will simply snap a photo on their cell phones and upload it, and since cell phones these days take some pretty large pictures, I want to resize the images to a standard size of 800x600 when they're uploaded and also create a thumbnail.
As with most things in the Python/Django world there's a library that does exactly what I need, specifically django-stdimage, which provides a custom model field to handle resizing photos and creating thumbnails.
After installing django-stdimage via pip, updating my model class to use the new form field, and then doing a South migration, I ran into this error:
! Cannot freeze field 'supporter.registrant.photo'
! (this field has class stdimage.fields.StdImageField)
! South cannot introspect some fields; this is probably because they are custom
! fields. If they worked in 0.6 or below, this is because we have removed the
! models parser (it often broke things).
! To fix this, read http://south.aeracode.org/wiki/MyFieldsDontWork
To give a little background, South used to handle automatic introspection of custom fields, but according to the explanation of why it no longer does that in newer versions of South, "... when it broke, it broke spectacularly." Fair enough! So the expectation on newer versions of South is that you'll provide the necessary information to allow South to handle the custom fields.
Giving South what it needs to do migrations involving custom fields is simple enough but since this is the first time
I've had to deal with it I thought I'd help my future self and
potentially others by sharing how to handle it.
There are a couple of different scenarios with South and custom fields that you can read more about in the South docs, but in the case of django-stdimage you simply have to add the location of the custom field to South's introspection rules.
In the models.py that contains the model in which you're using django-stdimage, add the following somewhere at the top of the file:
from south.modelsinspector import add_introspection_rules
add_introspection_rules([], ['^stdimage\.fields\.StdImageField'])
Now when you run the South schema migration South will know where to find StdImageField and will be able to handle the migration.
If you have more complex custom fields be sure and read the South docs since if your custom field adds new attributes or doesn't extend another class for which there are already introspection rules, you'll have to provide the introspection rules yourself as opposed to simply providing the location of the custom field class.
I'll share the link to the app in which I'm using django-stdimage and put the code for the entire app up on GitHub soon.
Saturday, April 6, 2013
Wednesday, March 6, 2013
Downloading from eMusic on Linux
eMusic is an excellent, long-standing digital music store and one I've been a member of for years, but to say they aren't friendly to GNU/Linux would be putting it mildly. They used to have an official (in permanent beta, but still) download manager but they've since done away with it so there is no official way to download music from eMusic on GNU/Linux.
Since I do some other audio/voiceover stuff on Windows it wasn't a huge deal for me to suck it up and download from eMusic on Windows, but that has since stopped working for no apparent reason. I click download, the download manager pops up, and nothing happens. Perhaps not coincidentally this is precisely the time the Google Music Manager stopped working as well, and also when my Windows Media Center PC stopped downloading guide data. Methinks a Windows patch of some sort borked all this stuff in one fell swoop but I'd rather not be using Windows anyway so I took it as a wake-up call.
Thankfully there's an unofficial, Java-based eMusic download manager called eMusic/J and though it used to work without any additional configuration in the past, due to recent-ish changes in how eMusic does their downloads it had quit working for me too. Today I finally had a few minutes to dig into the situation and get it figured out.
Turns out what you have to do is trick emusic into think you installed the official eMusic Download Manager even though you didn't, at which point you can download the .emx files eMusic uses to tell the Download Manager what to download, and those .emx files can be used by eMusic/J to do the downloads.
Here's the big secret on how to tell eMusic that you installed the Download Manager. You ready?
Go to this URL in the browser on GNU/Linux where you want to use eMusic/J:
http://www.emusic.com/dlm/install/
Congratulations! You just fake installed the Download Manager. Now when you go to download music from eMusic you'll be asked what you want to do with the .emx file, and you can simply either download it or tell your browser to use eMusic/J to open those files.
Seriously eMusic, I love you guys but forcing people to use a Download Manager that apparently breaks at the drop of a hat when a Windows update is issued is pretty crappy. Luckily using this trick eMusic/J works ... for now at least.
Since I do some other audio/voiceover stuff on Windows it wasn't a huge deal for me to suck it up and download from eMusic on Windows, but that has since stopped working for no apparent reason. I click download, the download manager pops up, and nothing happens. Perhaps not coincidentally this is precisely the time the Google Music Manager stopped working as well, and also when my Windows Media Center PC stopped downloading guide data. Methinks a Windows patch of some sort borked all this stuff in one fell swoop but I'd rather not be using Windows anyway so I took it as a wake-up call.
Thankfully there's an unofficial, Java-based eMusic download manager called eMusic/J and though it used to work without any additional configuration in the past, due to recent-ish changes in how eMusic does their downloads it had quit working for me too. Today I finally had a few minutes to dig into the situation and get it figured out.
Turns out what you have to do is trick emusic into think you installed the official eMusic Download Manager even though you didn't, at which point you can download the .emx files eMusic uses to tell the Download Manager what to download, and those .emx files can be used by eMusic/J to do the downloads.
Here's the big secret on how to tell eMusic that you installed the Download Manager. You ready?
Go to this URL in the browser on GNU/Linux where you want to use eMusic/J:
http://www.emusic.com/dlm/install/
Congratulations! You just fake installed the Download Manager. Now when you go to download music from eMusic you'll be asked what you want to do with the .emx file, and you can simply either download it or tell your browser to use eMusic/J to open those files.
Seriously eMusic, I love you guys but forcing people to use a Download Manager that apparently breaks at the drop of a hat when a Windows update is issued is pretty crappy. Luckily using this trick eMusic/J works ... for now at least.
Wednesday, February 20, 2013
Generating and Sorting on a Transient Property in a Django Model Class
I ran into an interesting little issue in a Django application today that led to what I thought was some pretty powerful stuff in Python and Django that I hadn't had to use before, so I thought I'd share.
I'm working on an application that to keep it generic I'll simply say deals with requests from users, and these requests require various levels of approval depending on the type and severity of the request.
Here's a basic Django model class that contains the fields relative to this example:
Basic stuff so far.
Where the wrinkle comes in is there is a page in the application that lists the requests ordered by date, but the date by which each request will be displayed and sorted depends upon the impact of the request as follows:
Because I want to keep the business logic in the model (fat models FTW!), I looked into using a Python property to add a transient attribute to my UserRequest class that is the result of a function call.
Basically what this means in concrete terms is I'm adding a calendar_date attribute to my model class and the value of calendar_date is set by calling a function in the class itself that contains the aforementioned conditional logic around the request impact.
Here's the modified model class:
The big addition here is the _get_calendar_date function that returns a calendar date based on the impact of the request. This is added as an attribute to the class, but it's not saved to the database which is exactly what I wanted in this instance.
That solves one piece of the puzzle, but the other piece is sorting the requests based on this new transient property, which again can't be done by using order_by on the QuerySet since the field on which we want to order isn't in the database.
This is where Python's sorted() function comes in. sorted() can take any iterable (which the Django QuerySet is) and sort it based on a provided comparator function, and this is where I can leverage the transient calendar_date property to sort the requests.
Putting all the pieces together in the view function, here's how it looks:
Python's sorted() function is pretty powerful and straight-forward -- throw it an iterable and what to compare on, and it does the heavy lifting for you.
The only potentially tricky part of this is setting the key since it uses Python's lambda statement (great explanation here), which is Python's way of kinda sorta dipping a toe into the functional programming waters by letting you define an anonymous inline function.
In this case how the lambda plays out is to sort the QuerySet based on two attributes: first the transient calendar_date property that isn't stored in the database, and subsequently the request_impact attribute that is stored in the database. Perfect!
I'm sure none of this is rocket science to seasoned Python and Django veterans but since I hadn't run into the need to do this before I thought I'd document what I came up with both for my own future reference and hopefully for the benefit of others who made need to so something similar.
I'm working on an application that to keep it generic I'll simply say deals with requests from users, and these requests require various levels of approval depending on the type and severity of the request.
Here's a basic Django model class that contains the fields relative to this example:
Basic stuff so far.
Where the wrinkle comes in is there is a page in the application that lists the requests ordered by date, but the date by which each request will be displayed and sorted depends upon the impact of the request as follows:
- minor: use datetime_supervisor_approved
- major: use datetime_admin_approved
- severe: use datetime_president_approved
- The date on which I want to sort varies for each request
- I don't want to store an additional, redundant field in the database simply to have a consistent name to use for sorting
- Because the sort date isn't stored in the database I can't use order_by on a QuerySet
Because I want to keep the business logic in the model (fat models FTW!), I looked into using a Python property to add a transient attribute to my UserRequest class that is the result of a function call.
Basically what this means in concrete terms is I'm adding a calendar_date attribute to my model class and the value of calendar_date is set by calling a function in the class itself that contains the aforementioned conditional logic around the request impact.
Here's the modified model class:
The big addition here is the _get_calendar_date function that returns a calendar date based on the impact of the request. This is added as an attribute to the class, but it's not saved to the database which is exactly what I wanted in this instance.
That solves one piece of the puzzle, but the other piece is sorting the requests based on this new transient property, which again can't be done by using order_by on the QuerySet since the field on which we want to order isn't in the database.
This is where Python's sorted() function comes in. sorted() can take any iterable (which the Django QuerySet is) and sort it based on a provided comparator function, and this is where I can leverage the transient calendar_date property to sort the requests.
Putting all the pieces together in the view function, here's how it looks:
Python's sorted() function is pretty powerful and straight-forward -- throw it an iterable and what to compare on, and it does the heavy lifting for you.
The only potentially tricky part of this is setting the key since it uses Python's lambda statement (great explanation here), which is Python's way of kinda sorta dipping a toe into the functional programming waters by letting you define an anonymous inline function.
In this case how the lambda plays out is to sort the QuerySet based on two attributes: first the transient calendar_date property that isn't stored in the database, and subsequently the request_impact attribute that is stored in the database. Perfect!
I'm sure none of this is rocket science to seasoned Python and Django veterans but since I hadn't run into the need to do this before I thought I'd document what I came up with both for my own future reference and hopefully for the benefit of others who made need to so something similar.
Sunday, February 17, 2013
Installing and Configuring NextPVR as a Replacement for Windows Media Center
If you follow me on Google+ you'll know I had a recent rant about Windows Media Center, which after running fine for about a year suddenly decided as of January 29 it was done downloading the program guide and by extension was therefore done recording any TV shows.
I'll spare you more ranting and simply say that none of the suggestions I got (which I appreciate!) worked, and rather than spending more time figuring out why, I decided to try something different.
NextPVR is an awesome free (as in beer, not as in freedom unfortunately ...) PVR application for Windows that with a little bit of tweaking handily replaced Windows Media Center. It can even download guide data, which is apparently something WMC no longer feels like doing.
Installing Raspbmc was easy enough and the base install is working fine, but based on what I've read it may not (yet) be up to functioning as a full-blown PVR like I'm used to, and will likely serve better (as some folks on Google+ suggested) as a front-end to something else.
That led me to searching around for a WMC replacement that will run on Windows 7.
Given my huge preference for free (as in freedom) software why continue to use Windows and not move to something like MythTV? Honestly I'll probably go the MythTV route eventually, but in the near term I was just looking for something that would quickly allow me to schedule and record shows again. I'm running WMC on a very nice Acer Revo, which has a DVD player form factor and is designed to function specifically as a media center, complete with a nice pop-out dual-function mouse/keyboard pad. I'll go the full free software route another weekend.
As I looked around I came across this article on Lifehacker about turning a Raspberry Pi into a media center, and it too was focused on the Pi as a front-end to another machine that does the heavy lifting. Honestly I like this idea anyway -- sticking a quiet, low-power Raspberry Pi behind every TV other than the one with the NextPVR box seems like a slick solution.
In my case I'm using a SiliconDust HDHomeRun as my tuner, which means I do not have a tuner card in the media center PC. The HDHomeRun makes three tuners available over ethernet, and NextPVR sees them out of the box, though there is some configuration and installation of codecs necessary to get things working properly.
With the HDHomeRun I simply had to go into devices, choose each tuner, and click the "Scan" button. I have digital cable so this pulled back a little over 500 channels. Note that you have to do this for each tuner. You can tell it to copy a configuration from one tuner to another, but I found that to be much slower than performing the scan on each one individually.
The first problem I ran into after the channel scan is that although NextPVR sees the HDHomeRun fine, when I tried to watch live TV I got no picture or sound. I could see that it was talking to the HDHomeRun since I saw the tuner light come on and it was going through the motions of changing channels, and even had the channel names correct, but there was no picture or sound.
If you have this issue don't panic; it's very likely you just need to go into the NextPVR settings and choose the correct decoder. If you don't have valid MPEG2 or AC3 codecs installed, which are the video and audio formats the HDHomeRun uses, you'll have to do that separately.
In my case to get the video working I had to enable the MPEG2 and H.264 video decoders, and for those I selected the "Microsoft DTV-DVD Video Decoder" that was already installed on my machine.
As you change these settings, make sure and restart NextPVR. I'm not sure that's supposed to be required but with both audio and video I had to restart NextPVR for the changes to take effect.
To get the audio working, I had to install additional codecs since I didn't have any on the machine that could handle AC3 audio. For MPEG1 audio the Microsoft DTV-DVD Audio Decoder should work (though I can't confirm since I'm not sure I tested any MPEG1 audio specifically), but it definitely didn't work for me for any of the other audio types.
Some suggestions for fixing this issue were to install Windows Media Center since it comes with codecs that won't otherwise be on the machine, but from what I could tell anything WMC installs is pretty MS-specific, so you're better off installing more agnostic codecs.
Luckily there are some excellent free (as in beer) codecs available. What I ended up installing was the Windows 7 Codec Pack. This comes with codecs for just about every audio and video format you can imagine and gave me what I needed to fix the lack of audio.
One the Windows 7 Codec Pack was installed, I selected "ffdshow Audio Decoder" for all the audio types other than MPEG1, and after restarting NextPVR I was getting audio. Again I think the only audio type I was really verifying was AC3 since that's what the HDHomeRun uses, so if you have a different setup that requires different decoders, you may need to try some different options.
In the case of the guide data I decided to throw a little money at this problem. You don't necessarily have to pay for program guide data but from the little bit of research I did it seems like the free solutions would require some ongoing interaction on my part (or some scripting), and I don't want to turn recording TV into another job for myself, so I decided to pay for it. If you want to investigate solutions like XMLTV and others, the NextPVR wiki page on program data (EPG) is a great place to start.
Specifically, I decided to pay $25/year for a Schedules Direct account. It integrates completely seamlessly with NextPVR and based on what I read was well worth the money compared to dealing with this yourself. (I'm not averse to a little scripting and hacking but I have enough side jobs at the moment!)
After getting a Schedules Direct account, you then create a listing in Schedules Direct, which is done by providing your zip code and choosing your cable provider.
Once the listing is created in Schedules Direct you then go into the Channels section of NextPVR, enter your Schedules Direct login information, pick a listing, and then program guide data will be continually updated for you.
That's not quite the end of the story though. You still have to map the channels so NextPVR knows which channel in NextPVR goes with which channel in the lineup.
This is where I think NextPVR could use a little bit of work. You can do a bulk mapping, which works pretty well overall (though verify because it did mismatch some in my case), but if you do the bulk mapping and manually change any specific mappings it asks you every time you change one if you want to have it scan and try to do a bulk match even after it already did that. Minor annoyance but seems like it'd be an easy fix as well.
With all that in place, in the Channels section of the NextPVR setting I clicked the Update EPG button and a couple of minutes later, program guide data was available. (Take that, Windows Media Center!) Then I could finally go in and start recreating the season passes I lost when I tried reinstalling WMC.
Once I get that figured out then I can have the Raspberry Pi hooked up to a TV in another room and watch either live TV or recordings from the NextPVR box, which will be very slick. I'll do another post on that once I get it working.
The UI may not be quite as slick, strictly speaking, as WMC, but it's also a lot cleaner and zippier, and I'll take that any day. With the guide data in place I'm very happy with the functionality and don't think I'll miss anything at all from WMC.
Longer term I'll definitely look into MythTV so I have a free as in freedom solution, but this was a quick and easy project that at least got me up and running again after the WMC meltdown.
The first real test of my new setup will be when NextPVR records The Walking Dead tonight. As WMC found out if it screws that up, it's history.
I'll spare you more ranting and simply say that none of the suggestions I got (which I appreciate!) worked, and rather than spending more time figuring out why, I decided to try something different.
NextPVR is an awesome free (as in beer, not as in freedom unfortunately ...) PVR application for Windows that with a little bit of tweaking handily replaced Windows Media Center. It can even download guide data, which is apparently something WMC no longer feels like doing.
Background
I wound up going down this road in a rather circuitous way. My initial goal for the weekend project was to get Raspbmc running on one of my Raspberry Pis. The latest version of XBMC has PVR functionality so I was anxious to try that out as a potential replacement for WMC.Installing Raspbmc was easy enough and the base install is working fine, but based on what I've read it may not (yet) be up to functioning as a full-blown PVR like I'm used to, and will likely serve better (as some folks on Google+ suggested) as a front-end to something else.
That led me to searching around for a WMC replacement that will run on Windows 7.
Given my huge preference for free (as in freedom) software why continue to use Windows and not move to something like MythTV? Honestly I'll probably go the MythTV route eventually, but in the near term I was just looking for something that would quickly allow me to schedule and record shows again. I'm running WMC on a very nice Acer Revo, which has a DVD player form factor and is designed to function specifically as a media center, complete with a nice pop-out dual-function mouse/keyboard pad. I'll go the full free software route another weekend.
As I looked around I came across this article on Lifehacker about turning a Raspberry Pi into a media center, and it too was focused on the Pi as a front-end to another machine that does the heavy lifting. Honestly I like this idea anyway -- sticking a quiet, low-power Raspberry Pi behind every TV other than the one with the NextPVR box seems like a slick solution.
Installing NextPVR
Installing NextPVR is very straight-forward -- I mostly just did what's outlined in the Lifehacker article.- Download the latest version
- Download the patches and copy them into the directory in which you installed NextPVR
- Start NextPVR and adjust the settings
In my case I'm using a SiliconDust HDHomeRun as my tuner, which means I do not have a tuner card in the media center PC. The HDHomeRun makes three tuners available over ethernet, and NextPVR sees them out of the box, though there is some configuration and installation of codecs necessary to get things working properly.
Watching Live TV on NextPVR
Next you'll want to configure NextPVR to be able to watch live TV, and the first step in this process is to have NextPVR scan your tuner devices to get a channel listing.With the HDHomeRun I simply had to go into devices, choose each tuner, and click the "Scan" button. I have digital cable so this pulled back a little over 500 channels. Note that you have to do this for each tuner. You can tell it to copy a configuration from one tuner to another, but I found that to be much slower than performing the scan on each one individually.
The first problem I ran into after the channel scan is that although NextPVR sees the HDHomeRun fine, when I tried to watch live TV I got no picture or sound. I could see that it was talking to the HDHomeRun since I saw the tuner light come on and it was going through the motions of changing channels, and even had the channel names correct, but there was no picture or sound.
If you have this issue don't panic; it's very likely you just need to go into the NextPVR settings and choose the correct decoder. If you don't have valid MPEG2 or AC3 codecs installed, which are the video and audio formats the HDHomeRun uses, you'll have to do that separately.
In my case to get the video working I had to enable the MPEG2 and H.264 video decoders, and for those I selected the "Microsoft DTV-DVD Video Decoder" that was already installed on my machine.
As you change these settings, make sure and restart NextPVR. I'm not sure that's supposed to be required but with both audio and video I had to restart NextPVR for the changes to take effect.
To get the audio working, I had to install additional codecs since I didn't have any on the machine that could handle AC3 audio. For MPEG1 audio the Microsoft DTV-DVD Audio Decoder should work (though I can't confirm since I'm not sure I tested any MPEG1 audio specifically), but it definitely didn't work for me for any of the other audio types.
Some suggestions for fixing this issue were to install Windows Media Center since it comes with codecs that won't otherwise be on the machine, but from what I could tell anything WMC installs is pretty MS-specific, so you're better off installing more agnostic codecs.
Luckily there are some excellent free (as in beer) codecs available. What I ended up installing was the Windows 7 Codec Pack. This comes with codecs for just about every audio and video format you can imagine and gave me what I needed to fix the lack of audio.
One the Windows 7 Codec Pack was installed, I selected "ffdshow Audio Decoder" for all the audio types other than MPEG1, and after restarting NextPVR I was getting audio. Again I think the only audio type I was really verifying was AC3 since that's what the HDHomeRun uses, so if you have a different setup that requires different decoders, you may need to try some different options.
Downloading Program Guide Data (EPG) and Mapping Channels
With video and audio both working there was one last problem to solve, namely program guide data. Without guide data you won't know what's on and, as I discovered the hard way with WMC, you won't be able to record anything.In the case of the guide data I decided to throw a little money at this problem. You don't necessarily have to pay for program guide data but from the little bit of research I did it seems like the free solutions would require some ongoing interaction on my part (or some scripting), and I don't want to turn recording TV into another job for myself, so I decided to pay for it. If you want to investigate solutions like XMLTV and others, the NextPVR wiki page on program data (EPG) is a great place to start.
Specifically, I decided to pay $25/year for a Schedules Direct account. It integrates completely seamlessly with NextPVR and based on what I read was well worth the money compared to dealing with this yourself. (I'm not averse to a little scripting and hacking but I have enough side jobs at the moment!)
After getting a Schedules Direct account, you then create a listing in Schedules Direct, which is done by providing your zip code and choosing your cable provider.
Once the listing is created in Schedules Direct you then go into the Channels section of NextPVR, enter your Schedules Direct login information, pick a listing, and then program guide data will be continually updated for you.
That's not quite the end of the story though. You still have to map the channels so NextPVR knows which channel in NextPVR goes with which channel in the lineup.
This is where I think NextPVR could use a little bit of work. You can do a bulk mapping, which works pretty well overall (though verify because it did mismatch some in my case), but if you do the bulk mapping and manually change any specific mappings it asks you every time you change one if you want to have it scan and try to do a bulk match even after it already did that. Minor annoyance but seems like it'd be an easy fix as well.
With all that in place, in the Channels section of the NextPVR setting I clicked the Update EPG button and a couple of minutes later, program guide data was available. (Take that, Windows Media Center!) Then I could finally go in and start recreating the season passes I lost when I tried reinstalling WMC.
As for the Raspberry Pi ...
Since I basically took a huge detour from my original goal of getting XBMC up and running on my Raspberry Pi, that piece isn't quite done yet. Raspbmc itself is running fine, and I installed the PVR Client for NextPVR, but it's not working properly for some reason so I still need to dig into that.Once I get that figured out then I can have the Raspberry Pi hooked up to a TV in another room and watch either live TV or recordings from the NextPVR box, which will be very slick. I'll do another post on that once I get it working.
Conclusion
Overall I'm quite impressed with NextPVR so far. The installation was quite simple, it saw the HDHomeRun without any problems, and the codec issues I ran into were very easy to solve.The UI may not be quite as slick, strictly speaking, as WMC, but it's also a lot cleaner and zippier, and I'll take that any day. With the guide data in place I'm very happy with the functionality and don't think I'll miss anything at all from WMC.
Longer term I'll definitely look into MythTV so I have a free as in freedom solution, but this was a quick and easy project that at least got me up and running again after the WMC meltdown.
The first real test of my new setup will be when NextPVR records The Walking Dead tonight. As WMC found out if it screws that up, it's history.
Tuesday, January 22, 2013
Setting Up Django On a Raspberry Pi
This past weekend I finally got a chance to set up one of my two Raspberry Pis to use as a Django server so I thought I'd share the steps I went through both to save someone else attempting to do this some time as well as get any feedback in case there are different/better ways to do any of this.
I'm running this from my house (URL forthcoming once I get the real Django app finalized and put on the Raspberry Pi) using dyndns.org. I don't cover that aspect of things in this post but I'm happy to write that up as well if people are interested.
You can create a virtualenv for your application in your home directory or wherever you prefer your files to be located, just be aware that depending on where you put your files and whether or not you have to execute virtualenv-specific tasks (pip install, etc.) you may have to explicitly specify which python or pip command to use for it to work properly.
For the purposes of this example we’ll create a ‘foo’ application. The steps below where you’re running django-admin.py startproject and firing up the development server are simply to test to make sure everything’s working properly. If you’re doing this all for the first time, I’d suggest doing this both to get familiar with it as well as to double-check that everything with the environment is working as expected.
A note on databases: I’m not addressing the broader topic of which database server you should use simply because it’s a separate discussion that’s potentially fraught with zealous opinions and general peril, and is a bit tangential to the present discussion.
If you’re doing all this just to learn, or for your own personal low-use “screwing around” type applications, SQLite (http://www.sqlite.org/), which is what comes with Django, will work just fine, so that’s what I outlined above.
I’m writing this up as part of setting up a site for alt.* Radio (http://codebassradio.net/shows/alt/), which is a weekly show I do on the super-awesome CodeBass Radio (http://codebassradio.net), so for that I’ll be running PostgreSQL on a separate machine. For low-traffic stuff you might even be able to get away with running MySQL or Postgres directly on the Raspberry Pi but I have a couple of unused machines laying around so I figured I might as well keep the Raspberry Pi focused on the Django side of things.
Note that if you DO want to use PostgreSQL, you’ll have to do this on the Raspberry Pi:
# upstream server for gunicorn
upstream gunicorn {
server localhost:8000;
}
# nginx server for the host
server {
listen 80;
server_name foo.com www.foo.com;
root /home/pi/foo;
access_log /var/log/nginx/foo_access.log;
error_log /var/log/nginx/foo_error.log;
# try to serve a static file and if it doesn't exist, pass to gunicorn
try_files $uri @gunicorn;
# rules for gunicorn
location @gunicorn {
proxy_pass http://gunicorn;
proxy_redirect off;
proxy_read_timeout 5m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Fowarded-For $proxy_add_x_forwarded_for;
}
}
If you want to hit the Raspberry Pi from a browser on another machine you’ll have to also add the IP and host names to /etc/hosts on any machines from which you want to hit that host, but use the IP of the Raspberry Pi instead of 127.0.0.1.
With that modification in /etc/hosts locally on the Raspberry Pi, you can do a curl foo.com to see if you get the default Django page (or your application if you took things a bit further than are outlined in this tutorial). If you do, everything’s working! If not, start over and don’t screw up this time!
Seriously though if you need any help just comment here or email/Google+/Twitter me and I’ll be happy to assist.
I'm running this from my house (URL forthcoming once I get the real Django app finalized and put on the Raspberry Pi) using dyndns.org. I don't cover that aspect of things in this post but I'm happy to write that up as well if people are interested.
General Comments and Assumptions
- Using latest Raspbian “wheezy” distro as of 1/19/2013 (http://www.raspberrypi.org/downloads)
- We’lll be using Nginx (http://nginx.org) as the web server/proxy and Gunicorn (http://gunicorn.org) as the WSGI server
- I used http://www.apreche.net/complete-single-server-django-stack-tutorial/ heavily as I was creating this, so many thanks to the author of that tutorial. If you’re looking for more details on more stuff related to setting this up on Ubuntu definitely use that guide.
Reconfigure the Keyboard Mapping
If you already did this at some other point with your Raspberry Pi or your keyboard is working properly you can skip this step. After the base install of Raspbian “wheezy” in my case I noticed when in vim I couldn’t type things like #, |, and @ so I had to do the following:- sudo dpkg-reconfigure keyboard-configuration
- Follow the prompts to set things appropriate for your keyboard (US English in my case)
- Restart the Raspberry Pi (sudo reboot)
Install Required Server-Wide Tools and Libraries
- sudo apt-get install vim python-dev python-setuptools nginx supervisor
- sudo easy_install pip
- sudo pip install virtualenv virtualenvwrapper
- Configure virtualenvwrapper
- sudo vim /etc/bash.bashrc
- Add this line at the bottom of the file:
source /usr/local/bin/virtualenvwrapper.sh - Save the file and exit vim
- Log out and log back in
Create and Configure a virtualenv for Your Application
tl;dr version of the following paragraph: Run the following commands from your home directory unless you have a reason not to and know what you’re doing. For the purposes of this tutorial I’m using the default ‘pi’ user on the Raspberry Pi.You can create a virtualenv for your application in your home directory or wherever you prefer your files to be located, just be aware that depending on where you put your files and whether or not you have to execute virtualenv-specific tasks (pip install, etc.) you may have to explicitly specify which python or pip command to use for it to work properly.
For the purposes of this example we’ll create a ‘foo’ application. The steps below where you’re running django-admin.py startproject and firing up the development server are simply to test to make sure everything’s working properly. If you’re doing this all for the first time, I’d suggest doing this both to get familiar with it as well as to double-check that everything with the environment is working as expected.
- mkvirtualenv foo
- Note that this will activate the virtualenv. You MUST have the virtualenv activated when performing the steps that follow for everything to work properly.
- If you’re used to using virtualenv directly as opposed to virtualenvwrapper, note that when you create a virtualenv using mkvirtualenv, it will put the virtualenv files (again, if you’re doing this from your home directory) in ~/.virtualenvs/foo
- pip install django docutils south gunicorn
- django-admin.py startproject foo
- cd foo
- python manage.py runserver
- The first time you run this you may see a ton of blank lines output on the Raspberry Pi. It’ll eventually finish, or if you hit Ctrl-C when this happens and try again a couple of times it’ll eventually calm down and the next time you run things this won’t happen. I’m not sure what’s going on here since the Raspberry Pi is the only platform on which I’ve seen this happen.
- Once the development server starts up you should see “0 errors found [other stuff here] Development server is running at http://127.0.0.1:8000” If you don’t see that, double-check everything and try again.
- You can make a call to the development server by sshing into the Raspberry Pi from another machine and doing curl localhost:8000
Enable Gunicorn and South and Enable a Database in Your Application
- cd ~/foo/foo
- vim settings.py
- In the DATABASES section make the following changes:
- ‘ENGINE’: ‘django.db.backends.sqlite3’
- ‘NAME’: ‘/home/pi/foo/foo.db’
- In the INSTALLED_APPS section, enable the admin and admindocs (optional):
- Uncomment # ‘django.contrib.admin’
- Uncomment # ‘django/contribu.admindocs’
- Also in the INSTALLED_APPS section, add ‘south’ and ‘gunicorn’ to the list of installed apps.
- Save the file and exit vim
- Test Gunicorn:
- python manage.py run_gunicorn (you can hit Ctrl-C to kill it once you see it’s working)
A note on databases: I’m not addressing the broader topic of which database server you should use simply because it’s a separate discussion that’s potentially fraught with zealous opinions and general peril, and is a bit tangential to the present discussion.
If you’re doing all this just to learn, or for your own personal low-use “screwing around” type applications, SQLite (http://www.sqlite.org/), which is what comes with Django, will work just fine, so that’s what I outlined above.
I’m writing this up as part of setting up a site for alt.* Radio (http://codebassradio.net/shows/alt/), which is a weekly show I do on the super-awesome CodeBass Radio (http://codebassradio.net), so for that I’ll be running PostgreSQL on a separate machine. For low-traffic stuff you might even be able to get away with running MySQL or Postgres directly on the Raspberry Pi but I have a couple of unused machines laying around so I figured I might as well keep the Raspberry Pi focused on the Django side of things.
Note that if you DO want to use PostgreSQL, you’ll have to do this on the Raspberry Pi:
- sudo apt-get install libpq-dev
- pip install psycopg2 (with your virtualenv active)
Configure Supervisor
Supervisor (http://supervisord.org/), which we installed earlier, is a tool that lets you easily create startup/restart scripts for services that aren’t installed and managed through apt-get. In our case we’ll need this for Gunicorn so it’ll fire up when the machine boots and automatically restart if it crashes.- cd /etc/supervisor/conf.d
- sudo vim gunicorn.conf
- Put the following in the new file:
[program:gunicorn]
command = /home/pi/.virtualenvs/foo/bin/python /home/pi/foo/manage.py run_gunicorn -w 4
directory = /home/pi/foo
user = pi
autostart = true
autorestart = true
stdout_logfile = /var/log/supervisor/gunicorn.log
stderr_logfile = /var/log/supervisor/gunicorn_err.log - Save the file and exit vim
- sudo service supervisor restart
- sudo supervisorctl start gunicorn
- Note that you may get an “already started” error here. If you get a “no such process” error, that means supervisor didn’t load the new configuration file. If that happens:
- sudo ps -wef | grep supervisor
- sudo kill -9 SUPERVISOR_PROCESS_ID (use the real process ID)
- sudo service supervisor start
Configure Nginx
- sudo rm -f /etc/nginx/sites-enabled/default
- sudo vim /etc/nginx/sites-available/foo
- Put the following in the foo file:
# upstream server for gunicorn
upstream gunicorn {
server localhost:8000;
}
# nginx server for the host
server {
listen 80;
server_name foo.com www.foo.com;
root /home/pi/foo;
access_log /var/log/nginx/foo_access.log;
error_log /var/log/nginx/foo_error.log;
# try to serve a static file and if it doesn't exist, pass to gunicorn
try_files $uri @gunicorn;
# rules for gunicorn
location @gunicorn {
proxy_pass http://gunicorn;
proxy_redirect off;
proxy_read_timeout 5m;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Fowarded-For $proxy_add_x_forwarded_for;
}
}
- Save the file and exit vim
- sudo ln -s /etc/nginx/sites-available/foo /etc/nginx/sites-enabled/foo
- sudo service nginx restart
Verify Everything’s Working
If you’re not using a real hostname/URL, make sure and add it to /etc/hosts on the Raspberry Pi so you can easily test. In this case we used foo.com as the hostname, so do the following:- sudo vim /etc/hosts
- Add this line:
127.0.0.1 foo.com www.foo.com - Save the file and exit vim
If you want to hit the Raspberry Pi from a browser on another machine you’ll have to also add the IP and host names to /etc/hosts on any machines from which you want to hit that host, but use the IP of the Raspberry Pi instead of 127.0.0.1.
With that modification in /etc/hosts locally on the Raspberry Pi, you can do a curl foo.com to see if you get the default Django page (or your application if you took things a bit further than are outlined in this tutorial). If you do, everything’s working! If not, start over and don’t screw up this time!
Seriously though if you need any help just comment here or email/Google+/Twitter me and I’ll be happy to assist.
Saturday, November 3, 2012
How To Create a PyCharm Launcher on Ubuntu 12.10
I'm absolutely loving using PyCharm for my Python and Django development, but one of the lingering things I've been meaning to nail down once and for all is creating a launcher for PyCharm in Ubuntu 12.10. Despite the automated way you can attempt to do this from within PyCharm itself and all the other recommendations I've read I was unable to get it working.
In the mean time I also bought of copy of IntelliJ IDEA when they had some crazy back to school sale a couple of months ago (I still dabble in Groovy and Grails a bit). I was having the same issues with creating a launcher for IDEA and the typical tricks I use for Eclipse weren't working, but luckily I came across this post that explains how to do it. The only change I made is pointing to idea64.vmoptions instead of just idea.vmoptions -- other than that it works great.
That got me thinking -- since PyCharm and IDEA are both made by JetBrains, and both run on Java, chances are how they work is pretty darn similar. So I decided to copy the IDEA launcher script and modify it for PyCharm, and lo and behold it worked!
Here's my modified version of the IDEA launcher that works for PyCharm.
#!/bin/bash
export JAVA_HOME=/opt/java/jdk1.7.0_09
export JDK_HOME=/opt/java/jdk1.7.0_09
export PYCHARM_HOME=/home/mwoodward/pycharm-2.6.2
export PYCHARM_VM_OPTIONS="$PYCHARM_HOME/bin/pycharm64.vmoptions"
export PYCHARM_PROPERTIES="$PYCHARM_HOME/bin/idea.properties"
cd "$PYCHARM_HOME/bin"
export LIBXCB_ALLOW_SLOPPY_LOCK=1
./pycharm.sh
Obviously adjust all the paths as necessary for your machine. Make sure you chmod +x on the file, and with that in place you can open up Main Menu (sudo apt-get install alacarte if you don't already have it installed) and add a launcher. For reference the icon lives in PyCharm's bin directory.
Hope that helps someone else who has run into this issue.
Saturday, October 20, 2012
Using Python to Compare Document IDs in Two CouchDB Databases
I'm doing a bit of research into what may or may not be an issue with a specific database in our BigCouch cluster, but regardless of the outcome of that side of things I thought I'd share how I used Python and couchdb-python to dig into the problem.
In our six-server BigCouch cluster we noticed that on the database for one of our most heavily trafficked applications the document counts displayed in Futon for each of the cluster members don't match. As I said above this may or may not be a problem (I'm waiting on further information on that particular point), but I was curious which documents were missing from the cluster member that has the lowest document count. (The interesting thing is the missing documents aren't truly inaccessible from the server with the lower document count, but we'll get to that in a moment.)
BigCouch is based on Apache CouchDB but adds true clustering as well as some other very cool features, but for those of you not familiar with CouchDB, you communicate with CouchDB through a RESTful HTTP interface and all the data coming and going is JSON. The point here is it's very simple to interact with CouchDB with any tool that talks HTTP.
Dealing with raw HTTP and JSON may not be difficult but isn't terribly Pythonic either, which is where couchdb-python comes in. couchdb-python lets you interact with CouchDB via simple Python objects and handles the marshaling of data between JSON and native Python datatypes for you. It's very slick, very fast, and makes using CouchDB from Python a joy.
In order to get to the bottom of my problem, I wanted to connect to two different BigCouch cluster members, get a list of all the document IDs in a specific database on each server, and then generate a list of the document IDs that don't exist on the server with the lower total document count.
Here's what I came up with:
>>> import couchdb
>>> couch1 = couchdb.Server('http://couch1:5984/')
>>> couch2 = couchdb.Server('http://couch2:5984/')
>>> db1 = couch1['dbname']
>>> db2 = couch2['dbname']
>>> ids1 = []
>>> ids2 = []
>>> for id in db1:
... ids1.append(id)
...
>>> for id in db2:
... ids2.append(id)
...
>>> missing_ids = list(set(ids1) - set(ids2))
What that gives me, thanks to the awesomeness of Python and its ability to subtract one set from another (note that you can also use the difference() method on the set object to achieve the same result), is a list of the document IDs that are in the first list that aren't in the second list.
The interesting part came when I took one of the supposedly missing IDs and tried to pull up that document from the database in which it supposedly doesn't exist:
>>> doc = db2['supposedly_missing_id_here']
I was surprised to see that it returned the document just fine, meaning it must be getting it from another member of the cluster, but I'm still digging into what the expected behavior is on all of this. (It's entirely possible I'm obsessing over consistent document counts when I don't need to be.)
So what did I learn through all of this?
In our six-server BigCouch cluster we noticed that on the database for one of our most heavily trafficked applications the document counts displayed in Futon for each of the cluster members don't match. As I said above this may or may not be a problem (I'm waiting on further information on that particular point), but I was curious which documents were missing from the cluster member that has the lowest document count. (The interesting thing is the missing documents aren't truly inaccessible from the server with the lower document count, but we'll get to that in a moment.)
BigCouch is based on Apache CouchDB but adds true clustering as well as some other very cool features, but for those of you not familiar with CouchDB, you communicate with CouchDB through a RESTful HTTP interface and all the data coming and going is JSON. The point here is it's very simple to interact with CouchDB with any tool that talks HTTP.
Dealing with raw HTTP and JSON may not be difficult but isn't terribly Pythonic either, which is where couchdb-python comes in. couchdb-python lets you interact with CouchDB via simple Python objects and handles the marshaling of data between JSON and native Python datatypes for you. It's very slick, very fast, and makes using CouchDB from Python a joy.
In order to get to the bottom of my problem, I wanted to connect to two different BigCouch cluster members, get a list of all the document IDs in a specific database on each server, and then generate a list of the document IDs that don't exist on the server with the lower total document count.
Here's what I came up with:
>>> import couchdb
>>> couch1 = couchdb.Server('http://couch1:5984/')
>>> couch2 = couchdb.Server('http://couch2:5984/')
>>> db1 = couch1['dbname']
>>> db2 = couch2['dbname']
>>> ids1 = []
>>> ids2 = []
>>> for id in db1:
... ids1.append(id)
...
>>> for id in db2:
... ids2.append(id)
...
>>> missing_ids = list(set(ids1) - set(ids2))
What that gives me, thanks to the awesomeness of Python and its ability to subtract one set from another (note that you can also use the difference() method on the set object to achieve the same result), is a list of the document IDs that are in the first list that aren't in the second list.
The interesting part came when I took one of the supposedly missing IDs and tried to pull up that document from the database in which it supposedly doesn't exist:
>>> doc = db2['supposedly_missing_id_here']
I was surprised to see that it returned the document just fine, meaning it must be getting it from another member of the cluster, but I'm still digging into what the expected behavior is on all of this. (It's entirely possible I'm obsessing over consistent document counts when I don't need to be.)
So what did I learn through all of this?
- The more I use Python the more I love it. Between little tasks like this and the fantastic experience I'm having working on our first full-blown Django project, I'm in geek heaven.
- couchdb-python is awesome, and I'm looking forward to using it on a real project soon.
- Even though we've been using CouchDB and BigCouch with great success for a couple of years now, I'm still learning what's going on under the hood, which for me is a big part of the fun.
Subscribe to:
Posts (Atom)