Tuesday, January 21, 2014

Python + Oracle on Ubuntu Server 12.04

Affectionately known among all non-masochists in the world of IT as The Seventh Circle of Hell (with real hell being preferable), working with Oracle is always a hair-tearing nightmarish fork-in-the-eye please-for-the-love-of-god-kill-me-now experience that none but those who look to Ted Bundy, Jeffrey Dahmer, and John Wayne Gacy for moral and spiritual guidance would wish upon even their most reviled enemies.

Yes, it's that bad. And apparently nowhere is it worse than when one attempts to get Oracle working with Python on Ubuntu.

I'm not even talking about installing the Oracle database server itself here people, I'm just needing a Python application to talk to an existing Oracle database. One would think, as with every other database server on the planet (and yes, I'm including that other slice of hell SQL Server in that statement since it's a damn sight simpler to get working -- even on Linux -- than Oracle), you'd simply apt-get and/or pip install a library or two and be done with it.

If you actually do think that, you've already forgotten that this is Oracle we're talking about.

That said, one does what one has to do to keep the paychecks coming, so if you need to do this here's the steps to make it all happen. (Note that on Step 1 I'm assuming you have already installed all the other Python packages you may need. I'm focusing on the stuff you may not have that you definitely need.)
  1. sudo apt-get install libaio1 alien
  2. Download the RPM of version 11.2.0.4.0 of the Oracle client from http://www.oracle.com/technetwork/topics/linuxx86-64soft-092277.html (note that as of the date of this writing the 12.x version doesn't work, or at least didn't for me)
    1. You have to have an Oracle Web Account and use that to log in and download this, which makes using wget on the target server itself or automating the process for use with something like a Vagrant provisioning script rather problematic. Short version, you'll have to download this locally and then scp it up to the target server. What I did is downloaded and converted this RPM as well as the other necessary RPM, converted them once, and put them in a git repo from which I can clone in my Vagrant provisioning script. Whether or not that adheres to the licensing agreement, I don't know and I don't care. If you're paranoid, check with a lawyer before repeating my solution on this.
    2. If the version number differs slightly from what I have here, adjust later steps accordingly.
  3. Download the RPM of the Python 2.7/Oracle 11g version of the cx_Oracle Python libraries from http://cx-oracle.sourceforge.net/
  4. scp the RPMs up to the target server as needed.
  5. Convert the RPMs to Debian packages using alien:
    sudo alien -d FILENAME.rpm (where FILENAME is of course the name of each of the two RPM files)
  6. sudo dpkg -i oracle-installclient11.2-basic_11.2.0.4.0-2_amd64.deb
  7. sudo vim /etc/ld.so.conf.d/oracle.conf
    1. Note: this file won't already exist, so you'll be creating this as a new file in this step
  8. Enter the following in the newly created oracle.conf file and save it:
    /usr/lib/oracle/11.2/client64/lib
  9. export ORACLE_HOME=/usr/lib/oracle/11.2/client64
  10. export LD_LIBRARY_PATH=$ORACLE_HOME/lib
  11. sudo ldconfig
  12. sudo dpkg -i cx-oracle_5.1.2-2_amd64.deb
  13. cd /usr/lib/python2.7
  14. sudo mv site-packages/cx_Oracle* dist-packages
  15. sudo rmdir site-packages
  16. sudo ln -s dist-packages site-packages
  17. Verify installation by opening a Python interpreter and run the following:
    import cx_Oracle
    1. If you don't get an import error, everything is working properly
As far as automating this for use with Vagrant,  in my provisioning script I simply echoed the export statements in the steps above into /etc/environment, did source /etc/environment and followed that with ldconfig. Other than that the steps in the bash script are pretty much what's above, but if people are interested in seeing the script let me know and I can post it.

And there you have it. A lot of trial and error and head bashing went into that final solution, and since I kind of cobbled together the steps from various resources I'll post those below in case you want to see some of the other solutions and source material.

Happy Oracleing. Or not.

References

  1. http://iambusychangingtheworld.blogspot.com/2013/06/python-oracle-sqlalchemy-on-ubuntu-1304.html 
  2. http://maxolasersquad.blogspot.com/2011/04/cxoracle-on-ubuntu-1104-natty.html 
  3. https://linuxindetails.wordpress.com/2009/12/26/installation-of-python-cx_oracle-module-for-debian-squeeze/
  4. http://stackoverflow.com/questions/12538238/python-module-cx-oracle-module-could-not-be-found

Wednesday, October 2, 2013

Dropbox on Linux Mint 15

I noticed on Linux Mint that the Dropbox icons and menu options don't appear in my Dropbox directory. I did a little research and this is because by default when you install Dropbox it'll assume you're using Nautilus as your file browser, but by default Linux Mint uses Nemo (which is a fork of Nautilus).

Luckily it's an easy fix:
sudo apt-get install nemo-dropbox

Then quit all running instances of Nemo:
nemo --quit

When you open up your Dropbox directory again you'll see the familiar Dropbox icons.

FreeTDS on Linux Mint 15

If you're trying out or moving to Linux Mint and you're used to setting up FreeTDS on Ubuntu, you'll find that things are just slightly different on Mint.

Quick and easy fix:
sudo apt-get install freetds-bin tdsodbc

Also note that libtdsodbc.so in a different place on Mint than on Ubuntu, so instead of it being located at /usr/local/lib/libtdsodbc.so it'll be here:
/usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so

Saturday, July 13, 2013

Generating CSV Files in Django

This is a very quick tip since it's so simple but I did run into one little wrinkle with string encoding while doing this so I thought I'd share.

I had a request to generate a CSV of all the Old Dog Haven "Walk for Old Dogs" registrants so the fine folks managing the event can do email blasts, print registration sheets, and the like.

Since Python has CSV functionality as part of the standard library the generation of the CSV data was very easy, and since the Django HTTP Response object was quite wisely designed to behave as a file-like object, writing the CSV data to the HTTP response was dead simple as well.

The only thing I ran into while writing the data to the response object was there were some non-ASCII characters in the database which threw this error:
'ascii' codec can't encode character u'\xe9' in position 4: ordinal not in range(128)
 
To get around this it was just a matter of tacking .encode('utf-8') to the end of the string objects when writing out the CSV data.

Here's the final result.

Monday, July 1, 2013

Custom Managers in Django

I had to add a bit of functionality to the Old Dog Haven "Walk For Old Dogs" registration and sponsorship site today (feel free to sponsor me!), specifically a way for the good folks running the event to see an aggregate total of the sponsorship dollars by registrant.

Here's the Django models for Registrant and Sponsor:
Pretty straight-forward -- main thing to notice is that Sponsor has a foreign key relationship with Registrant as opposed to the other way around. And yes, in its current incarnation sponsors can only sponsor one registrant; if they want to sponsor multiple registrants they simply create another sponsor record. (It's agile baby -- release early and often! Multiple sponsorships can happen next year!)

While calculating the total sponsorship dollars by each registrant could be handled via the Django ORM, I decided to use the opportunity to execute SQL directly in a function in a custom manager.

You can read more about custom managers in the Django docs, but the short version is that when you call something like Foo.objects.all() the objects bit refers to the model's Manager class, and it has all sorts of default functionality built in allowing you to easily interact with your model.

Referring back to the Sponsor and Registrant model code above, notice that since we're using a custom model manager, in the Sponsor model I added the line objects = SponsorManager() to override the default model manager with the custom one for which we'll see the code below.

As with most things in Django, model managers are easily extended and overridden, and in this case I wanted to add a new function to the Sponsor manager to return a list of registrants with a sum of their sponsorship amounts. This was as simple as adding a SponsorManager model that extends model.Manager and adding a function called with_sponsorship_totals to return the information I need.
Again, pretty straight-forward. SponsorManager extends models.Manager, we add the code to run the SQL query we want to execute, loop over the results and add instances of the Registrant model to the results list, and return the results.

One of the other great things I should point out about Python is the ability to add fields to classes on the fly. The Registrant class doesn't have a total_sponsorship_amount or team_name field defined, but I can add those as I loop over my results so they're available in the list returned by the custom manager function.

Another approach to solving this problem might be to add a function to the Registrant model itself that would calculate the total sponsorship amount for each registrant, but I opted to create a custom manager in this case to experiment a little, and I also thought it taking this approach would be more efficient than calculating individually for each registrant.

As you can see from this basic example, creating a custom manager for your Django models is an extremely simple, powerful way to add functionality to your application and is nothing to shy away from. By creating fat models and extending model managers where needed you can keep the business logic where it belongs and avoid a lot of unnecessary complexity in views and other areas of your application.

Saturday, April 6, 2013

django-stdimage Custom Fields and South Migrations

I'm putting the finishing touches on my latest Django site (a registration site for a dog rescue charity walk) and part of the functionality is the ability for users to upload photos. Since most people will simply snap a photo on their cell phones and upload it, and since cell phones these days take some pretty large pictures, I want to resize the images to a standard size of 800x600 when they're uploaded and also create a thumbnail.

As with most things in the Python/Django world there's a library that does exactly what I need, specifically django-stdimage, which provides a custom model field to handle resizing photos and creating thumbnails.

After installing django-stdimage via pip, updating my model class to use the new form field, and then doing a South migration, I ran into this error:

 ! Cannot freeze field 'supporter.registrant.photo'
 ! (this field has class stdimage.fields.StdImageField)

 ! South cannot introspect some fields; this is probably because they are custom
 ! fields. If they worked in 0.6 or below, this is because we have removed the
 ! models parser (it often broke things).
 ! To fix this, read http://south.aeracode.org/wiki/MyFieldsDontWork


To give a little background, South used to handle automatic introspection of custom fields, but according to the explanation of why it no longer does that in newer versions of South, "... when it broke, it broke spectacularly." Fair enough! So the expectation on newer versions of South is that you'll provide the necessary information to allow South to handle the custom fields.
Giving South what it needs to do migrations involving custom fields is simple enough but since this is the first time I've had to deal with it I thought I'd help my future self and potentially others by sharing how to handle it.

There are a couple of different scenarios with South and custom fields that you can read more about in the South docs, but in the case of django-stdimage you simply have to add the location of the custom field to South's introspection rules.

In the models.py that contains the model in which you're using django-stdimage, add the following somewhere at the top of the file:

from south.modelsinspector import add_introspection_rules

add_introspection_rules([], ['^stdimage\.fields\.StdImageField'])


Now when you run the South schema migration South will know where to find StdImageField and will be able to handle the migration.

If you have more complex custom fields be sure and read the South docs since if your custom field adds new attributes or doesn't extend another class for which there are already introspection rules, you'll have to provide the introspection rules yourself as opposed to simply providing the location of the custom field class.

I'll share the link to the app in which I'm using django-stdimage and put the code for the entire app up on GitHub soon.

Wednesday, March 6, 2013

Downloading from eMusic on Linux

eMusic is an excellent, long-standing digital music store and one I've been a member of for years, but to say they aren't friendly to GNU/Linux would be putting it mildly. They used to have an official (in permanent beta, but still) download manager but they've since done away with it so there is no official way to download music from eMusic on GNU/Linux.

Since I do some other audio/voiceover stuff on Windows it wasn't a huge deal for me to suck it up and download from eMusic on Windows, but that has since stopped working for no apparent reason. I click download, the download manager pops up, and nothing happens. Perhaps not coincidentally this is precisely the time the Google Music Manager stopped working as well, and also when my Windows Media Center PC stopped downloading guide data. Methinks a Windows patch of some sort borked all this stuff in one fell swoop but I'd rather not be using Windows anyway so I took it as a wake-up call.

Thankfully there's an unofficial, Java-based eMusic download manager called eMusic/J and though it used to work without any additional configuration in the past, due to recent-ish changes in how eMusic does their downloads it had quit working for me too. Today I finally had a few minutes to dig into the situation and get it figured out.

Turns out what you have to do is trick emusic into think you installed the official eMusic Download Manager even though you didn't, at which point you can download the .emx files eMusic uses to tell the Download Manager what to download, and those .emx files can be used by eMusic/J to do the downloads.

Here's the big secret on how to tell eMusic that you installed the Download Manager. You ready?

Go to this URL in the browser on GNU/Linux where you want to use eMusic/J:
http://www.emusic.com/dlm/install/

Congratulations! You just fake installed the Download Manager. Now when you go to download music from eMusic you'll be asked what you want to do with the .emx file, and you can simply either download it or tell your browser to use eMusic/J to open those files.

Seriously eMusic, I love you guys but forcing people to use a Download Manager that apparently breaks at the drop of a hat when a Windows update is issued is pretty crappy. Luckily using this trick eMusic/J works ... for now at least.