Wednesday, February 25, 2015

Dynamically Adding Forms To and Removing Forms From Django Formsets

For my latest big project I am running into numerous situations in which I need to dynamically add forms to and remove forms from Django formsets, which turned into an interesting and fun challenge, and led to a lot of learning about how Django formsets work along with some ancillary details (like the behavior of the jQuery UI datepicker when you manipulate the DOM) that I wasn't necessarily expecting to encounter.

To set the stage for the discussion by way of a concrete example, one aspect of the application in question is the ability to request leave (e.g. vacation, sick leave, etc.), and as you might imagine on the leave request page you can add days to and remove days from your leave request.

In the first iteration of this feature (random aside: we do Kanban and try our darndest to focus on "good enough" features, particularly on early iterations, so that we can deliver the most usable functionality in the least amount of time, looping back around to make improvements as time permits), each form in the leave formset has the following fields:
  • date of leave
  • start time of leave
  • end time of leave
  • hours of leave
  • minutes of leave
Why does the user have to specify the number of hours and minutes they're taking for their leave if they're providing the start and end times, you might ask? Basically it boils down to people with access to the leave calendar wanting to know the specific times people are out, which may not always match up with the amount of leave they're taking.

For example if you take a full day of leave, let's say you're out 8 am to 5 pm, but there's a lunch hour in the mix, so the clock time you're out is 9 hours, but the amount of leave you have to take for the full day is only 8 hours. And again, "good enough" is the focus on this first iteration. This basic functionality lets people request leave, and in future iterations, time permitting, we'll make the system smarter and remove some of the burden off the user with things like calculating hours based on start/end times, taking into account weekends and holidays, letting people select date spans, and all sorts of other niceties that are in our backlog.

But back to the discussion at hand, namely how this all happens with Django formsets. Since the last thing I wanted to do was have the leave form code as a variable in a JavaScript function, I decided to go the route of grabbing a new instance of the form via Ajax when the user clicks the "Add Day" button. That way the leave form code lives in one place, in a normal Django template, and is therefore much more maintainable.

This immediately got interesting, however, given how Django formsets work. By default (i.e. if you aren't using the prefix option in the formset constructor), forms in Django formsets will all have ids along the lines of id_form-0-field_name, and names along the lines of form-0-field_name, where 0 is the index corresponding to the form's position in the overall formset. So if you have multiple forms the first form will have an index of 0, the second one will have an index of 1, etc.

Also in the mix is the Django formset's management form, which contains the information necessary for Django to manage and process the formset, such as the total number of forms, the minimum and maximum number of forms allowed, and so on. The key point here is that if you're going to be dynamically adding and removing forms, you also need to be updating the form-TOTAL_FORMS value in the management form in order for things to stay in sync.

As you might be guessing by now, the syntax of the form fields in a formset doesn't quite mix with making an Ajax call out of the blue to an otherwise oblivious Django template to get a new form. Thankfully that was pretty easy to resolve by making sure to get the next form number and including that in the Ajax call:


And the form template (small snippet here) simply includes the form number in all the necessary places to make the form gibe with what the formset expects:


So when you get the form back from the Ajax call it has the correct number in all the fields, and all is right with the world.

Or so it would seem. [cue dramatic music]

At this point if all you ever want to do is ADD forms to a formset, this approach works perfectly. Where things get weird is when you start removing forms from your formset, since the aforementioned form indices get all kinds of screwed up (to use a technical term) if you don't adjust them as you go.

When I first started adding the remove functionality I was optimistically thinking, "Oh Django probably doesn't care about the form indices so much. I'm sure as long as the total number of forms it has in the management form matches the number of forms in the formset, it won't care if the indices go 0, 1, 4, 7, or whatever."

And that's actually partially true. I've gone through so many iterations of everything prior to writing this blog post that you'll have to forgive me for slightly fuzzy details here, but my recollection of the first incarnation of this feature is that yes, indeed, I could add and remove rows to my heart's content, and as long as I was ensuring the next form number was incremented, and the total forms value in the management form was right, I could post the formset, validate the formset, route back with errors, and that all worked.

Where things proved interesting is when our QA folks took a crack at this, they (as QA folks tend to do) did something I hadn't:

  1. Add a few rows to the form, and set the dates (N.B. we're using jQuery UI's datepicker for this; that fact becomes important in a moment)
  2. Add another new row and don't set the date
  3. Delete one of the rows in the middle of the set of rows
  4. Set the date using the datepicker in the last row that was added
At that point what would happen is that the date selected in the final row would change the value of the date in one of the rows above it. Also if you submitted the form that way with errors, when it came back and rendered the populated forms, some of the forms were blank.

This turned out to be a great find that set me down the path of learning a whole lot of stuff I'm glad I learned now before it bit me later.

Attacking the two problems (1. Django forms don't re-render correctly in an error state, and 2. datepicker changes the wrong date) in sequence, my first thought went back to the Django formset indices and my optimistic assumption that the indices didn't matter as long as the number of forms in the formset matched the total forms value in the management form.

This, not surprisingly, turned out to not be the case. As I said earlier if you only add forms things are incremented nicely and nothing gets out of whack, but I was now finding that when a form was removed, what needed to happen is that all the existing form fields needed to be reindexed so they started with 0 and incremented sequentially. A bit of JavaScript handled that without too much trouble. (I'll put the solution below since it involves problem #2 as well.)

Even after I got the form indices back into a sequential state as forms were removed, however, the datepicker was still behaving badly. I'd add and remove forms in random order and in some cases the datepicker would still change the wrong date field (in a predictable fashion with respect to the index), or throw an error that it couldn't find the form field to which it was attached.

That latter error led me to the solution for this problem. The long and short of it is that when you remove something from the DOM that has a datepicker attached to it, you have to destroy and reattach all the datepicker elements because each datepicker retains a reference to the field to which it was originally attached.

To sum up what has to happen when a form is removed for everything to behave properly:
  1. Resequence all the form indices so they start at 0 and increment sequentially; and
  2. Destroy all remaining datepickers and recreate them
To keep things simple the remove link on each form looks like this:


That way I didn't have to keep track of the index on the delete button too, I could simply reference its form container parent and go from there.

With that in place the delete row function wound up looking like this:


Main things of interest there are getting a handle on the delete link's parent form via the form class, and lines 13-16 where the datepickers are destroyed and rebound so they get attached to the correct form field.

And of course the call to updateFormElementIndices(), the bit that resequences the form indices, which was inspired by various snippets and other solutions I found to address this issue and looks like this:


Pretty straight-forward, but to go over what's happening there:

  1. Get a handle on all the forms of the class passed to the function and loop over them
  2. Get the current index number of the form being updated
  3. Update the form index to the loop increment (0-based, sequential)
  4. Get all the inputs with the class matching the form class passed in, plus 'Input' at the end
  5. Loop over the inputs, changing the IDs and names in similar fashion to how we updated the form ID
With all that in place Django was happy, the datepicker was happy, and (hopefully) our QA folks will be happy as well.

As I said this was a great find not only because I learned a ton slogging through this today, but also because this was the first incarnation of a pattern that will occur in numerous areas throughout this application (adding multiple phone numbers being the next example), so it was nice to catch this early and fix it in such a way that I can do it right on the future instances of similar functionality.

As far as the code goes, it works for the leave form I've been describing here, but I'm sure as I work on the future similar functionality I'll find ways to make it more generic and flexible for more generalized use. The reliance on having to put related classes on the forms and inputs makes things handy for the JavaScript, but is an additional requirement as far as the development goes that may not ultimately be necessary.

But there you have it. It works, I learned a ton, and I share it here both so it might help someone else having to do this, and also so someone who knows more than I do can point out that I may be doing it in a more complicated way than necessary.

Friday, January 9, 2015

Towards a More Ergonomic Development Setup

I just turned 45. Forty. Frickin'. Five. For the past six months or so (probably right on schedule) I've been experiencing neck, shoulder, wrist, forearm, and elbow pain. It comes and goes but on the bad days it's bad enough to keep me awake at night, so I decided rather than continue to ignore it and keep working directly on a laptop as I have for years now, I better heed my body's warnings (as well as my girlfriend's prompting) so I can keep programming for another 45 years. (Joking. If I'm still blogging about nerd stuff when I'm 90 please put me out of my and everyone else's misery.)

I already made the move to a standing desk, specifically a GeekDesk, a few years ago. At the same time I bought a Herman Miller Embody chair, which I subsequently never used, because standing desk. (I'm reeeeal smart sometimes.) Seriously though both these changes made a huge, huge difference. I no longer have back issues, I have much more energy, and I feel much more alert and engaged when I work while standing. And that Herman Miller chair is absolutely the best chair I've ever owned. It's fantastic on the rare occasion I actually sit in it.

The desk and chair are only one piece of the ergonomic puzzle, however, and the years of twisted wrists on a laptop for way too many hours a day finally started to catch up with me, so this is day one of putting the laptop in a docking station, using an external 24" monitor on a monitor stand, and using an ergonomic keyboard and mouse.

So far, so good! My wrists and arms felt better nearly immediately, and with the monitor much higher my head and neck are now looking straight out instead of me being hunched over and leaning in to see my laptop screen. It feels weird, but it's good weird and I have a feeling after a week or so when I'm totally used to it, and my aches and pains have subsided, I'll feel much better than I have in a long time.

I have two new monitors on the way and a dual-monitor stand, but for now I'm using a Lenovo Easy Reach monitor stand with a ThinkPad Ultra Dock for my ThinkPad T540p. With my GeekDesk down to a height of 40" everything is looking and feeling great.

The biggest change in the setup for me is the new keyboard and mouse, and for that I went with the Microsoft Sculpt Ergonomic Desktop. Yes, the keyboard is a little weird and I think it'll take a few days for me to get back up to my full typing speed, partially because the keyboard is a lot different than what I'm used to, and partially because I (like most people, I assume) don't type 100% correctly. Who knew getting used to doing Bs with my left hand instead of my right would be such a challenge?

I've tried a lot of ergonomic keyboards over the years. I almost went with one I had and tolerated (I'll stop short of saying I liked it) a few years ago, namely the Logitech Wave Mk550, but then I remembered my two major annoyances about that setup. First, when you're used to a laptop keyboard moving to full-stroke keys feels like a TON more work, and my hands felt it. Second is the numeric keypad on the right-hand side of the keyboard. (Seriously, does anyone use those things?) Not only does that make the keyboard feel off-center to me, but it means you're reaching pretty far to get to the mouse (one of the things I LOVE about using a laptop, particularly a ThinkPad, is I don't have to move my hands to move the pointer around), and that gets annoying and hard on the elbow after a while.

The Microsoft Sculpt keyboard addresses both of these annoyances. It has very, very nice laptop-style keys that feel great under my fingers, and the numeric keypad is a completely separate piece of hardware that is currently residing in my closet where it will likely stay for eternity. This makes the mouse much closer so it's less annoying and hard on the forearm and elbow to grab the mouse. I'm still getting used to the keyboard layout but thus far I really like the feel of this keyboard.

The mouse itself is also designed to be ergonomic so it's a little bit funky; it's more of a tall ball than the flat oval-shaped mouse you may be used to. Combined with a mouse pad with a gel wrist pad it puts the hand and wrist in a much more natural, comfortable position, and there's even a nice notch for the thumb. Lefties beware, however: I don't see how you could use this mouse if you're not right-handed since the thumb notch would be on the wrong side.

Changing up all my equipment is a bit jarring but I'll get used to it quickly enough and my body will thank me for it. I'll be curious to see how I'm typing and feeling at the end of the first week with all this, and how things will change again when I get the dual monitor setup going. I'm also debating whether or not I'll need a keyboard tray mounted under the desk since I can adjust the height of the desk itself. Time will tell on that.

If anyone has any ergonomic tips as I move into my new setup I'd love to hear them!

Friday, October 10, 2014

Nginx and Extended Validation SSL Certificates

Quick tip on setting up Nginx with Extended Validation (EV) SSL certificates since this took a bit of trial and error for me this morning and I found a lot of conflicting and in some cases incorrect information while searching around.

If after configuring Nginx with your SSL certificate you're getting an untrusted certificate error, or a 400 error saying the certificate wasn't sent, you're likely missing the intermediate certificate bundle that's required on EV certificates, which if you use Verisign certs is located here:
https://knowledge.verisign.com/support/ssl-certificates-support/index?page=content&actp=CROSSLINK&id=AR2128

According to the Nginx documentation (pro tip: start with the docs, not with what you pull up on StackOverflow), you need to concatenate your host-specific SSL certificate and the intermediate certificate bundle into a single file, with the host-specific SSL certificate first in the file. So you'll end up with a single file (foo.crt) that contains three certificate blocks, and just make sure the one for your host (i.e. the one that goes with your key) comes first.

Once I had all that in place the browser was happy with what Nginx was giving it as far as SSL is concerned.

Friday, May 2, 2014

Ubuntu on Dell XPS 15 (8947-sLV)

The beta version of my excellent Dell Sputnik (XPS 13) wasn't quite cutting the mustard for some side freelance work I'm doing since it only has 4GB of RAM, but I loved the machine so much I opted to get its big brother, the XPS 15. Specifically I got this model, and yes since I needed it pronto I actually bought it in person at the Microsoft Store in Bellevue, WA. I'll always hate myself a little for going to a Microsoft Store but it was darn handy to have one nearby and just go get the new machine as opposed to waiting to have one shipped.

When I got it home I didn't even boot into Windows 8.1, I just stuck an Ubuntu 14.04 USB stick in the machine and installed that. The only trick to installing Ubuntu is you have to go into the BIOS, turn off Secure Boot, and turn on Legacy Boot Mode.

Once Ubuntu is installed everything works out of the box. The screen runs at the full, mind-blowing 3200x1800 resolution, and even the touch screen works. No issues with sound card, WiFi, or anything else. Awesome.

The only annoyance is the default settings for the Synaptics trackpad are a little jumpy. OK, more than a little jumpy. Luckily with a bit of research I found an easy fix that definitely did the trick for me. Basically you just need to change the FingerLow and FingerHigh settings to reduce the sensitivity and the cursor jumps around no more. I used the settings in the link but you can adjust as needed if those specific settings don't work for you.

Only remaining thing to figure out is what to do with this ridiculous amount of screen real estate. Couldn't be more pleased with this laptop!

Tuesday, January 21, 2014

Python + Oracle on Ubuntu Server 12.04

Affectionately known among all non-masochists in the world of IT as The Seventh Circle of Hell (with real hell being preferable), working with Oracle is always a hair-tearing nightmarish fork-in-the-eye please-for-the-love-of-god-kill-me-now experience that none but those who look to Ted Bundy, Jeffrey Dahmer, and John Wayne Gacy for moral and spiritual guidance would wish upon even their most reviled enemies.

Yes, it's that bad. And apparently nowhere is it worse than when one attempts to get Oracle working with Python on Ubuntu.

I'm not even talking about installing the Oracle database server itself here people, I'm just needing a Python application to talk to an existing Oracle database. One would think, as with every other database server on the planet (and yes, I'm including that other slice of hell SQL Server in that statement since it's a damn sight simpler to get working -- even on Linux -- than Oracle), you'd simply apt-get and/or pip install a library or two and be done with it.

If you actually do think that, you've already forgotten that this is Oracle we're talking about.

That said, one does what one has to do to keep the paychecks coming, so if you need to do this here's the steps to make it all happen. (Note that on Step 1 I'm assuming you have already installed all the other Python packages you may need. I'm focusing on the stuff you may not have that you definitely need.)
  1. sudo apt-get install libaio1 alien
  2. Download the RPM of version 11.2.0.4.0 of the Oracle client from http://www.oracle.com/technetwork/topics/linuxx86-64soft-092277.html (note that as of the date of this writing the 12.x version doesn't work, or at least didn't for me)
    1. You have to have an Oracle Web Account and use that to log in and download this, which makes using wget on the target server itself or automating the process for use with something like a Vagrant provisioning script rather problematic. Short version, you'll have to download this locally and then scp it up to the target server. What I did is downloaded and converted this RPM as well as the other necessary RPM, converted them once, and put them in a git repo from which I can clone in my Vagrant provisioning script. Whether or not that adheres to the licensing agreement, I don't know and I don't care. If you're paranoid, check with a lawyer before repeating my solution on this.
    2. If the version number differs slightly from what I have here, adjust later steps accordingly.
  3. Download the RPM of the Python 2.7/Oracle 11g version of the cx_Oracle Python libraries from http://cx-oracle.sourceforge.net/
  4. scp the RPMs up to the target server as needed.
  5. Convert the RPMs to Debian packages using alien:
    sudo alien -d FILENAME.rpm (where FILENAME is of course the name of each of the two RPM files)
  6. sudo dpkg -i oracle-installclient11.2-basic_11.2.0.4.0-2_amd64.deb
  7. sudo vim /etc/ld.so.conf.d/oracle.conf
    1. Note: this file won't already exist, so you'll be creating this as a new file in this step
  8. Enter the following in the newly created oracle.conf file and save it:
    /usr/lib/oracle/11.2/client64/lib
  9. export ORACLE_HOME=/usr/lib/oracle/11.2/client64
  10. export LD_LIBRARY_PATH=$ORACLE_HOME/lib
  11. sudo ldconfig
  12. sudo dpkg -i cx-oracle_5.1.2-2_amd64.deb
  13. cd /usr/lib/python2.7
  14. sudo mv site-packages/cx_Oracle* dist-packages
  15. sudo rmdir site-packages
  16. sudo ln -s dist-packages site-packages
  17. Verify installation by opening a Python interpreter and run the following:
    import cx_Oracle
    1. If you don't get an import error, everything is working properly
As far as automating this for use with Vagrant,  in my provisioning script I simply echoed the export statements in the steps above into /etc/environment, did source /etc/environment and followed that with ldconfig. Other than that the steps in the bash script are pretty much what's above, but if people are interested in seeing the script let me know and I can post it.

And there you have it. A lot of trial and error and head bashing went into that final solution, and since I kind of cobbled together the steps from various resources I'll post those below in case you want to see some of the other solutions and source material.

Happy Oracleing. Or not.

References

  1. http://iambusychangingtheworld.blogspot.com/2013/06/python-oracle-sqlalchemy-on-ubuntu-1304.html 
  2. http://maxolasersquad.blogspot.com/2011/04/cxoracle-on-ubuntu-1104-natty.html 
  3. https://linuxindetails.wordpress.com/2009/12/26/installation-of-python-cx_oracle-module-for-debian-squeeze/
  4. http://stackoverflow.com/questions/12538238/python-module-cx-oracle-module-could-not-be-found

Wednesday, October 2, 2013

Dropbox on Linux Mint 15

I noticed on Linux Mint that the Dropbox icons and menu options don't appear in my Dropbox directory. I did a little research and this is because by default when you install Dropbox it'll assume you're using Nautilus as your file browser, but by default Linux Mint uses Nemo (which is a fork of Nautilus).

Luckily it's an easy fix:
sudo apt-get install nemo-dropbox

Then quit all running instances of Nemo:
nemo --quit

When you open up your Dropbox directory again you'll see the familiar Dropbox icons.

FreeTDS on Linux Mint 15

If you're trying out or moving to Linux Mint and you're used to setting up FreeTDS on Ubuntu, you'll find that things are just slightly different on Mint.

Quick and easy fix:
sudo apt-get install freetds-bin tdsodbc

Also note that libtdsodbc.so in a different place on Mint than on Ubuntu, so instead of it being located at /usr/local/lib/libtdsodbc.so it'll be here:
/usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so