Friday, April 24, 2015

Python 3 + Oracle on Ubuntu Server 14.04

I'll invite my readers to check my previous post on this topic to get a sense of how I feel about Oracle.

Small updates it seems to get cx_Oracle working with Python 3 on Ubuntu 14.04. Assume sudo/root for all the following.

  1. Download the RPM versions of both the basic and SDK clients for 11.2.0.4 from http://www.oracle.com/technetwork/topics/linuxx86-64soft-092277.html
    1. Note: If you're not familiar with the idiotic hell that is Oracle downloads, you have to do this on a browser where you can click a radio button and THEN download, so wget from the server on wish you actually want to install this stuff is unfortunately a no go.
    2. The 12.x version of this stuff still doesn't work with cx_Oracle. At least not for me.
  2. scp the rpm files to the target server
  3. ssh to the target server
  4. apt-get install libaio1 alien
  5. For both the RPMs:
    alien -d RPM_FILE_NAME
  6. For both the newly created Debian packages:
    dpkg -i DEBIAN_PACKAGE_NAME
  7. touch /etc/ld.so.conf.d/oracle.conf
  8. echo "/usr/lib/oracle/11.2/client64/lib" > /etc/ld.so.conf.d/oracle.conf
  9. echo "export ORACLE_HOME=/usr/lib/oracle/11.2/client64" >> /etc/environment
  10. echo "export LD_LIBRARY_PATH=\$ORACLE_HOME/lib" >> /etc/environment
  11. source /etc/environment
  12. ldconfig
  13. pip3 install cx-oracle
  14. Put the relevant tnsnames.ora in $ORACLE_HOME/network/admin
Note that unlike my previous instructions you can install cx_Oracle using pip3 once you have everything in place. Maybe you could use pip before but I seemed to have better luck with the converted RPM, and at that point I was kind of tearing my hair out so I probably didn't go back to take a run at it using pip once the Oracle bits were sorted.

The big change here is the necessity to install both the basic client and the SDK client packages for Oracle; it worked with Python 2.7 on Ubuntu 12.04 without the SDK package.



Friday, March 20, 2015

Speaking at dev.Objective()

I'm very pleased to announce that I've been selected to speak at dev.Objective(), which is coming up in Minneapolis (more specifically Bloomington), MN on May 12-15. (And yes, if the conference name seems vaguely familiar this is the evolution of the former cf.Objective() conference.)

My session is Naked and Afraid: Mobile Offline Access to Emergency Data, in which I'll be discussing how to build cross-platform mobile apps that sync and cache data so you can have access to it even if the Internet and mobile networks are down.

Since I quit doing CFML/ColdFusion several years ago and moved over to Python I stopped speaking at and attending cf.Objective(), but it's a fantastic conference so I'm excited to be making my way back. It's going to be really great to reconnect with a lot of people I haven't seen in quite a while.

If you're a web developer looking for a great conference with a lot of great sessions presented by world-class speakers this is it! Hope to see you there.

Sunday, March 15, 2015

Resolving Mini WAN Miniport Code 31 Errors on Windows 8.1

After a recent Windows update I could no longer connect to one of my VPNs using the native Windows VPN connectivity (i.e. not a client like Cisco AnyConnect). When I tried to connect from the networking charm (I think that's what they call the bar on the right-hand side of the screen) it would hang for a while but never connect, and after that happened when I clicked on the networking icon on the lower right the charm bar wouldn't even come back.

I figured it might be a driver issue so I opened up Device Manager and sure enough, under the network adapters section there were yellow exclamation marks next to every one of the WAN Miniports that show up after you try to connect to a VPN.

I found varying reports of how to fix the problem but here's what worked for me.

  1. Try to connect to a VPN that will cause things to error out (the WAN Miniports didn't even show up in the network adapter list for me until I did this)
  2. Open Device Manager
  3. Expand "Network adapters"
  4. Right-click every one of the WAN Miniport devices (I even did the ones that didn't have a yellow exclamation point next to them) and do the following:
    1. Click "Update Driver Software"
    2. Click "Browse my computer for driver software"
    3. Click "Let me pick from a list of device drivers on my computer"
    4. Uncheck the "Show compatible hardware" box
    5. Under Manufacturer choose Microsoft
    6. Under Network Adapter choose "Microsoft KM-Test Loopback Adapter" (technically from what I understand you can pick anything that you can uninstall, but this one worked consistently for me)
    7. Click "Next"
    8. Ignore the "blah blah might be incompatbile" warning and click OK
    9. After Device Manager refreshes, right click the Microsoft KM-Test Loopback Adapter and click "Uninstall"
  5. After you've done that for every one of the WAN Miniports listed, reboot the machine. If you don't do this things won't be fixed.
That cleaned things up for me. We'll see if it works on subsequent reboots.

Friday, February 27, 2015

Cisco AnyConnect on Surface Pro 3

If you get the dreaded "failed to initialize connection subsystem" error when trying to use AnyConnect on the Surface Pro 3, or on Windows 8.1 in general for that matter (though it works fine for me on my ThinkPad T540p), there's a lot of differing opinions on how to fix it so I thought I'd share the one that worked for me.

What absolutely did not work for me (but that apparently works for other people based on frequency with which I encountered this solution) is just running AnyConnect through the Troubleshoot Compatibility wizard. That had zero effect for me.

What did work is this:
  1. Hit the Windows key and start typing "AnyConnect" to bring up AnyConnect in the search/launcher thingee (as I call it)
  2. Right-click on AnyConnect and choose "Open file location"
  3. Right-click on AnyConnect in Windows Explorer and choose "Properties"
  4. Click on the Compatibility tab
  5. Check the "Run this program in compatibility mode for" box and select "Windows 7" in the drop-down
  6. If you have a high-resolution display and AnyConnect looks fuzzy, and you care, you can also check the "Disable display scaling on high DPI settings" box
  7. Hit OK to save the changes.
  8. Reboot for good measure.
There have also been some previous issues with IE 11 updates breaking AnyConnect (yeah, wrap your head around that one) but those seem to have been resolved as long as you have your machine all patched up.

Wednesday, February 25, 2015

Dynamically Adding Forms To and Removing Forms From Django Formsets

For my latest big project I am running into numerous situations in which I need to dynamically add forms to and remove forms from Django formsets, which turned into an interesting and fun challenge, and led to a lot of learning about how Django formsets work along with some ancillary details (like the behavior of the jQuery UI datepicker when you manipulate the DOM) that I wasn't necessarily expecting to encounter.

To set the stage for the discussion by way of a concrete example, one aspect of the application in question is the ability to request leave (e.g. vacation, sick leave, etc.), and as you might imagine on the leave request page you can add days to and remove days from your leave request.

In the first iteration of this feature (random aside: we do Kanban and try our darndest to focus on "good enough" features, particularly on early iterations, so that we can deliver the most usable functionality in the least amount of time, looping back around to make improvements as time permits), each form in the leave formset has the following fields:
  • date of leave
  • start time of leave
  • end time of leave
  • hours of leave
  • minutes of leave
Why does the user have to specify the number of hours and minutes they're taking for their leave if they're providing the start and end times, you might ask? Basically it boils down to people with access to the leave calendar wanting to know the specific times people are out, which may not always match up with the amount of leave they're taking.

For example if you take a full day of leave, let's say you're out 8 am to 5 pm, but there's a lunch hour in the mix, so the clock time you're out is 9 hours, but the amount of leave you have to take for the full day is only 8 hours. And again, "good enough" is the focus on this first iteration. This basic functionality lets people request leave, and in future iterations, time permitting, we'll make the system smarter and remove some of the burden off the user with things like calculating hours based on start/end times, taking into account weekends and holidays, letting people select date spans, and all sorts of other niceties that are in our backlog.

But back to the discussion at hand, namely how this all happens with Django formsets. Since the last thing I wanted to do was have the leave form code as a variable in a JavaScript function, I decided to go the route of grabbing a new instance of the form via Ajax when the user clicks the "Add Day" button. That way the leave form code lives in one place, in a normal Django template, and is therefore much more maintainable.

This immediately got interesting, however, given how Django formsets work. By default (i.e. if you aren't using the prefix option in the formset constructor), forms in Django formsets will all have ids along the lines of id_form-0-field_name, and names along the lines of form-0-field_name, where 0 is the index corresponding to the form's position in the overall formset. So if you have multiple forms the first form will have an index of 0, the second one will have an index of 1, etc.

Also in the mix is the Django formset's management form, which contains the information necessary for Django to manage and process the formset, such as the total number of forms, the minimum and maximum number of forms allowed, and so on. The key point here is that if you're going to be dynamically adding and removing forms, you also need to be updating the form-TOTAL_FORMS value in the management form in order for things to stay in sync.

As you might be guessing by now, the syntax of the form fields in a formset doesn't quite mix with making an Ajax call out of the blue to an otherwise oblivious Django template to get a new form. Thankfully that was pretty easy to resolve by making sure to get the next form number and including that in the Ajax call:


And the form template (small snippet here) simply includes the form number in all the necessary places to make the form gibe with what the formset expects:


So when you get the form back from the Ajax call it has the correct number in all the fields, and all is right with the world.

Or so it would seem. [cue dramatic music]

At this point if all you ever want to do is ADD forms to a formset, this approach works perfectly. Where things get weird is when you start removing forms from your formset, since the aforementioned form indices get all kinds of screwed up (to use a technical term) if you don't adjust them as you go.

When I first started adding the remove functionality I was optimistically thinking, "Oh Django probably doesn't care about the form indices so much. I'm sure as long as the total number of forms it has in the management form matches the number of forms in the formset, it won't care if the indices go 0, 1, 4, 7, or whatever."

And that's actually partially true. I've gone through so many iterations of everything prior to writing this blog post that you'll have to forgive me for slightly fuzzy details here, but my recollection of the first incarnation of this feature is that yes, indeed, I could add and remove rows to my heart's content, and as long as I was ensuring the next form number was incremented, and the total forms value in the management form was right, I could post the formset, validate the formset, route back with errors, and that all worked.

Where things proved interesting is when our QA folks took a crack at this, they (as QA folks tend to do) did something I hadn't:

  1. Add a few rows to the form, and set the dates (N.B. we're using jQuery UI's datepicker for this; that fact becomes important in a moment)
  2. Add another new row and don't set the date
  3. Delete one of the rows in the middle of the set of rows
  4. Set the date using the datepicker in the last row that was added
At that point what would happen is that the date selected in the final row would change the value of the date in one of the rows above it. Also if you submitted the form that way with errors, when it came back and rendered the populated forms, some of the forms were blank.

This turned out to be a great find that set me down the path of learning a whole lot of stuff I'm glad I learned now before it bit me later.

Attacking the two problems (1. Django forms don't re-render correctly in an error state, and 2. datepicker changes the wrong date) in sequence, my first thought went back to the Django formset indices and my optimistic assumption that the indices didn't matter as long as the number of forms in the formset matched the total forms value in the management form.

This, not surprisingly, turned out to not be the case. As I said earlier if you only add forms things are incremented nicely and nothing gets out of whack, but I was now finding that when a form was removed, what needed to happen is that all the existing form fields needed to be reindexed so they started with 0 and incremented sequentially. A bit of JavaScript handled that without too much trouble. (I'll put the solution below since it involves problem #2 as well.)

Even after I got the form indices back into a sequential state as forms were removed, however, the datepicker was still behaving badly. I'd add and remove forms in random order and in some cases the datepicker would still change the wrong date field (in a predictable fashion with respect to the index), or throw an error that it couldn't find the form field to which it was attached.

That latter error led me to the solution for this problem. The long and short of it is that when you remove something from the DOM that has a datepicker attached to it, you have to destroy and reattach all the datepicker elements because each datepicker retains a reference to the field to which it was originally attached.

To sum up what has to happen when a form is removed for everything to behave properly:
  1. Resequence all the form indices so they start at 0 and increment sequentially; and
  2. Destroy all remaining datepickers and recreate them
To keep things simple the remove link on each form looks like this:


That way I didn't have to keep track of the index on the delete button too, I could simply reference its form container parent and go from there.

With that in place the delete row function wound up looking like this:


Main things of interest there are getting a handle on the delete link's parent form via the form class, and lines 13-16 where the datepickers are destroyed and rebound so they get attached to the correct form field.

And of course the call to updateFormElementIndices(), the bit that resequences the form indices, which was inspired by various snippets and other solutions I found to address this issue and looks like this:


Pretty straight-forward, but to go over what's happening there:

  1. Get a handle on all the forms of the class passed to the function and loop over them
  2. Get the current index number of the form being updated
  3. Update the form index to the loop increment (0-based, sequential)
  4. Get all the inputs with the class matching the form class passed in, plus 'Input' at the end
  5. Loop over the inputs, changing the IDs and names in similar fashion to how we updated the form ID
With all that in place Django was happy, the datepicker was happy, and (hopefully) our QA folks will be happy as well.

As I said this was a great find not only because I learned a ton slogging through this today, but also because this was the first incarnation of a pattern that will occur in numerous areas throughout this application (adding multiple phone numbers being the next example), so it was nice to catch this early and fix it in such a way that I can do it right on the future instances of similar functionality.

As far as the code goes, it works for the leave form I've been describing here, but I'm sure as I work on the future similar functionality I'll find ways to make it more generic and flexible for more generalized use. The reliance on having to put related classes on the forms and inputs makes things handy for the JavaScript, but is an additional requirement as far as the development goes that may not ultimately be necessary.

But there you have it. It works, I learned a ton, and I share it here both so it might help someone else having to do this, and also so someone who knows more than I do can point out that I may be doing it in a more complicated way than necessary.

Friday, January 9, 2015

Towards a More Ergonomic Development Setup

I just turned 45. Forty. Frickin'. Five. For the past six months or so (probably right on schedule) I've been experiencing neck, shoulder, wrist, forearm, and elbow pain. It comes and goes but on the bad days it's bad enough to keep me awake at night, so I decided rather than continue to ignore it and keep working directly on a laptop as I have for years now, I better heed my body's warnings (as well as my girlfriend's prompting) so I can keep programming for another 45 years. (Joking. If I'm still blogging about nerd stuff when I'm 90 please put me out of my and everyone else's misery.)

I already made the move to a standing desk, specifically a GeekDesk, a few years ago. At the same time I bought a Herman Miller Embody chair, which I subsequently never used, because standing desk. (I'm reeeeal smart sometimes.) Seriously though both these changes made a huge, huge difference. I no longer have back issues, I have much more energy, and I feel much more alert and engaged when I work while standing. And that Herman Miller chair is absolutely the best chair I've ever owned. It's fantastic on the rare occasion I actually sit in it.

The desk and chair are only one piece of the ergonomic puzzle, however, and the years of twisted wrists on a laptop for way too many hours a day finally started to catch up with me, so this is day one of putting the laptop in a docking station, using an external 24" monitor on a monitor stand, and using an ergonomic keyboard and mouse.

So far, so good! My wrists and arms felt better nearly immediately, and with the monitor much higher my head and neck are now looking straight out instead of me being hunched over and leaning in to see my laptop screen. It feels weird, but it's good weird and I have a feeling after a week or so when I'm totally used to it, and my aches and pains have subsided, I'll feel much better than I have in a long time.

I have two new monitors on the way and a dual-monitor stand, but for now I'm using a Lenovo Easy Reach monitor stand with a ThinkPad Ultra Dock for my ThinkPad T540p. With my GeekDesk down to a height of 40" everything is looking and feeling great.

The biggest change in the setup for me is the new keyboard and mouse, and for that I went with the Microsoft Sculpt Ergonomic Desktop. Yes, the keyboard is a little weird and I think it'll take a few days for me to get back up to my full typing speed, partially because the keyboard is a lot different than what I'm used to, and partially because I (like most people, I assume) don't type 100% correctly. Who knew getting used to doing Bs with my left hand instead of my right would be such a challenge?

I've tried a lot of ergonomic keyboards over the years. I almost went with one I had and tolerated (I'll stop short of saying I liked it) a few years ago, namely the Logitech Wave Mk550, but then I remembered my two major annoyances about that setup. First, when you're used to a laptop keyboard moving to full-stroke keys feels like a TON more work, and my hands felt it. Second is the numeric keypad on the right-hand side of the keyboard. (Seriously, does anyone use those things?) Not only does that make the keyboard feel off-center to me, but it means you're reaching pretty far to get to the mouse (one of the things I LOVE about using a laptop, particularly a ThinkPad, is I don't have to move my hands to move the pointer around), and that gets annoying and hard on the elbow after a while.

The Microsoft Sculpt keyboard addresses both of these annoyances. It has very, very nice laptop-style keys that feel great under my fingers, and the numeric keypad is a completely separate piece of hardware that is currently residing in my closet where it will likely stay for eternity. This makes the mouse much closer so it's less annoying and hard on the forearm and elbow to grab the mouse. I'm still getting used to the keyboard layout but thus far I really like the feel of this keyboard.

The mouse itself is also designed to be ergonomic so it's a little bit funky; it's more of a tall ball than the flat oval-shaped mouse you may be used to. Combined with a mouse pad with a gel wrist pad it puts the hand and wrist in a much more natural, comfortable position, and there's even a nice notch for the thumb. Lefties beware, however: I don't see how you could use this mouse if you're not right-handed since the thumb notch would be on the wrong side.

Changing up all my equipment is a bit jarring but I'll get used to it quickly enough and my body will thank me for it. I'll be curious to see how I'm typing and feeling at the end of the first week with all this, and how things will change again when I get the dual monitor setup going. I'm also debating whether or not I'll need a keyboard tray mounted under the desk since I can adjust the height of the desk itself. Time will tell on that.

If anyone has any ergonomic tips as I move into my new setup I'd love to hear them!

Friday, October 10, 2014

Nginx and Extended Validation SSL Certificates

Quick tip on setting up Nginx with Extended Validation (EV) SSL certificates since this took a bit of trial and error for me this morning and I found a lot of conflicting and in some cases incorrect information while searching around.

If after configuring Nginx with your SSL certificate you're getting an untrusted certificate error, or a 400 error saying the certificate wasn't sent, you're likely missing the intermediate certificate bundle that's required on EV certificates, which if you use Verisign certs is located here:
https://knowledge.verisign.com/support/ssl-certificates-support/index?page=content&actp=CROSSLINK&id=AR2128

According to the Nginx documentation (pro tip: start with the docs, not with what you pull up on StackOverflow), you need to concatenate your host-specific SSL certificate and the intermediate certificate bundle into a single file, with the host-specific SSL certificate first in the file. So you'll end up with a single file (foo.crt) that contains three certificate blocks, and just make sure the one for your host (i.e. the one that goes with your key) comes first.

Once I had all that in place the browser was happy with what Nginx was giving it as far as SSL is concerned.