Monday, March 26, 2012

Google Cloud Print

One of my many first world problems stems from the fact that I have both Comcast/Xfinity and Verizon Frontier FiOS for Internet in my house. I use Comcast for all my home/entertainment junk and Frontier for work.

This is all well and good until I have to print to my network printer from my work machine, because the network printer is on the Comcast side, not to mention that I wouldn't be able to hit it while I'm on the VPN for work anyway, and I print so infrequently that it's definitely not worth having two printers. (I told you this was a first world problem.)

Luckily there's a solution for this: Google Cloud Print. Google Cloud Print lets you register any printer and it then can be printed to from anywhere. If you don't have a cloud-ready printer (which I don't) you can follow these instructions to register any printer.

Note that this does not allow you to print from any application to this printer. You have to be printing something from Google Chrome specifically from what I can tell, but of course any document you need to print this way can first be uploaded to Google Docs and printed from there.

Handy stuff. Mark this first world problem solved.

Setting Default umask for SFTP on Ubuntu Server

Much as described in this blog post by Jeff Robbins, I have a situation where two sftp users in the same group are both uploading files to an Ubuntu 10.04 server using Dreamweaver. The issue is that by default the permissions are 755, so even though both users are in the same group, only the file owner has write permissions. Since the users need to be able to overwrite each other's files I needed a way to have the default permissions be 775.

What is outlined in the blog post above is exactly what I was after, but for some reason on Ubuntu server if you use what is the final edit in that post:

Subsystem sftp /usr/lib/openssh/sftp-server -u 0002

That results in "Connection closed" messages when you try to log in. The solution above that one works, just note the minor modification of pointing to /usr/lib/... instead of /usr/libexec/...

Subsystem sftp /bin/sh -cumask 0002; /usr/lib/openssh/sftp-server’

Restart ssh and you should be in business.

Thanks to Jeff for that very helpful blog post, and to Thad Meyer for pointing it out to me just last week (coincidentally enough).

Mongo GridFS Functions Now in OpenBD

Mongo GridFS functions are now in the OpenBD nightly release. Data in MongoDB is stored as BSON which currently has a 4MB size limitation so GridFS lets you store larger objects like files in Mongo.

You can see all the MongoDB functionality baked into OpenBD in the MongoDB category of the online manual.

Sunday, March 25, 2012

How to Move Your Posterous Blog to WordPress or Blogger

This may be premature since the fate of Posterous is still unclear after being bought by Twitter, but I decided given the uncertainty and since moving to Blogger was something I had been considering for a while anyway this was a good time to move everything off of Posterous.

I really do like Posterous and I think they do a lot of things about 1000 times better than any other blogging service. But if nothing else this points out yet again the fact that if you use a service of any kind you're at the mercy of that service or any company who may buy that service, so there's a lot to be said for controlling your own stuff.

As you may have read in the announcement to which I linked above Posterous is saying they'll provide tools to export your content to other services in the coming weeks, but not being one to wait around and see what happens I decided to get a jump on things and move some things to WordPress, and some others to Blogger, so I thought I'd share how I went about doing this and see if maybe there are better ways that I may have missed.

The interesting thing is WordPress is involved as a go-between even if you want to move to Blogger, but here's how I went about moving off of Posterous.

Moving From Posterous to WordPress

WordPress provides a Posterous importer so this is actually pretty easy with a few caveats.

If you have self-hosted WordPress and add the Posterous plugin, it's apparently very common to have it not work. I ran into this with the blog I was trying to move (specifically the ColdFusion Weekly podcast archives) from Posterous to a WordPress install on DreamHost.

Why DreamHost? I've used them in the past (been years though) for side projects now and then and especially for the money I've always had good luck with them, and since they offer unlimited disk space, bandwidth, databases, etc. for $100/yr I figured what the heck. I have a few small sites to move from Posterous to something else and this seemed like a good fit for these sites, especially given that the podcast files take up a chunk of disk space.

After I got DreamHost set up and used their one-click installer to install WordPress, I added the Posterous importer and tried to import the ColdFusion Weekly blog. No dice. I kept getting 403 errors which is apparently very common if you use the Posterous importer plugin on anything but wordpress.com.

The solution is to create a free blog on wordpress.com, run the Posterous importer from there, and then you can export from wordpress.com to WordPress's WXR XML format, and use that to import to your final destination.

Here's the steps:

  1. Create an account on wordpress.com if you don't already have one
  2. Create a new blog (the free one is fine)
  3. Click on "Tools" and then "Import" on the left-hand side
  4. Click on "Posterous" in the list of import tools that comes up
  5. Enter your Posterous login information and the URL for your Posterous site and click submit
  6. Wait :-)
If you have a ton of posts it can take a while to import but they send you an email when it's done.

Once the import to wordpress.com is done, you:
  1. Click on "Tools" and then "Export" on the left-hand side
  2. Leave the "All Content" radio button checked if you want everything (or choose what you want to export) and click "Download Export File"
  3. Go to the WordPress install where you want to import the blog content
  4. Click on "Tools" and then "Import" on the left-hand side
  5. Click on "WordPress"
  6. Select the export file you downloaded from wordpress.com and click "Upload file and import"
Then you're done, with some caveats.

What I ran into with the CF Weekly podcast blog is that Posterous didn't have the podcast files as attachments, so the link to each audio file showed up at the bottom of each post and was still pointing to posterous.com. That's no good.

To resolve that I used Firefox and the DownThemAll plugin to grab all the podcast files, and then SFTPd those up to DreamHost. At that point I did have to go through and add a link to each file in each post. This was relatively time consuming--maybe if Posterous provides real export tools at some point this won't be necessary. Also since WordPress (at least the install on DreamHost) has a file upload size limit of 7MB I had to do this manually and then they don't show up in the WordPress media library, so it was a bit of a hassle.

Main point here is once you import everything make sure and check the URLs for things like images, etc. so that they're not still pointing to Posterous and disappear if/when Posterous goes away or you decide to shut down your Posterous site.

Moving From Posterous to Blogger

Moving from Posterous to Blogger is a very similar situation since as I said to do a bulk import/export, at least from what I can tell, you have to use WordPress as an intermediary. The only other thing with Blogger is that Posterous does let you autopost from Posterous to Blogger so in theory if you set that up you could just auto-post each post from Posterous to Blogger, but note that Blogger does have an hourly limit on that so probably not the best best (not to mention there's no way to auto-post everything at once).

To get a blog from Posterous to blogger I did this:
  1. Follow the steps above to create a new (again, free is fine) blog on wordpress.com and import your Posterous blog to WordPress
  2. Follow the steps above to export and download your WordPress WXR file
So at this point you have your Posterous blog as a WXR file. Blogger doesn't let you import this, but thankfully there's a tool called wordpress2blogger that handles this quite nicely.

If your WXR file is < 1 MB in size (and you don't care about uploading the contents to a third party), you can upload your WXR file directly into wordpress2blogger and this will create and download an XML file that can be imported into Blogger.

If your WXR file is > 1MB in size (which mine for my main blog was), you can grab the code for wordpress2blogger and run it on your local machine. On Linux it wasn't bad to get this up and running at all since all the necessary Python stuff is already installed, but I can't speak to how much of a pain this would be on other platforms. I used the tool running locally on my main blog and it worked great.

Regardless of how you handle this (and there may be other tools out there), once you have your Blogger-format XML file you then:
  1. Log into Blogger and select (or create, if it doesn't exist) the blog to which you want to import your content
  2. Click on "Settings" and then "Other" on the left-hand side
  3. Click on "Import Blog," select your XML file, and hit submit
One thing to note is that at least in the case of a couple of blogs I've done, even though there's a checkbox to automatically publish imported posts, that didn't seem to work for me. So after the import is complete, you may have to go to Blogger's "all posts" page, hit the checkbox at the top of the list to select all your posts, and then click "Publish" to actually publish them.

Caveat here again is to check your asset paths. One of the blogs on which I used this method is the OpenCF Summit blog, and image paths are pointing to wordpress.com. Since that's free I guess it's not a big deal but it's definitely not self-contained so I may go back through and move all this stuff at some point.

Goes to show I should have been using something like S3 for my assets all along. :-)

Other Tools

Hopefully if/when Posterous does shut down they'll make good on their promise to make tools that let you export easily, but I came across another tool that lets you export everything from Posterous as a zip file. I haven't tried it, and it does cost ($9 for a single blog, $14 for multiple blogs under a single account), but this seems like a good way to just grab everything as static HTML so you at least have everything all in one place. I'm probably going to use this on my main blog as a backup even though I already have everything on Blogger (I'm mostly worried about asset paths and losing stuff on Posterous permanently) so if I do I'll follow up and let folks know how that went.

So there you have it. If you're more patient than I you can just wait and see what happens with Posterous, but if like me you want to jump ship now or have been thinking about doing it for a while anyway, these methods work well with the major potential issue being asset paths.

If you have other methods of doing this I'd love to hear them.

Saturday, March 24, 2012

Moved My Blog to Blogger

Quick note that I've moved my blog from Posterous to Blogger, and I did the same for the OpenCF Summit blog. Apologies for any hiccups with DNS, etc. during the transition. You may still see the old Posterous sites until DNS propagates.

Monday, March 19, 2012

CouchDB Resources List

Since I did quite a bit of research for my post on authentication and security in CouchDB I figured I'd share what I came across as a link dump. Enjoy!

Reference Material

General Info and Tutorials

CouchDB in Government

General Case Studies

Search

Saturday, March 17, 2012

The Definitive Guide to CouchDB Authentication and Security

With a bold title like that I suppose I should clarify a bit. I finally got frustrated enough with all the disparate and seemingly incomplete information on this topic to want to gather everything I know about this topic into a single place, both so I have it for my own reference but also in the hopes that it will help others.

Since CouchDB is just an HTTP resource and can be secured at that level along the same lines as you'd secure any HTTP resource, I should also point out that I will not be covering things like putting a proxy in front of CouchDB, using SSL with CouchDB, or anything along those lines. This post is strictly limited to how authentication and security work within CouchDB itself.

CouchDB security is powerful and granular but frankly it's also a bit quirky and counterintuitive. What I'm outlining here is my understanding of all of this after taking several runs at it, reading everything I could find on the Internet (yes, the whole Internet!), and a great deal of trial and error. That said if there's something I'm missing or not stating accurately here I would LOVE to be corrected.

Basically the way security works in CouchDB is that users are stored in the _users database (or elsewhere if you like; this can be changed in the config file), and security revolves around three user roles:

  • Server admin
  • Database admin
  • Database reader

Notice one missing? That's right, there is not a defined database reader/writer or database writer role. We'll get to that in a minute. And of course you can define your own roles provided that you write the functionality to make them meaningful to your databases.

Here's how the three basic roles play out:

  • Server admins can do anything across the entire server. This includes creating/deleting databases, managing users, and full admin access to all databases, i.e. full CRUD on all documents as well as the ability to create/modify/delete views, run compaction and replication, etc. In short, god mode.
  • Database admins have full read/write access (including design documents) on specific databases and can also modify security settings on a specific database. (I don't know if database admins can manage replication because I did not test that specifically.)
  • Database readers can only read documents and views on a specific database, and have no other permissions.

Even given all of this, reading and writing in CouchDB needs more clarification so you know what is and isn't allowed:

  • By default all databases are read/write enabled for anonymous users, even if you define database admins on a database. Note that this includes the ability to call design documents via GET, but does not include the ability to create/edit/delete design documents. Once you turn off admin party you have to be a server or database admin in order to manage design documents.
  • If you define any database readers on a database anonymous reads are disabled, but anonymous writes (of regular documents, not design documents) are still enabled.
  • In order to prohibit anonymous writes, you must create a design document containing a validation function in each database to handle this (much more on this below).
  • Regardless of any other settings server admins always have full access to everything, with the exception that if you create a validation function the admin user's access is impacted by any rules in that validation function. More on this below, but basically if you create a validation function looking for a specific user or role and the admin user doesn’t match the criteria, they'll be blocked just like anyone else.

Blocking Anonymous Writes

So now we come to the issue of blocking anonymous writes (meaning create/update/delete), and it's simple enough but I have no idea why this isn't done at the user level. Maybe there's a logical reason that isn't written down anywhere, but why you can't create a reader/writer user or role is a mystery to me.

But enough whining. Here’s how you do it.

To block anonymous writes you have to create a design document in the database that contains what’s called a validation function. This basically means that your design document must contain a validate_doc_update field, and the ID for this document follows the standard pattern for design documents, e.g. something like _design/blockAnonymousWrites The value of the validate_doc_update field is a function that will be run before all write operations, and it takes the new document, the old document (which would be null on create operations), and the user context in as arguments. This gives you access to everything you need to do simple things like check for a valid user, or more complex things like seeing if specific fields exist in the document that's about to be written or even if there are conflicts on an update operation with the old version of the document that you want to reject.

Here's a sample validation function that simply checks for a specific user name, foo, and rejects the write operation if the user is not foo:

function(new_doc, old_doc, userCtx) {   if(userCtx.name != 'foo') {     throw({forbidden: "Not Authorized"});   } }  

The userCtx object has properties of name and roles. The name property is the user name as a string, and roles is an array of role strings.

Let's say you wanted to limit write operations to the role bar. To accomplish that you'd use JavaScript's indexOf() method on the userCtx.roles array to see if the required role exists:

function(new_doc, old_doc, userCtx) {   if(userCtx.roles.indexOf('bar') == -1) {     throw({forbidden: "Not Authorized"});   } }  

Obviously on top of all of this you have access to all the properties of the document being posted as well as the old document if it's a revision, and you can use all that information to do whatever additional validation you need on the document data itself before allowing the document to be written to the database.

Creating Users

As far as creating users is concerned you can either do this in Futon or (as with everything in CouchDB) via the HTTP API. Note that if you create users via Futon you need to be aware that if you are logged in as admin and click the "Setup more admins" link you're creating a server admin. That means they have permission to do literally everything on that CouchDB server.

If you want to create a non-admin user make sure you're logged out and click on the "Signup" link, and you can create a user that way. Note that this doesn't work on BigCouch if you're hitting Futon on port 5984 since the _users database lives on port 5986 in BigCouch, and that backend port is by default only accessible via localhost; more on that below. And big thanks to Robert Newson on the CouchDB mailing list for pointing that out since I was tearing my hair out a bit after my recent migration to BigCouch.

If you want to create users via the HTTP API, in CouchDB 1.2 or higher you simply do a PUT to the _users database via curl or another HTTP tool, or make an HTTP call via your favorite scripting language. I'll show all the examples in curl since it's language agnostic and universally available (not to mention because I find curl so damn handy).

curl -X PUT http://mycouch:5984/_users/org.couchdb.user:bob -d '{"name":"bob", "password":"bobspassword", "roles":[], "type":"user"}' -H "Content-Type: application/json"  

That will create a user document with an ID of org.couchdb.user:bob and a user name of bob, and bob is not a server admin. In CouchDB 1.2 it will see the password field in the document and automatically create a password salt and hash the password for you.

On versions of CouchDB prior to 1.2, or with servers based on versions of CouchDB prior to 1.2 such as BigCouch 0.4.0 (which is based on CouchDB 1.1.1), the auto-salt-hash bit does not happen. This means you need to salt and hash the password information and store the hashed password and the salt in the user document.

As a reminder in case you weren't paying attention earlier: On BigCouch the _users database is on port 5986. This had me banging my head against my desk for the better part of an afternoon. It's probably documented somewhere but you know geeks and reading manuals, so I'm sharing that important tidbit in the hopes it helps someone else.

To create a user on CouchDB < 1.2 or BigCouch 0.4.0 (which again is based on CouchDB 1.1.1) you first need to:

  • Create a salt
  • Hash the concatenation of the password and the salt using SHA1
  • Include the salt used as the salt property of your user document, and the hashed password as the password_sha property of your user document

There are numerous ways to do all of this and you can see some examples in various languages and technologies on the CouchDB wiki, but since openssl is standard and a quick and easy way to do things I'll recap that method here.

First you need to generate a salt:
SALT=`openssl rand 16 | openssl md5`

Next echo that out just to make sure it got set properly:
echo $SALT

Next you concatenate whatever password you want + the salt, and then hash the password using SHA1:
echo -n "thepasswordhere$SALT" | openssl sha1

One caveat: if when you echo $SALT it contains (stdin) at the beginning like so:
(stdin)= 4e8096c4d0047e8d535df4b356b8d102

Make sure NOT to include the (stdin)= part in what you're going to put into CouchDB. Ignore (stdin)= and the space that follows and use only the hex value.

After generating a salt and hashing the password the end result that you put in CouchDB looks something like this (you'd obviously replace thehashedpassword and thesalt with the appropriate values):
curl -X PUT http://mycouch:5984/_users/org.couchdb.user:bob -d '{"name":"bob", "password_sha":"thehashedpassword", "salt":"thesalt", "roles":[], "type":"user"}' -H "Content-Type: application/json"

Of course if you know when you're creating the user that you want to grant them a specific role, you'd put that in the roles array. These roles will be contained in userCtx.roles in validation functions and you can act on that accordingly (see the above discussion about validation functions for more details).

And again note that if you're on BigCouch use port 5986 for the _users database!

Summary

To sum all this up, here's a handy-dandy chart.

If you want to ... You need to ...
  • Allow anonymous access to all functionality including creating and deleting databases
  • Do nothing! Leave admin party turned on. (At your own risk, of course.)
  • Disable anonymous server admin functionality (create/delete databases, etc.) but continue to allow anonymous read/write access (not including design documents) on all databases
  • Create at least one server admin user by clicking the "Fix this!" link next to the admin party warning on the lower right in Futon.
  • Allow a user who is not a server admin to have admin rights on a specific database
  • Create a non-server-admin user and assign them (by name or role) to be a database admin user on the specific database. This can be done via the "Security" icon at the top of Futon when you're in a specific database, or via the HTTP API.
  • Block anonymous reads on a specific database
  • Create a non-server-admin user in CouchDB and assign them (by name or role) to be a database reader on the specific database. This can be done via the "Security" icon at the top of Futon when you're in a specific database, or via the HTTP API.
  • Block anonymous writes on a specific database
  • Create a non-server-admin user in CouchDB and create a design document in the database that includes a validation function, specifically in a validate_doc_update property in the design document. The value of this property is a function (that you write) to check for a specific user name or role in the userCtx argument that is passed to the function, and you would throw an error in the function if the user or role is not one you want to write to the database.

And that's more or less all I know about CouchDB security. I'll end with some links if you want to explore further.

Any questions, corrections, or suggestions for clarification are very welcome. Hope some of you found this helpful!

Security/Validation Function Links

 

 

Saturday, March 10, 2012

A Reminder of the Power of CFML Custom Tags

Yeah, they've been around forever, and many people forgot all about custom tags when that whole CFC thing came about, but I still absolutely love custom tags and think they are incredibly useful in the view layer of an application.

I was reminded of this today while working on Enlist, which is an open source application for managing volunteers and events. It was again the focus on the hackfest at OpenCF Summit this year and we're pushing towards a 1.0 release before long.

One of the things that was added to the application at OpenCF Summit this year was DataTables, which is a really slick jQuery plugin that adds sorting, searching, and paging to HTML tables, and with their latest update it works fantastically well with Twitter Bootstrap.

I'm sure many of you are already familiar with DataTables but for those of you who aren't, the way it works is you simply add a little JavaScript at the top of a page containing a table to which you want to add the DataTables functionality. Identify the table by its ID and voila, you're done.

The trick comes in when you're using this on several pages, especially when you're adding a bit more to the JavaScript as far as specific functionality, placement of controls, etc. In that case what you wind up with is the same code on numerous pages with the only difference being the ID of the table  to which you're adding DataTables, and this doesn't give you much flexibility to do things like enable row clicking on one table but not another.

Enter the underappreciated CFML custom tag. This is a perfect use case for a custom tag, because this allows the JavaScript functionality of DataTables to be wrapped with a little CFML to add some flexibility and intelligence to what otherwise would be a lot of copy/pasta JavaScript code.

You can see the code for the custom tag on GitHub, but basically this wrapper for the DataTables JavaScript lets a table ID, table body ID, and row link be passed in, and that's then applied appropriately.

As for using the tag on a page, it's as simple as importing the tag library and then wrapping the HTML table with the custom tag:

<!--- datatable.cfm custom tag lives in /enlist/customtags ---> <cfimport prefix="tags" taglib="/enlist/customtags" /> ... <tags:datatable tableID="myTableID" tableBodyID="myTableBodyID">   <table id="myTableID">     <thead>       <tr>         <th>Foo</th>         <th>Bar</th>         <th>Baz</th>       </tr>     </thead>     <tbody id="myTableBodyID">       <tr>         <td>Foo!</td>         <td>Bar!</td>         <td>Baz!</td>       </tr>     </tbody>   </table> </tags:datatable>

There's a bit more to it in its real-world use which you can see here since it does row linking as well.

Simple enough, but oh so powerful. Now I have a flexible, reusable wrapper for DataTables that I can drop into any page and customize further as needed.

So as you're messing with all the highfalutin doo-dads we have at our disposal in CFML these days, don't forget about the lowly custom tags because they can add a ton of flexibility and power to your view layer.

 

Wednesday, March 7, 2012

SiliconDust HDHomeRun PRIME and MoCA

Quick tip if you're trying to get a networked TV tuner like the SiliconDust HDHomeRun PRIME working on the same jack with a MoCA box -- short answer for the impatient among you is you're going to need a diplexer. You can get something like this one which splits the signal into 5-860MHz and 950-2150MHz ranges, or something like this basic satellite/antenna diplexer from Radio Shack.

Plug the coax from the wall into the input of the diplexer, and send the lower range (which is labeled VHF on the basic diplexers) to the HomeRun and the higher range (which is labeled VHF on the basic diplexers) to the MoCA box and everything should work. Without this, when I was going wall -> MoCA box and MoCA box -> HomeRun, networking worked fine for me but there was no cable signal being sent to the HomeRun.

So longer explanation for those of you who follow my home networking trials and tribulations, with the threat of Moxi going away (though this has been put on hold indefinitely) I decided to investigate other solutions and I wasn't at all keen on spending $1.7 trillion on TiVos. (OK it's not really that much but it adds up quick for a three-room setup when you get the lifetime subscription.)

My brother pointed me to a deal on an Acer Revo HTPC that was too good to pass up, so I got that and an HDHomeRun for the tuner. Yes, I know, I'm always talking about how much I hate Windows, I'm an open source bigot, etc. etc. but the reality of the situation is that I'm not the only one in my house that watches TV so I have to at least be semi-congizant of ease of use, and much as I keep saying I'm going to build a MythTV box, every time I investigate things like remote controls, backend vs. frontend boxes, etc. it seems to get complicated rather quickly. If it were just me watching TV I'd go for it, but some people in my house just like to turn on the TV and have it work without any hassle. (Can you imagine?)

The good news on the MythTV front, however, is buying the Revo freed up another desktop PC I was previously using just for PlayOn and PlayLater so I'm going to put MythTV on there, and from what I've read it works with the HomeRun! At least that way I can maybe have MythTV in the man cave and have something more easy to use (in theory anyway) in the living room.

Back to the networking piece of this. When I had Verizon FIOS TV I had been using MoCA for networking and it worked great. When I switched from Verizon to Comcast, however, the MoCA didn't work in the rooms where I needed a cable TV signal and I didn't do enough research at the time to figure out why. I assumed it was a signal strength issue so I tried some amps but none of them worked.

Instead of digging into that more at the time I moved to powerline networking and while it's been decent overall, it's never been as solid and fast as the MoCA was and it wasn't working very well for the Revo to talk to the HomeRun. The picture would get jerky pretty often or the audio and video would get out of sync, so that wasn't going to fly. (That said, powerline networking worked quite reliably for streaming HD from the main Moxi unit to Moxi Mates in other rooms, so don't shy away from it based solely on me switching back!)

This prompted me to switch back to MoCA since that had worked fantastically well for me before, but I again ran into the issue with the cable signal. This time I did my homework and discovered that most people that were having this problem resolved it by using a diplexer, and luckily that fixed things for me too.

Next up is putting more RAM and a bigger hard drive in my other desktop machine and trying out MythTV so when I get a chance to do that I'll be sure and share how that goes.