We're in the process of porting an application that has been running on SQL Server over to the fabulous and amazing CouchDB. We were originally under the impression that everyone accessing data from this application in their own code was doing so through our web service, which would have made our job pretty simple since we could swap the guts of the web service methods out and return the same data types to the caller, but upon further investigation we discovered that people had written their own custom queries directly against the database.

This alone isn't a big deal but in some cases people were running queries that included LIKE clauses, and since we opted not to install CouchDB-Lucene given both time constraints as well as the fact that the LIKE queries against SQL Server were pretty limited in scope and number, I thought I'd share what we came up with to do string matching in views in CouchDB.

This is by no means to suggest you should not use CouchDB-Lucene if you want true full-text searching against data in CouchDB, but in our case this was an acceptable compromise.

Matching Fields That Start With a String in Couch

SQL Equivalent: "WHERE field LIKE 'foo%'"

Let's assume I have a database called test and in that database I have documents that have fields of firstName and lastName. I want to write a view that will let me do wildcard matches against first names that begin with a string.

This turns out to be pretty simple given how keys work in CouchDB map functions. Since a view emits a key and a value and we can use start and end keys in our calls to CouchDB, we simply provide the string against which we want to match as our start key and some end key that will ensure we don't get back more than what we're wanting.

For example, let's say I want to match all documents in my database that start with 'Mat' so I can retrieve all people with a first name of Matt, or Matthew, or Mathew, or Mat, or Mathias ... you get the idea.

First I write a view that in its map function emits firstName as the key:

function (doc) {
  if (doc.firstName && doc.lastName) {
    emit(doc.firstName, doc);
  }
}

Assume that my design document is 'people' and that's the map function for a view called 'byFirstName.' To call that view and get back only people with a first name staring with 'Mat' I use the following URL:

http://couch/test/_design/people/_view/byFirstName?startkey="Mat"&endkey="MatZ"

In case that wraps poorly in the blog post display, here's just the start and end keys:

startkey="Mat"
endkey="MatZ"

That tells CouchDB to start its output for that view with anything that starts with Mat and end once it hits anything that starts with MatZ.

Matching Specific Strings Contained in Fields

SQL Equivalent: "WHERE field LIKE '%KnownString%'"

We had some use cases where users had canned queries (i.e. users can't enter random search terms) that were looking for a specific term contained anywhere within a specific field. I say specific term here and in the example I use "KnownString" because if you know the string ahead of time this is a simple problem to solve, whereas ad hoc terms are more problematic, but I'll address that below.

Remember that within CouchDB views you have full access to JavaScript, so solving this use case is simply a matter of using a regex to match against the known term.

Let's say I want to pull all documents that have a bio field containing the term 'CouchDB':

function(doc) {
  if (doc.bio && doc.bio.toUpperCase().match(/\\bCOUCHDB\\b/)) {
    emit(doc._id, doc);
  }
}

Again, since I know the term ahead of time I can do a regex match against it quite easily in my view.

Matching Ad Hoc Strings Contained in Fields

SQL Equivalent: "WHERE field LIKE '%adHocSearchTerm%'"

Where things get tricky in CouchDB without using something like CouchDB-Lucene is matching ad hoc strings. "Tricky" is actually putting it mildly, because the real story is you can't do this in CouchDB. So in use cases where people had code that had a search box into which users could type anything, we had to come up with another solution.

What I've found as I've been using CouchDB more and more is that it can shift things that you used to do in the database layer up into the application layer, and vice-versa. So in this case it was simply a matter of coming up with a view that pulled back a subset of documents into the application code, and then doing the matching there.

One caveat here is that since our database contains thousands of documents, it wasn't really feasible to pull back all the documents in the database and then perform matching in the application layer. Since these documents all have a date associated with them, what we wound up doing is using date range as start and end keys as a way of reducing the number of documents we have to match against in the application. This wasn't a huge burden on users and certainly will improve performance.

We wound up limiting documents returned by year (i.e. the users have to choose a year in which to search), which is enough of a range to not make things too annoying for users, but is also a small enough set of documents not to kill performance on the application side.

To call the view that uses date as its key, the URL params look like this to pull back all documents for 2011 in descending date order:

?startkey="2012/01/01"&endkey="2011/01/01"&descending=true
 

Remember that when you order descending you essentially flip the start and end keys around, hence why 2012/01/01 is used as the start key.

Once I have the documents back, I then deserialize the JSON into something usable by CFML and then loop over the documents to do my further refinement by search term.

Leaving out the subset controlled by date I described above, assuming I wanted to find all people with a bio field that contained the search term entered by a user on a form, the code winds up looking something like this:

<cfhttp url="http://server/test/_design/people/_view/hasBio" 
        method="get" 
        result="peopleJSON" />
 
<cfset peopleReturned = 
        DeserializeJSON(peopleJSON.FileContent).rows />

<cfset matchingPeople = ArrayNew(1) />

<cfloop array="#peopleReturned#" index="person">
   <cfif FindNoCase(form.searchTerm, person.value.bio) neq 0>
    <cfset ArrayAppend(matchingPeople, person) />
  </cfif>
</cfloop>

What we wind up with there is the matchingPeople array will contain only the people who had the search term included in their bio field.

The big caveat here again is that if you have a huge number of documents you can get into trouble on the application side, so make sure and limit what you get back from CouchDB since you'll wind up looping over all of those documents to do your search term matching.

Hope that helps others do some quick and dirty LIKE type queries in CouchDB. If there's a better way to do any of these I'm all ears!

Quick note to anyone planning to attend my "Running Multiple CFML Engines on Apache Tomcat" talk at cf.Objective() -- even though this is only a one-hour session, with just a bit of prep work you can easily turn this into a hands-on session since I only have a few slides and it will be mostly demo. You don't have to follow along to get a ton of great info from this session, but if you want to follow along please grab the following ahead of time:

• Java (I assume this is already on most attendee's machine and configured properly)
• Apache Tomcat
• Apache Web Server
• Dealer's choice/mix and match: Open BlueDragon, Railo, and Adobe ColdFusion

Some additional notes:

• You do NOT need to install Tomcat ahead of time
• You SHOULD install Apache ahead of time
• If you want to use Adobe CF as one of your engines, you'll want to run the installer ahead of time and for the installation type choose "generate a WAR file" and have that available on your laptop. Note that even if you have Adobe CF installed on your machine already, you can run the installer again and generate a WAR file without affecting your existing installation.
• For Open BlueDragon and Railo, grab the WAR files and have those handy
• Your operating system doesn't matter--all the Tomcat stuff is pure Java, so whether you're on GNU/Linux, Windows, or Mac it's all good.

If you have questions/concerns ahead of time please comment here or email me. See you at cf.Objective()!

See PDF for details. Contact Justin De Gennaro at jdegenna@teksystems.com for more information.

Click here to download:

cf_developer_position.pdf (142 KB)

(download)

Click here to download:

cf_developer_position.pdf (142 KB)

Beginner Coldfusion Developer
Job Type: Full Time
Location: Roseville, CA with telecommuting available
Skills: ColdFusion (8.0)

WebEvents Global is seeking beginner ColdFusion developers to develop and maintain event management web sites, intelligent customer-focused websites, and automated marketing. Successful candidates will have 2-4 years of ColdFusion experience including 1 year of CF8, be excellent problem solvers, fast coders, and have superb and verifiable references.

WebEvents Global is a web-based event management software company headquartered in Roseville, California. We offer custom event management solutions from registration, campaign marketing, scheduling tools, sales automation tools, content management tools, and beyond.

Applicants MUST Have:

• 2-4+ years of ColdFusion experience preferably with 1+ year of CF8
• 1-3+ years experience in SQL Server 
• Demonstrated and successful project completion 
• Experience with web services is preferable 
• Exposure to frameworks like Fusebox  
• 100% fluency in English 
• Should possess excellent communication skills.

Additional consideration will be given to candidates who have:

• Experience with the event industry
• Experience with AJAX or jQuery 
• ActionScript experience 
• Thorough understanding of version control is necessary

On location or remote work is negotiable.

Benefits:

• Length (Permanent At-Will)
• Pay is negotiable based on experience 
• Medical 
• Dental 
• Vision 
• 401k plan
• 2-weeks PTO/year + 1 mandatory week off between Christmas - New Year 
• Paid company holidays

Please send your resume along with salary requirements to jobs@webeventsglobal.com

I've been helping my friend Brandon a bit today with Open BlueDragon, specifically with manipulating and displaying images that are stored in a database as BLOB data. Not terribly tricky stuff but it is a bit different than how you might deal with this in Adobe ColdFusion so I thought I'd write up a little how to.

In case you want to experiment from the ground up, and this also helps illustrate everything in detail, let's start by creating a database to hold BLOB data. I'll use MySQL, but this works exactly the same (as far as I know anyway) with any database. First, create a new database in MySQL, and then create a files table:

CREATE TABLE files  
(id int unsigned auto_increment not null primary key,  
file longblob not null);

Now we have a basic table with an auto-increment integer ID and our file data as a LONGBLOB. With your database in place, create a datasource in the OpenBD admin console called blobTest and point to this database.

Next let's create a simple file upload page called imagetest.cfm:

<form action="imagetest2.cfm" 
        method="post" 
        enctype="multipart/form-data">
    File: <input type="file" name="theFile" /><br />
    <input type="submit" />
</form>

Nothing fancy there. This form submits to imagetest2.cfm which looks like this:

<!--- upload the file --->
<cffile action="upload" 
        filefield="theFile" 
        destination="#ExpandPath('.')#" 
        nameconflict="makeunique" />

<!--- read the file into a variable --->
<cffile action="readbinary" 
        file="#ExpandPath('./#CFFILE.ServerFile#')#" 
        variable="fileBinary" />

<!--- delete the file from disk --->
<cffile action="delete" 
        file="#ExpandPath('./#CFFILE.ServerFile#"')#" />

<!--- insert the binary data into the db --->
<cfquery name="insertBlob" datasource="blobTest">
    INSERT INTO files (file) 
    VALUES(<cfqueryparam value="#fileBinary#" 
                    cfsqltype="cf_sql_blob" />)
</cfquery>

File inserted.

In imagetest2.cfm we upload the file, read the file from disk using the READBINARY action of CFFILE, delete the uploaded file from disk (since we're storing the image in the database we don't need the file on disk any longer), and then insert the binary data into the database using CFQUERYPARAM and a CFSQLTYPE of CF_SQL_BLOB. So it's clear, the file field in the database contains the binary representation of the file that was uploaded.

Finally let's take a look at how to pull the binary data for the image back out of the database, resize it, and then display it in the browser, which we do in a file called imagetest3.cfm:

<!--- get the most recently inserted image out of the db --->
<cfquery name="getimages" datasource="blobTest">
    SELECT id, file 
    FROM files 
    ORDER BY id DESC 
    LIMIT 1
</cfquery>

<!--- resize the image to a width of 200 pixels --->
<cfset imgNew = ImageNew(getimages.file) />
<cfset ImageResize(imgNew, 200) />

<!--- output the image to the browser (assuming image is jpg) --->
<cfoutput>
    <img src="#ToDataUri(imgNew, 'image/jpg')#" />
</cfoutput>

Here we first grab the most recently inserted file from MySQL. If you want to display the image in the browser without manipulating it, meaning what's in the database is exactly what you want to show to the user, you can skip down to the CFOUTPUT section and use getimages.file as the first argument to the ToDataUri() function. I'll elaborate on ToDataUri() in a second.

In Brandon's case he was asking how he would resize the image and do some other manipulation after getting it out of the database, which luckily is pretty simple. First we use the ImageNew() function to create a new image object since this is the type of object that all the image manipulation functions work with.

Next we resize the image to a width of 200 pixels using ImageResize().This function takes an image object as the first argument, and a width as the second argument. An optional third argument is height, and an optional fourth argument is quality. For this example we're just blindly resizing to a width of 200 pixels no matter what the original size was (which of course isn't likely what you'd be doing in a real world app), and if you omit height it will change the height proportional to the width. At this point we have the variable imgNew as binary image data and the image has been resized to a width of 200 pixels.

Now we want to output the image to the browser. Adobe ColdFusion accomplishes this via the CFIMAGE tag with an action of WriteToBrowser, but WriteToBrowser is not implemented in Open BlueDragon. Instead, Open BlueDragon introduced the ToDataUri() function, which is a more flexible, standardized way of writing binary data to the browser. ToDataUri() is not specific to image data and can be used with data of any valid MIME type, and the MIME type is specified in the second argument of the ToDataUri() function.

The data URI scheme allows you to call inline data on a page as if it were an external HTTP resource. You can see what the URI looks like by right-clicking on the image displayed in your version of imagetest3.cfm and choosing "view image" in Firefox (or the equivalent in another browser). Mine looks like this:

data:image/jpg;base64,/{base64_data_here}

Where {base64_data_here} is the base 64-encoded version of the image in question. ToDataUri() converts binary data to base 64 on the fly so you don't need to do that yourself ahead of time using functions like ToBase64() or ImageReadBase64().

That's it for this little tutorial. Hope it helps others dealing with BLOB data in OpenBD.

In my previous few posts on Open BlueDragon for Google App Engine (GAE) I've done a basic introduction to GAE, discussed the Google Datastore, and dug into how to use the virtual file system and the GAE Image Service to handle file uploads and simple image processing.

In this post we'll pick back up with my sample app (.tar.gz, 15.3 MB) from OpenCF Summit 2011 and look into two more of the excellent services available to you on GAE, specifically the mail and XMPP services.

Mail? So What?

Sending email is of course a big yawn--anyone can do that from any platform quite easily. But what if you want your application to be able to receive mail and take actions based on the mail it receives?

The standard solution is typically to set up a dummy email address (myapp@foo.com) and then have your application poll the mailbox for new mail, which in the case of a CFML app means setting up a scheduled task to do a CFPOP call to the mailbox in question. You then get back a list of messages and have to handle them as needed for your application.

This works, but it's a pretty passive, braindead approach to solving the problem, and you wind up having to write a bunch of code to manage a mailbox designed for humans that's being used by your application.

With GAE you can have your application respond to incoming email directly without having to set up any mailboxes, because the ability for your app to receive mail is built right into the GAE platform. You simply enable mail services in your appengine-web.xml file, configure specific email addresses or wildcard email addresses in web.xml, and your application can receive mail.

Configuring Inbound Services

Let's take a look at appengine-web.xml, which if you're following along with the sample app is in the war/WEB-INF directory. Towards the bottom of the file you'll see a block of inbound services defined:

<inbound-services>
    <service>xmpp_message</service>
    <service>xmpp_presence</service>
    <service>xmpp_subscribe</service>
    <service>mail</service>
</inbound-services>

Note that by default there are no inbound services enabled on your GAE application, so you do have to specify these. We'll be covering the XMPP services in this post as well, so you can see in the block above this is where specific granular XMPP services such as messaging, presence, and subscription are enabled.

That's it! With that simple block in appengine-web.xml when you deploy your application to GAE, it will be ready to receive email and interact with XMPP messages.

At this point you're probably wondering basic things like the address to which you can send email, how to handle the email once it's received, how to configure specific email addresses, and a host of other things, so let's dig in a bit more.

How the Magic Works

Once you have mail defined as an inbound service, your application essentially gets a mail server on the internet at the address YOUR_APP_ID.appspotmail.com. So with mail in the inbound services block in your config file, once you fire up your app on GAE mail can be sent to ANY_ADDRESS@YOUR_APP_ID.appspotmail.com. We'll talk more about how the ANY_ADDRESS bit works in a minute when we look at writing code to handle incoming mail.

Google very cleverly designed many of their services to send messages to your application via an HTTP POST to a specific URL. In the case of inbound mail, any mail sent to ANY_ADDRESS@YOUR_APP_ID.appspotmail.com generates an HTTP POST to the URL /_ah/mail/{EMAIL_ADDRESS} with email address representing the full email address including the YOUR_APP_ID.appspotmail.com domain.

Since there's a lot of magic behind the scenes that you don't have to configure, let's review how this works before moving on to the code for handling incoming mail.

1. You enable mail as an inbound service in appengine-web.xml
2. When deployed to GAE, your app can then receive email at ANY_ADDRESS@YOUR_APP_ID.appspotmail.com
3. When mail is received at YOUR_APP_ID.appspotmail.com, this generates an HTTP POST to the url /_ah/mail/{EMAIL_ADDRESS} within your application

Now let's look at how to write code to process incoming mail.

Processing Inbound Mail

With very little effort on our part our application can receive mail. That's cool in theory, but how do we write code to react to this inbound mail?

Since all mail comes in as an HTTP POST, the way you handle inbound mail is more or less the same way you'd handle any POST request to your application. This makes handling all of this very simple since handling POST requests is something with which we're all too familiar.

From a CFML standpoint there's a bit of a disconnect here since, at least until I figure out a clever hack, the thing that receives the mail has to be a Java servlet. Not to worry, the code's quite straight-forward and we can hand off any real work we want to do to CFML once we get the mail in the servlet.

Bit of a detour to explain why (at this point) CFML can't handle the inbound mail directly. In order for your application to take action when mail is received, you have to define a servlet mapping for the special incoming mail URL of /_ah/mail/{EMAIL_ADDRESS}. Servlet mappings do support wildcards, so for the sample application I've defined a servlet mapping of /_ah/mail/* that will route all inbound mail, regardless of the specific email address, to the same servlet.

You could of course define specific servlet mappings for specific addresses, e.g. one servlet mapping for /_ah/mail/user1@yourappid.appspotmail.com and another for /_ah/mail/user2@yourappid.appspotmail.com if you needed two completely separate servlets to handle the different users. I hinted at this above, but I'll point out here again that you do not need to configure mailboxes or anything along those lines for specific users. Your application will receive absolutely any mail sent to ANYONE@yourappid.appspotmail.com and how you handle the ANYONE part is up to you to define in your code.

Back to the CFML discussion. When I was first messing with the in the local GAE environment, I first thought I'd be clever and just define a real directory path in my application of /_ah/mail, throw an index.cfm file in there to act as a controller, and I'd be off to the races. That actually kind of worked locally, but the issue is when you deploy to GAE it won't let you deploy your application if you have /_ah as a real directory since that's a reserved directory for things like the admin console.

With that door shut, I next figured I could define the servlet mapping for /_ah/mail/* to point to the CFML servlet like so:

<servlet-mapping>
    <servlet-name>cfmServlet</servlet-name>
    <url-pattern>/_ah/mail/*</url-pattern>
</servlet-mapping>

This kind of worked as well, meaning I could at least see it was hitting the CFML servlet since I could dump things in Application.cfc and see what was going on. The issue here turned out to be that I coudln't really leverage anything like onRequestStart in Application.cfc and take any action there, because when a POST comes in to /_ah/mail/* there's no page argument, which is what onRequestStart expects.

I'm convinced there's a way to get this to work (or I'm at least convinced I haven't yet exhausted all avenues of failure here), but since I was under a time crunch to get this rolling for OpenCF Summit, I decided to write a simple servlet to handle inbound mail and then hand things off to CFML for some additional work. I haven't had time yet to revisit this so if anyone has any ideas based on what I'm describing of how we could get mail to hit CFML directly I'd be very interested to hear them.

The MailReceiverServlet

In the sample app project the code for the MailReceiverServlet is in the src directory, and then in the package org.opencfsummit. As I mentioned above it's pretty straight-forward, but let's go over the mail specific parts since this will help shed some light on how mail comes in via the POST to /_ah/mail.

Inside the servlet's doPost method we first declare some variables and set some defaults starting on line 23:

Properties props = new Properties();
Session session = Session.getDefaultInstance(props, null);
MimeMessage message;
Address[] toAddresses = null;
Address[] fromAddresses = null;
ArrayList<String> recipients = null;
ArrayList<String> senders = null;

Couple of things to note here. First, just so there's no confusion the Session in this case is a javax.mail.Session, not an HttpSession. So you can see here we're dealing with actual Java mail stuff when the POST hits our servlet, which is also evidenced by the MimeMessage and javax.mail.Address arrays. The two string ArrayLists at the end we'll use in a minute as an easy way to pass off the data we want to be available in CFML.

Next up in MailReceiverServlet we get the data from the inbound mail and parse out the to and from addresses:

try {
    message = new MimeMessage(session, request.getInputStream());

    toAddresses = message.getAllRecipients();
    recipients = new ArrayList<String>();

    fromAddresses = message.getFrom();
    senders = new ArrayList<String>();

    for (Address address : toAddresses) {
        recipients.add(address.toString());
    }

    for (Address address : fromAddresses) {
        senders.add(address.toString());
    }
} catch (MessagingException me) {
    // gulp
}

Let's step through that code briefly.

First, we instantiate a new MimeMessage using a mail session and the data coming from the request's input stream. Once we have a handle on the message, we can get all of the to addresses contained in the message (message.getAllRecipients()) and all the from addresses as well (message.getFrom()).

We then loop over each address array (toAddresses and fromAddresses) and add each address as a String to the ArrayLists (recipients and senders).

With the simple string versions of the addresses in ArrayLists, we'll then set these as attributes in the request object and forward the request to a CFML page:

request.setAttribute("senders", senders);
request.setAttribute("recipients", receipients);
getServletContext().getRequestDispatcher("/mailhandler.cfm").forward(request, response);

What this last chunk of code does it puts the senders and recipients ArrayLists in the request object, which translates into the request scope on the CFML side, and then forwards the request to mailhandler.cfm for some additional work.

And of course we have to define this servlet and a servlet mapping in our web.xml:

<servlet>
    <servlet-name>mailReceiver</servlet-name>
    <servlet-class>org.opencfsummit.MailReceiverServlet</servlet-class>
</servlet>
<servlet-mapping>
    <servlet-name>mailReceiver</servlet-name>
    <url-pattern>/_ah/mail/*
</servlet-mapping>

mailhandler.cfm

With the initial incoming mail request handled by our MailReceiverServlet and the request forwarded to mailhandler.cfm (which is in the war directory if you're following along in the sample app), we can make quick work of the rest of our task.

This might not be the most practical example, but for the purposes of demonstration what we're doing in mailhandler.cfm is if anyone sends an email to stallmanbook@opencfsummit2011.appspotmail.com the application will respond by emailing a copy of Richard Stallman's Free Software, Free Society as a PDF attachment. Of course you can do anything you want when your app receives an email but this is the little sample I dreamed up. And this is live by the way, so go ahead and try sending an email to stallmanbook@opencfsummit2011.appspotmail.com to get your free gift. ;-)

In mailhandler.cfm we first check to see if the recipients array (which is the ArrayList of strings that we created in MailReceiverServlet and then put in the request object) contains the appropriate email address for this particular action:

<cfif request.recipients.contains("stallmanbook@opencfsummit2011.appspotmail.com")>

Remember that when we configured the servlet mapping for MailReceiverServlet we used a wildcard in the URL pattern (/_ah/mail/*), so all inbound mail to any address will hit MailReceiverServlet. We only want to our app to send a copy of Stallman's book when one of the "to" addresses on the inbound email is stallmanbook@opencfsummit2011.appspotmail.com

After we check to see if the correct email address is one of the "to" addresses, we loop over the "from" addresses and use our old friend CFMAIL to send an email with the PDF attachment to each "from" address:

<cfloop index="i" from="1" to="#ArrayLen(request.senders)#">
    <cfmail from="stallmanbook@opencfsummit2011.appspotmail.com" 
                to="#request.senders[i]#" 
                subject="Stallman Book!" 
                mimeattach="#ExpandPath('/pdfs/fsfs-ii-2.pdf')#">
Here is the Stallman book you asked for. Enjoy!
    </cfmail>
</cfloop>

Pretty slick, huh?

Now that we're well-versed in receiving mail, let's move on to XMPP. Luckily since many of the basic concepts are the same we'll be able to pick up the pace a bit.

Google App Engine XMPP Service

The GAE XMPP service works very similarly to the mail service, with a few more options specific to XMPP. If you're not familiar with XMPP, XMPP stands for "Extensible Messaging and Presence Protocol" and while it can be used in a variety of ways, the usage people are most familiar with is for instant messaging. If you use Google Talk, you use XMPP. Also you may be running something like your own Jabber or OpenFire server in your organizaion for non-public IM, and these are based on XMPP.

As a quick reminder, when we configured inbound services for XMPP on our app we enabled:

• xmpp_message: allows the app to receive XMPP messages
• xmpp_presence: allows other XMPP users to see your applicaton's IM user presence (online, offline, busy, etc.)
• xmpp_subscribe: allows other XMPP users to "friend" your application's IM user as well as exchange message and status data

As with inbound mail, in order to get this running on your app all you need to do is specify XMPP inbound services in appengine-web.xml. With that in place once you deploy your app it is XMPP enabled.

Also as with inbound mail, your application gets its own XMPP IDs automatically. So once you enable XMPP your application can interact with the XMPP world using one of these two ID formats:

• YOUR_APP_ID@appspot.com
• ANYTHING@YOUR_APP_ID.appspotchat.com

(And yes, the domain is different between these two ID formats. The first is appspot.com, the second is appspotchat.com.)

Yet again as with inbound mail (I hope you're sensing a pattern), when an XMPP message hits your application it generates an HTTP POST to /_ah/xmpp/message/chat so all you have to do is write whatever code you want to handle incoming XMPP messages.

The XmppReceiverServlet

As discussed above it isn't really possible (yet) to have a POST to /_ah/xmpp/message/chat hit CFML directly, so we need a servlet that will handle POSTS to this URL pattern. The code for XmppReceiverServlet in the sample app projects is in the src directory, in the org.opencfsummit package.

First we declare some static final service type stuff in our servlet starting on line 21:

private static final Logger log = Logger.getLogger(XmppReceiverServlet.class.getName());
private static final XMPPService xmppService = XMPPServiceFactory.getXMPPService();

This gives us a logger so we can log incoming messages, as well as the service we'll be using to parse the inbound XMPP message. Side note: logging on GAE is a bit quirky so make sure and check my post about configuring logging on GAE if you run into issues.

In the doPost method in XmppReceiverServlet we'll parse the message, do some logging, parrot the message back to the sender (just to show we can send XMPP), and then we'll again forward the request on to a CFML page to do some additional stuff.

Message message = xmppService.parseMessage(request);
JID jid = message.getFromJid();
String body = message.getBody();
log.info(jid.getId() + " --> opencfsummit2011: " + body);

message = new MessageBuilder().withRecipientJids(jid).withBody(body).build();
xmppService.sendMessage(message);

request.setAttribute("toJID", "opencfsummit2011@appspot.com");
request.setAttribute("fromJID", jid.getId());
request.setAttribute("messageType", message.getMessageType().toString());
request.setAttribute("body", body);

getServletContext().getRequestDispatcher("/xmppdump.cfm").forward(request, response);

That's a bit of a mouthful so let's walk through this.

First we use the xmppService to parse the XMPP message out of the request object. This gives us an object of type com.google.appengine.api.xmpp.Message and represents the incoming XMPP message. Next, we get the user ID (JID) of the XMPP user who sent the message as an object of type com.google.appengine.api.xmpp.JID, and get the body of the message as a string. We then log this info which I was simply using for debugging purposes as I was working on the sample app.

Next, we build a new XMPP message using the MessageBuilder, and use the xmppService to send the user's message back to them. Obviously this isn't a great real-world example but getting the message you send bounced back to you is a nice way to make sure things are working properly.

After that, we put all the relevant information from the XMPP message into the request object so we'll have it available in the request scope on the CFML side. We then forward the request to xmppdump.cfm.

As with any servlet we need to add this to our web.xml and set up a servlet mapping so this servlet will respond to POSTs to /_ah/xmpp/message/chat:

<servlet>
    <servlet-name>xmppReceiver</servlet-name>
    <servlet-class>org.opencfsummit.XmppReceiverServlet</servlet-class>
</servlet>
<servlet-mapping>
    <servlet-name>xmppReceiver</servlet-name>
    <url-pattern>/_ah/xmpp/message/chat/</url-pattern>
</servlet-mapping>

xmppdump.cfm

Once the request is forwarded to xmppdump.cfm (which is in the war directory of the project) we can do anything we want with this information on the CFML side. In this case I'm first emailing myself the contents of CFDUMP for debugging/testing purposes:

<cfsavecontent variable="foo">
    <cfdump />
</cfsavecontent>

<cfmail from="mpwoodward@gmail.com"
            to="matt@mattwoodward.com" 
            subject="foo" type="html">
#foo#
</cfmail>

One thing I didn't mention earlier about sending mail from your GAE app is that the from address must be either:

• An administrator/developer on the application, or
• The address of the user who's currently signed in with their Google Account, or
• A valid "receiving" email address for the application (e.g. whatever@myappid.appspotmail.com)

Next, I decided just for fun I'd leverage CFHTTP to create a bit of an XMPP-to-SMS service. Any IMs that come into the application are turned into an SMS and sent to a (in this case hard-coded) mobile number using Twilio.

<cfhttp result="smsTestSend" 
        method="post" 
        url="https://MY_TWILIO_URL" 
        username="bar" 
        password="baz">
    <cfhttpparam type="formfield" name="From" value="5551234567" />
    <cfhttpparam type="formfield" name="To" value="5559876543" />
    <cfhttpparam type="formfield" name="Body" value="#request.body#" />
</cfhttp>

As you can see Twilio just takes an HTTP POST and turns it into an SMS message, and here since I have the body of the inbound XMPP message available as request.body, that's what becomes the body of the SMS message.

Yet another example that's of dubious use in the real world, but I hope it illustrates the points.

Try It Out!

This sample is live as well. I removed the part that sends text messages but you can log into Google Talk and add opencfsummit2011@appspot.com as a buddy, and any IMs you send will be bounced back to you. OK, enough testing. Stop talking to yourself. ;-) As a next step I was going to try to implement an Eliza-type bot but as with most things these days just didn't yet have the time.

Conclusion

In this post you learned how to leverage two more services available to you in the amazing Google App Engine platform, specifically the inbound mail and XMPP services. They're extremely easy to set up, incredibly powerful, and with just a bit of servlet code in the middle this all plays very nicely with Open BlueDragon for GAE.

If you have comments or questions fire away, and if there are specific aspects of GAE you'd like to know more about (I have some additional posts in mind) please let me know. Thanks!

In my previous two posts in this series I offered up my introduction to Google App Engine and covered using the Google Datastore from CFML. In this post I'm going to cover using the Virtual File System (VFS) in Open BlueDragon for Google App Engine (GAE), and talk a bit about the GAE Image Service and how to leverage it in your CFML applications.

Uploading Files in Google App Engine Applications

One of the limitations in GAE is that your applications don't have the ability to write to the file system. Not only was this a bit of a challenge to get CFML applications running at all on GAE since on-the-fly compiling and writing to the file system is at the heart of how CFML engines tend to work, this also prevents you from doing things like allowing your users to upload files to your application.

Necessity is the mother of invention, as the saying goes, and this case was no exception. To get around the inability to write to the file system, Vince Bonfanti created the virtual file system (VFS) project for GAE, which is an Apache Commons VFS plugin that creates a writeable filesystem on GAE by leveraging GAE's datastore and memcache APIs. Note that the GAE VFS project is independent of OpenBD so if you are developing GAE apps in Java you can leverage the VFS as well.

In my previous post I went over pieces of the sample application I created for OpenCF Summit 2011, and in this post we'll dig into one aspect of that application in detail so you'll want to download the sample app (.tar.gz file, 15.3 MB) if you haven't already. Unzip this file and import as a new project in Eclipse and you'll be able to follow along as well as run the app from inside Eclipse.

Uploading Photos in the Sample Application

With the sample project running, pull up http://localhost:8888/photos.cfm in your browser.

Now in the sample application code, let's take a look at the photos.cfm file in the war directory. Starting on line 60 you'll see the photo upload form:

<form name="photoForm"
        action="_uploadPhoto.cfm"
        method="post"
        enctype="multipart/form-data">
    Photo: <input type="file" name="photo" /><br />
    <input type="submit" value="Upload" />
</form>

This is all completely standard stuff, and as you can see the form posts to _uploadPhoto.cfm. The first couple of chunks of _uploadPhoto.cfm look like this:

<cfif not DirectoryExists(ExpandPath('photos'))>
    <cfdirectory action="create" directory="photos" />
</cfif>

<cffile action="upload"
        destination="photos"
        filefield="photo" />

First we check to see if the photos directory exists and if it doesn't, we use CFDIRECTORY to create it. Then we use CFFILE to upload the file.

At this point you're probably scratching your head wondering what's different from how you'd normally do things. That's OK; we'll get to the VFS magic in a bit. Just remember that without the VFS you wouldn't be able to write to the GAE filesystem so you should be impressed that nothing looks different so far.

We'll go over the rest of the code in _uploadPhoto.cfm in a bit, but go ahead and upload a photo on http://localhost:8888/photos.cfm so you have at least one photo in the applicaton. Once you upload a photo it just outputs all photos on that same screen so you should see something like this:

Where Do Files Get Stored?

Now that we have a photo uploaded into the application, let's take a look at where these files actually get stored and how to configure things so the VFS works in our CFML application running on GAE.

As mentioned above, the VFS uses the Google Datastore and Memcache APIs, so let's open up our local Datastore browser and see what's going on there. In your browser go to http://localhost:8888/_ah/admin and click on Datastore Viewer on the left.

Next select GaeFileObject as the Entity Kind in the Datastore viewer and click List Entities.

By default the VFS isn't configured to allow you to upload to arbitrary directories, so let's take a look at how to enable this in our application.

Configuring the VFS for File System Access

To enable the VFS for file uploads and other file system interaction, you need to add a servlet-mapping in your application's web.xml file. This is how you tell your application that when certain URL patterns are used in your application that it's going to be interacting with the VFS.

In Eclipse open up war/WEB-INF/web.xml and scroll down to line 87. You should see a comment block about the VFS:

<!--
    Configuring and mapping the GaeVfsServlet allows the Servlet/JSP engine (not the CFML engine) to
    "see" and serve application-generated static content (generated images, PDFs, uploaded files, etc...) that
    resides in a virtual file system location.
    See http://code.google.com/p/gaevfs
-->

By default the VFS servlet is commented out, so if you want to use the VFS in your applications you'll need to uncomment the servlet block for the VFS, which in the sample application is lines 92 - 103 in web.xml

The next step is to add virtual directories as servlet mappings to web.xml so that the application knows to use the VFS servlet for specific URL patterns. On line 104 in web.xml you see the following:

<servlet-mapping>
    <servlet-name>gaevfs</servlet-name>
    <url-pattern>/photos/*</url-pattern>
</servlet-mapping>

In case you were wondering how we could reference a /photos director in our CFML code when that directory doesn't actually exist on the file system, this is how. When a pattern of /photos/* is matched that work is handed off to the VFS servlet, which does its magic behind the scenes to allow you to use CFML tags such as CFFILE, CFDIRECTORY, CFLOG, and others without actually having write access to the file system.

Take another look at photos.cfm, and on line 48 you'll see a CFDIRECTORY call:

<cfdirectory action="list" directory="photos" name="photos" />

<cfif photos.RecordCount gt 0>
    <h2>Smile! You're on OpenBD on GAE!</h2>
    <cfloop query="photos">
        <img src="photos/#photos.name#" /><br />
    </cfloop>
<cfelse>
    <p>No photos. Why don't you upload one?</p>
</cfif>

As you can see from that code CFDIRECTORY works exactly as you'd expect, returning a query object containing the directory listing from the VFS directory "photos." If any photo files are found, they are then displayed using an image tag which also references the virtual photos directory.

Image Manipulation With the GAE Image Service

If you've perused the GAE documentation at all you'll see that one of the many benefits of running your applciations on GAE is the amazing array of services avaialble on the GAE platform, one of which is an Image Service. (I'll cover the mail and XMPP services in my next post.)

One of the interesting things to note about OpenBD for GAE is that it leverages many of these underlying services without you having to worry about it. For example, another restriction on GAE is that your application cannot make HTTP calls other than through the GAE URL Fetch API. What this means is that the CFHTTP tag on OpenBD for GAE uses the URL Fetch service under the hood without you having to do anything differently in your CFML code.

Let's go a bit further in the _uploadPhoto.cfm file in the sample application. Since we don't have any idea what size of image will be uploaded to the application, I decided to arbitrarily resize all images coming in to 300 x 200 pixels so they fit nicely on the page. Obviously if this were a real-world application we'd do some calculations so we could retain the correct aspect ratio for uploaded images and not resize at all unless they went beyond our desired boundaries, but for the purposes of the sample application I wanted to keep things simple.

After the file is uploaded using the CFFILE tag, the file is immediately read back from the VFS as binary data on line 9, which is necessary in order to manipulate the image using the GAE image service.

<cffile action="readbinary"
        file="#CFFILE.ServerFile#"
        variable="imgOrigData" />

In the next block of code we leverage the GAE image service to resize the image to 300 x 200 pixels:

<cfscript>
    isf = CreateObject("java", "com.google.appengine.api.images.ImageServiceFactory");

    imgSvc = isf.getImageService();
    imgOrig = isf.makeImage(imgOrigData);

    resize = isf.makeResize(300,200);

    imgNew = imgSvc.applyTransform(resize, imgOrig);

    imgNewData = imgNew.getImageData();
</cfscript>

At this point we have the resized image in the variable imgNewData, so the final step is to write that to disk using CFFILE, which of course is again leveraging the VFS:

<cffile action="write"
        file="#CFFILE.ServerFile#"
        output="#imgNewData#" />

We then use a CFLOCATION tag to send the user page to the photos.cfm page, which displays all the images in the /photos VFS directory.

That's all for this post! I hope it provides some good information about the GAE VFS that's included with OpenBD for GAE as well as giving a bit of an introduction to the services avaiable on GAE using the Image Service as an example.

As always if you have questions, corrections, etc. please post a comment. Next up is a post on the mail and XMPP services which I'll have ready soon.

At OpenCF Summit 2011 we were very lucky to have Chris Schalk from Google come present on Google App Engine. If you're not familiar with Google App Engine (GAE) you should be! It's an absolutely fantastic application platform as a service offering from Google with great functionality, very slick features, and incredibly generous quotas for free application hosting. And if you need to go beyond these quotas, you simply configure billing and pay nominal fees for what you use over the free quotas.

GAE lets you deploy Python and Java applications, but one of the most interesting things going on with Java these days is the numerous different languages that run on the JVM. The Java platform being available on GAE opens up some very cool options.

At OpenCF Summit I followed Chris's presentation with one specific to Open BlueDragon on Google App Engine (GAE). OpenBD is a Java-based CFML runtime engine that allows you to deploy CFML applications to any standard servlet container, and also allows you to deploy your CFML applications to GAE. This is a great option for CFML developers since it's a quick and easy--not to mention free for many apps!--way to get your CFML applications online without having to worry about setting up a server yourself or getting a hosting account. Not to mention that if you deploy your CFML apps on GAE you get the benefits of running on Google's infrastructure and have on-demand scalability for your apps.

If you're not familiar with CFML, it's an incredibly powerful dynamic scripting language and framework that runs on the JVM. Think of it as a Java Tag Library on steroids. Even if you choose to build the backend of your applications in Java, CFML is a fantastic view layer language that's a great alternative to JSP, it interacts seamlessly with Java code, and it makes a lot of things that are quite verbose in Java extremely quick and easy. Of course CFML is a full-fledged language as well so you can build entire applications in it quickly.

The tools available for GAE make it very easy to work with. If you use Eclipse, a great option is to grab the GAE plugin for Eclipse. This gives you the entire GAE environment that will run right inside Eclipse and let you develop and test locally. Then when you're ready to deploy to GAE, it's a right-click away.

We also have great tools for OpenBD. In addition to being able to grab the GAE edition of OpenBD and drop that into an Eclipse project, you can use the new OpenBD Desktop. This is a desktop application that runs on GNU/Linux, Mac, and Windows, and lets you set up a local development server in seconds. Once development is complete you can then deploy your CFML application to a standard JEE WAR, or you can deploy straight to GAE from OpenBD Desktop.

In this post I'm going to cover getting up and running with GAE in Eclipse, and in my next post I'll go over the demo application I built for my presentation at OpenCF Summit, and in that post I'll highlight some of the cool features not only of OpenBD for GAE but GAE itself.

I'm going to cover how to set things up in Eclipse in this blog post, but I'll have another how to and screencast covering OpenBD Desktop soon.

Installing the Google App Engine Plugin for Eclipse

I'm going to assume my audience is mostly Java or CFML developers who are already somewhat familiar with Eclipse, but if you need assistance with this piece of things please leave me a comment and I will be happy to help.

The GAE plugin for Eclipse installs in the same way as any other add-on for Eclipse. You simply open Eclipse, go to Help -> Install New Software and paste in the appropriate update site URL for your version of Eclipse. This will download everything you need to work with GAE from within Eclipse, including the GAE for Java SDK.

Once the plugin installs and you restart Eclipse, you'll notice new Google icons in your Eclipse toolbar:

As well as a new Google right-click menu:

To create a new GAE project, you go to File -> New -> Project, and in the Google folder choose Web Application Project.

You can then run your GAE application from within Eclipse by right-clicking the project and choosing Run As -> Web Application. Running in debug mode and all the other Java functionality with which you may be familiar is of course also available.

Note that because GAE is a platform as a service offering, the entirety of the Java world isn't necessarily available to you. If you're curious what is and isn't available check the JRE Class Whitelist in the GAE docs.

Installing Open BlueDragon for GAE

With a new GAE project created in Eclipse, installing OpenBD for GAE is as easy as downloading a zip file, unzipping, and copying files into your GAE project's war directory.

When you download and unzip OpenBD for GAE you'll see these contents:

To add OpenBD to your GAE project, go into the war directory in the unzipped OpenBD for GAE directory, and copy all the files in the OpenBD war directory into the war directory in your Eclipse GAE project. Note that you will overwrite any files with the same name in the Eclipse project, which is what you want to do.

What you do not want to do is delete all of the existing files in the war directory in the Eclipse project and have only the OpenBD GAE war files in the Ecliipse project. To put it another way, you are merging the OpenBD for GAE files with the files that are already in the Eclipse project, and any files with the same name will be replaced by the OpenBD for GAE files.

With these files in place, right-click on the project in Eclipse and choose Run As -> Web Application. You should see something similar to this in the Eclipse console:

If everything started up successfully you can then navigate to http://localhost:8888 in a browser and see this:

Working with CFML Code

Working with CFML code in an OpenBD for GAE project is no different that typical CFML development. The only real thing you need to be aware of is that your CFML code must be placed in the war directory in your Eclipse project. This is the root of your application. (Note that if you haven't worked with CFML or perhaps use a CFML editor other than Eclipse, you'll want to install CFEclipse, which is a great open source CFML plugin for Eclipse.)

Let's add a CFML file to our project so you can get a feel for working with CFML code in the context of an OpenBD for GAE project.

In your Eclipse project right-click the war directory and choose New -> File and name the file test.cfm. In this newly created file, add the following code:

 <cfset name = "Matt" />  <cfoutput>Hello #name#! Today is #DateFormat(Now())#.</cfoutput> 

Save the file, and then go to http://localhost:8888/test.cfm in your browser. You should see this:

Up until recently there have been some differences in supported syntax and functionality between "regular" OpenBD and OpenBD for GAE, but as of the next release of OpenBD the regular Java edition and the GAE edition will have the same exact functionality available, other than where specific functionality is not allowed on the GAE platform. The current nightly builds of OpenBD are based on this new unified codebase between the two editions of OpenBD.

Most CFML code will work fine on OpenBD for GAE. For example, the Mach-II framework as well as ColdSpring both work perfectly, and these frameworks are being used for the open source Enlist application that we started developing during the hackfest at OpenCF Summit.

What's Next?

Probably the major thing developers will run into immediately when building apps for GAE is that a traditional relational database is not available other than through Google App Engine for Business. If you aren't on the GAE for Business platform, you'll be using the Google Datastore, which is a high-performance, highly scalable key-value ("NoSQL") datastore that can be accessed via JDO or JPA, and also in CFML via GAE-specific functionality built into OpenBD for GAE.

I'll cover the Google Datastore as well as some of the amazing features of the GAE platform (including receiving mail and receiving/sending XMPP messages) in my next post.

A co-worker contacted me today asking how he might go about solving what turned out to be a rather interesting issue.

From a CFML application (running on Open BlueDragon) he's calling a .NET web service and getting an array of objects back. By the time all the SOAP magic happens the objects in the array on the CFML side are Java objects.

What he wanted to do next was loop over this array of Java objects and, for each object in the array, call every getXXX() method contained in the object. But the application is getting numerous different types of objects back, some of which have a large number of get methods in them, and he didn't want to have to hard-code a method call for each one. In addition, the get methods may change from time to time and while we're supposed to be notified when changes occur, we didn't want to rely on that.

So consider the following pseudocode:

<cfinvoke webservice="url to webservice"
    method="methodname"
    returnvariable="myJavaObjects" />

So at this point the variable myJavaObjects is an array of homogeneous Java objects.

Next, we want to loop over that array and for each Java object, call all of its get methods.

My first thought was that this is one of those rare cases where Evaluate() might be justified. But I also thought there had to be a better way, so perhaps against my co-worker's will we spent about an hour hammering through some experiments. I'll spare you the various things we tried and cut to the chase of the final solution.

One thing I learned while working through this is that CFINVOKE works on Java objects. Who knew? OK, maybe you knew, but I hadn't ever had cause to try it before so I didn't know.

So step one is once we get the array of Java objects back from the web service, since we know they're homogeneous objects, we'll just use some Java reflection magic on the first one to create an array of the method names beginning with get:

 <!--- this returns an array of java.lang.reflect.Method objects --->
 <cfset methods = myJavaObjects[1].getClass().getMethods() />
 <!--- now we'll create an array of method names starting with get --->
 <cfset methodNames = [] />
 <cfloop array="#methods#" index="method">
     <cfif Left(method.getName(), 3) == "get">
         <cfset ArrayAppend(methodNames, method.getName()) />
     </cfif>
 </cfloop>
 

So now we have an array of strings that are the method names from the Java object that start with get. In the actual application we're actually omitting some of the methods starting with get because they're not relevant (e.g. getClass(), getSerializer(), etc.) so I'm just keeping it simple for the purposes of illustration.

The next step is to loop over the array of Java objects, and on each loop iteration, call each get method and for demo purposes simply output the results. Here's where we use CFINVOKE to call methods dynamically on the Java objects:

 <cfloop array="#myJavaObjects#" index="javaObject">
     <cfloop array="#methodNames#" index="methodName">
         <cfinvoke component="#javaObject#" method="#methodName#" returnvariable="foo" />
         <cfoutput>#foo#<br /></cfoutput>
     </cfloop>
 </cfloop>
 

And with that, we're getting an array of Java objects back from a .NET web service (with the .NET object to Java object translation being handled transparently by the web service engine of course), we're using a bit of Java reflection to get a list of the getters from the Java object, and we're then looping over the array of Java objects and calling the get methods on each Java object.

As an aside, during experimentation we went down the path of using Java reflection directly, but that got pretty messy and didn't seem to offer any benefit over doing things at a higher level in CFML. Interestingly, while we were messing with some things we had an error generated from CFINVOKE reminding me that under the hood, CFINVOKE is doing all the Java reflection nastiness for you.

Not sure how handy a tip that will be for others but I wanted to blog it while it was still fresh in my mind. There's probably several other ways to solve this problem so if others have approached this differently I'd love to hear about it.