Monday, March 7, 2011

Using the Virtual File System and Image Service in Google App Engine with CFML

In my previous two posts in this series I offered up my introduction to Google App Engine and covered using the Google Datastore from CFML. In this post I'm going to cover using the Virtual File System (VFS) in Open BlueDragon for Google App Engine (GAE), and talk a bit about the GAE Image Service and how to leverage it in your CFML applications.

Uploading Files in Google App Engine Applications

One of the limitations in GAE is that your applications don't have the ability to write to the file system. Not only was this a bit of a challenge to get CFML applications running at all on GAE since on-the-fly compiling and writing to the file system is at the heart of how CFML engines tend to work, this also prevents you from doing things like allowing your users to upload files to your application.

Necessity is the mother of invention, as the saying goes, and this case was no exception. To get around the inability to write to the file system, Vince Bonfanti created the virtual file system (VFS) project for GAE, which is an Apache Commons VFS plugin that creates a writeable filesystem on GAE by leveraging GAE's datastore and memcache APIs. Note that the GAE VFS project is independent of OpenBD so if you are developing GAE apps in Java you can leverage the VFS as well.

In my previous post I went over pieces of the sample application I created for OpenCF Summit 2011, and in this post we'll dig into one aspect of that application in detail so you'll want to download the sample app (.tar.gz file, 15.3 MB) if you haven't already. Unzip this file and import as a new project in Eclipse and you'll be able to follow along as well as run the app from inside Eclipse.

Uploading Photos in the Sample Application

With the sample project running, pull up http://localhost:8888/photos.cfm in your browser.


Now in the sample application code, let's take a look at the photos.cfm file in the war directory. Starting on line 60 you'll see the photo upload form:

<form name="photoForm"
    Photo: <input type="file" name="photo" /><br />
    <input type="submit" value="Upload" />

This is all completely standard stuff, and as you can see the form posts to _uploadPhoto.cfm. The first couple of chunks of _uploadPhoto.cfm look like this:

<cfif not DirectoryExists(ExpandPath('photos'))>
    <cfdirectory action="create" directory="photos" />

<cffile action="upload"
        filefield="photo" />

First we check to see if the photos directory exists and if it doesn't, we use CFDIRECTORY to create it. Then we use CFFILE to upload the file.

At this point you're probably scratching your head wondering what's different from how you'd normally do things. That's OK; we'll get to the VFS magic in a bit. Just remember that without the VFS you wouldn't be able to write to the GAE filesystem so you should be impressed that nothing looks different so far.

We'll go over the rest of the code in _uploadPhoto.cfm in a bit, but go ahead and upload a photo on http://localhost:8888/photos.cfm so you have at least one photo in the applicaton. Once you upload a photo it just outputs all photos on that same screen so you should see something like this:




Where Do Files Get Stored?

Now that we have a photo uploaded into the application, let's take a look at where these files actually get stored and how to configure things so the VFS works in our CFML application running on GAE.

As mentioned above, the VFS uses the Google Datastore and Memcache APIs, so let's open up our local Datastore browser and see what's going on there. In your browser go to http://localhost:8888/_ah/admin and click on Datastore Viewer on the left.


You'll notice that in the Entity Kind drop-down there is an entry for /photos/your_file_name.jpg which indicates the photo is being stored in the Datastore, even though in the CFML code we're doing what we'd normally do to write an uploaded file to the filesystem. If you click on "List Entities" for your photo's Entity Kind you'll notice that there are four records (at least in my case) for the file.


Why there are four records for this one file frankly I don't know, and it's not all that important to this discussion. At least that's my excuse for not knowing at this point. ;-) Just realize that when you upload a file this is what happens in the Datastore as the GAE VFS handles the writing of the file.

Next select GaeFileObject as the Entity Kind in the Datastore viewer and click List Entities.


Here you'll see listed some of the directories that OpenBD uses to do its work behind the scenes, but you'll also notice that there's an entity with an ID/Name of /photos that has a filetype of "folder," and there's also an entity for the photo itself, and for that entity you'll see that it includes the content size of the file and it has a filetype of "file."

By default the VFS isn't configured to allow you to upload to arbitrary directories, so let's take a look at how to enable this in our application.

Configuring the VFS for File System Access

To enable the VFS for file uploads and other file system interaction, you need to add a servlet-mapping in your application's web.xml file. This is how you tell your application that when certain URL patterns are used in your application that it's going to be interacting with the VFS.

In Eclipse open up war/WEB-INF/web.xml and scroll down to line 87. You should see a comment block about the VFS:

    Configuring and mapping the GaeVfsServlet allows the Servlet/JSP engine (not the CFML engine) to
    "see" and serve application-generated static content (generated images, PDFs, uploaded files, etc...) that
    resides in a virtual file system location.

By default the VFS servlet is commented out, so if you want to use the VFS in your applications you'll need to uncomment the servlet block for the VFS, which in the sample application is lines 92 - 103 in web.xml

The next step is to add virtual directories as servlet mappings to web.xml so that the application knows to use the VFS servlet for specific URL patterns. On line 104 in web.xml you see the following:


In case you were wondering how we could reference a /photos director in our CFML code when that directory doesn't actually exist on the file system, this is how. When a pattern of /photos/* is matched that work is handed off to the VFS servlet, which does its magic behind the scenes to allow you to use CFML tags such as CFFILE, CFDIRECTORY, CFLOG, and others without actually having write access to the file system.

Take another look at photos.cfm, and on line 48 you'll see a CFDIRECTORY call:

<cfdirectory action="list" directory="photos" name="photos" />

<cfif photos.RecordCount gt 0>
    <h2>Smile! You're on OpenBD on GAE!</h2>
    <cfloop query="photos">
        <img src="photos/" /><br />
    <p>No photos. Why don't you upload one?</p>

As you can see from that code CFDIRECTORY works exactly as you'd expect, returning a query object containing the directory listing from the VFS directory "photos." If any photo files are found, they are then displayed using an image tag which also references the virtual photos directory.

Image Manipulation With the GAE Image Service

If you've perused the GAE documentation at all you'll see that one of the many benefits of running your applciations on GAE is the amazing array of services avaialble on the GAE platform, one of which is an Image Service. (I'll cover the mail and XMPP services in my next post.)

One of the interesting things to note about OpenBD for GAE is that it leverages many of these underlying services without you having to worry about it. For example, another restriction on GAE is that your application cannot make HTTP calls other than through the GAE URL Fetch API. What this means is that the CFHTTP tag on OpenBD for GAE uses the URL Fetch service under the hood without you having to do anything differently in your CFML code.

Let's go a bit further in the _uploadPhoto.cfm file in the sample application. Since we don't have any idea what size of image will be uploaded to the application, I decided to arbitrarily resize all images coming in to 300 x 200 pixels so they fit nicely on the page. Obviously if this were a real-world application we'd do some calculations so we could retain the correct aspect ratio for uploaded images and not resize at all unless they went beyond our desired boundaries, but for the purposes of the sample application I wanted to keep things simple.

After the file is uploaded using the CFFILE tag, the file is immediately read back from the VFS as binary data on line 9, which is necessary in order to manipulate the image using the GAE image service.

<cffile action="readbinary"
        variable="imgOrigData" />

In the next block of code we leverage the GAE image service to resize the image to 300 x 200 pixels:

    isf = CreateObject("java", "");

    imgSvc = isf.getImageService();
    imgOrig = isf.makeImage(imgOrigData);

    resize = isf.makeResize(300,200);

    imgNew = imgSvc.applyTransform(resize, imgOrig);

    imgNewData = imgNew.getImageData();

At this point we have the resized image in the variable imgNewData, so the final step is to write that to disk using CFFILE, which of course is again leveraging the VFS:

<cffile action="write"
        output="#imgNewData#" />

We then use a CFLOCATION tag to send the user page to the photos.cfm page, which displays all the images in the /photos VFS directory.

That's all for this post! I hope it provides some good information about the GAE VFS that's included with OpenBD for GAE as well as giving a bit of an introduction to the services avaiable on GAE using the Image Service as an example.

As always if you have questions, corrections, etc. please post a comment. Next up is a post on the mail and XMPP services which I'll have ready soon.

1 comment:

Jason Blum said...

Nice. cffile action="append" doesn't seem to work - guess I have to check and create the file if it doesn't already exist.Another awesome post Matt - thanks for taking the time to do it.I think the time may be near for an OpenBDWACK?