Saturday, July 23, 2016

Running a Django Application on Windows Server 2012 with IIS

This is a first for me since under normal circumstances we run all our Django applications on Linux with Nginx, but we're in the process of developing an application for another department and due to the requirements around this project, we'll be handing the code off to them to deploy. They don't have any experience with Linux or web servers other than IIS, so I recently took up the challenge of figuring out how to run Django applications on Windows Server 2012 with IIS.

Based on the dated or complete lack of information around this I'm assuming it's not something that's very common in the wild, so I thought I'd share what I came up with in case others need to do this.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Assumptions and Caveats

  1. The operating system is Windows Server 2012 R2, 64-bit. If another variant of the operating system is being used, these instructions may not work properly.
  2. All of the software versions referenced in this document are current as of the date of this blog post.
  3. All actions are performed as the Administrator user, or another user with Administrator rights.
  4. Depending on the audience, some of these steps may seem unnecessarily pedantic. They were originally written to be as foolproof as possible and to make these steps as thorough and complete as they can be. I decided not to simplify or abbreviate anything for this blog post.
  5. The version of Django that is current as of the time of this writing is 1.9.8. If any of the Django-related configuration settings change in the future, the settings outlined in this document may no longer be valid. For example, the details of the call to the Django WSGI handler are specific to this version of Django. They have been different in prior versions of Django, and may need to be updated in the future to reflect changes in Django.
  6. We'll be creating a sample application called foo directly on the server. In real-world circumstances you'll be deploying a completed application to the server, but I thought this was a good way to illustrate the steps and how all the pieces fit together.
  7. If you install things in different locations than outlined here, or put your application in a different location than outlined here, obviously you'll have to adjust paths in configuration settings accordingly.

Prerequisites for Running Django Applications on IIS

If you want to target Windows Server and IIS as the production environment for your Django application, there aren't really any issues of concern other than including the wfastcgi Python package (https://pypi.python.org/pypi/wfastcgi) in your application.

wfastcgi is maintained by Microsoft as part of Python Tools for Visual Studio (https://www.visualstudio.com/en-us/features/python-vs.aspx), and provides a file that serves as the entry point of the IIS handler for WSGI applications in Python. It's similar in purpose to a tool like Gunicorn (http://gunicorn.org), with the end result being that requests that come into IIS are handed off to the Python application for processing.

wfastcgi can be installed with pip:
pip install wfastcgi

You’ll also want to add wfastcgi to your Django application's requirements.txt file to make sure it gets installed as part of the deployment of your application.

After installing wfastcgi the file wfastcgi.py will be in your Python environment's site-packages directory. We’ll point to this file later when we're configuring the IIS handler for Django.

Install Python

  1. Download the latest Windows x86-64 executable installer from https://python.org
  2. Double-click the Python installer. If you get a security warning at this point, click "Run" to continue.
  3. In the Python Setup dialog that appears, click "Customize Installation"
  4. On the Optional Features step, leave the default settings and click "Next"
  5. On the Advanced Options step, make the following changes:
    1. Check the "Install for all users" box (note that this will also check the "Precompile standard library" box)
    2. Check "Add Python to environment variables"
    3. In the input box below "Customize install location" change the value to the following:
      C:\Python35
  6. Click "Install"
  7. When the installation is complete, click "Close"

Confirm the Python Installation

  1. Open a new Command Prompt. Note that if you already had a command prompt open you will need to close and reopen it, as the Python installation process added new environment variables and any open command prompts will not have the new environment variables available.
  2. Type python and press Enter. This should open a Python interpreter:
    1. If instead of seeing the Python interpreter you see the error "'python' is not recognized as an internal or external command ..." this likely means the "Add Python to environment variables" box was not checked on the Advanced Options step of the installation process. If this occurs you can uninstall and reinstall Python and make sure to check the "Add Python to environment variables" checkbox, or add the following to the System environment "Path" variable:
      C:\Python35;C:\Python35\Scripts
  3. Exit the Python interpreter by typing exit() and pressing Enter.
  4. Close the Command Prompt

Install IIS

Even if you already have IIS installed, you may need to go through these steps to install the CGI feature if it wasn't installed as part of the original IIS installation.
  1. Open the Control Panel
  2. In the search box in the top right, type "windows features" (without the quotes)
  3. In the search results under "Programs and Features" click "Turn Windows features on or off." This launches the Add Roles and Features Wizard.
  4. On the "Before you begin" step, click "Next"
  5. On the "Select installation type" leave the "Role-based or feature-based installation" radio button selected and click "Next"
  6. On the "Select destination server" step, leave the current server highlighted and click "Next"
  7. On the "Select server roles" step, scroll to the bottom of the list and check "Web Server (IIS)"
  8. In the "Add features that are required for Web Server (IIS)?" dialog that appears, leave the "Include management tools (if applicable)" checkbox checked and click "Add Features"
  9. On the "Select server roles" step, now that "Web Server (IIS)" is checked, click "Next"
  10. On the "Select features" step, leave the defaults and click "Next"
  11. On the "Web Server Role (IIS)" step, click "Next"
  12. On the "Select role services" step, scroll down to "Application Development," expand that section, and check the "CGI" box. This will also check the "Application Development" checkbox. With "CGI" checked, click "Next."
  13. On the "Confirmation" step, click "Install"
  14. Once the installation completes, click "Close"
  15. Close Server Manager
  16. Close Control Panel

Verify the IIS Installation

  1. Open a web browser on the server
  2. Enter http://localhost in the address bar and press Enter. You should see the default IIS page.
  3. If you don't see the default IIS page:
    1. Open Control Panel
    2. Type "services" in the search box
    3. Under "Administrative Tools" click "View local services"
    4. Scroll to the bottom of the list and ensure you see "World Wide Web Publishing Service" listed, and that the status is "Running"

Create and Configure a Python Virtual Environment for Your Application

Python virtual environments are used to isolate application-specific Python environments from the system-wide Python environment. Even if you will only be running one application on a server, it's good practice to create a virtual environment for your application to keep the virtual environment free from any unanticipated side effects of system updates.
  1. Open a Command Prompt
  2. Navigate to the root of the C: drive:
    cd C:\ [then press Enter]
  3. Create a new directory in which to store your virtual environments:
    mkdir virtualenvs [then press Enter]
  4. Navigate to the newly created virtualenvs directory:
    cd virtualenvs [then press Enter]
  5. Create a new virtual environment. We'll use the name foo for the purposes of illustration. Note that if you change this, you'll have to adjust the rest of the steps in this post accordingly.
    python -m venv foo [then press Enter]

Activate the Virtual Environment and Upgrade pip

With the virtual environment created, the next steps are to activate the virtual environment and update pip, Python's package installer.

First, we'll activate the virtual environment:
  1. Open a Command Prompt
  2. Navigate to the virtual environment's directory:
    cd C:\virtualenvs\foo [then press Enter]
  3. Activate the virtual environment:
    Scripts\activate.bat [then press Enter]
  4. At this point you will see (foo) at the beginning of the command prompt, indicating the virtual environment is active:
    1. Once a virtual environment is active, all Python-related commands such as python, pip, etc. will be executed against the active virtual environment. For example, with the virtual environment active if a package is installed with pip, that package will only be available in the virtual environment, not in the system's global Python environment.
Next, to ensure we don't run into installation or security issues with pip, let's upgrade it:
  1. In the same Command Prompt you opened above, with the virtual environment activated, run the following command:
    python -m pip install --upgrade pip [then press Enter]
    1. If an update for pip is available, it will be downloaded and installed

Install Django and Create a Sample Django Application

Typically, you'll be installing requirements from a requirements.txt file that is part of the codebase for your application, but for the purposes of this installation guide we'll install Django in our virtual environment, both to illustrate the steps involved as well as to ensure the virtual environment is working properly. Then we'll create a sample Django application that we'll use later to configure with IIS.
  1. In the same Command Prompt you opened above, with the virtual environment activated, install Django:
    pip install django [then press Enter]
  2. As outlined in the prerequisites section at the beginning of this document, you'll also need to install wfastcgi in order for IIS to be able to pass requests off to Django:
    pip install wfastcgi [then press Enter]
  3. Navigate to the root of the C: drive:
    cd C:\ [then press Enter]
  4. Create a new directory in which we'll put our application:
    mkdir apps [then press Enter]
  5. Navigate to the newly created apps directory:
    cd apps [then press Enter]
  6. Create a new Django project called foo:
    django-admin.py startproject foo [then press Enter]
  7. Navigate to the newly created foo project directory:
    cd foo [then press Enter]
  8. Start the Django development server:
    python manage.py runserver [then press Enter]
    1. You should see the following output:
  9. In a browser on the server, go to http://localhost:8000
    You should see the following:
  10. Go back to the Command Prompt and press Ctrl-C to stop the Django development server.

Configure IIS to Serve Django Applications

Thus far we've accomplished the following:
  1. Installed Python
  2. Installed IIS with CGI
  3. Created and activated a Python virtual environment
  4. Installed Django and wfastcgi in a Python virtual environment
  5. Created a new Django project
  6. Ran the Django project using the Django development server
With all the underlying infrastructure in place, the last step is to configure IIS to serve Django applications. The Django development server is not designed for production use, so using a production-quality web server such as IIS is mandatory for production applications.

As with previous steps you'll need to adjust accordingly for the actual application you want to deploy, but for the purposes of this documentation we'll configure IIS to serve the test Django application we created in the previous steps.

Configure FastCGI in IIS

  1. Open the IIS Manager by clicking the Windows button, clicking on Administrative Tools, and double-clicking Internet Information Services (IIS) Manager
  2. Click on the name of the server in the list on the left. If you see the following dialog box, check the box "Do not show this message" and click "No." (You can always get to this later if necessary by clicking "Get New Web Platform Components" under "Actions" on the right-hand side of IIS Manager.)
  3. Double-click the "FastCGI Settings" icon
  4. Under "Actions" on the right-hand side click "Add application ..."
  5. In the Add FastCGI Application dialog, in the "Full Path" box, type the path to the Python executable for the application's virtual environment:
    C:\virtualenvs\foo\Scripts\python.exe
  6. In the Arguments input box, type the path to the wfastcgi.py file in the application's virtual environment:
    C:\virtualenvs\foo\Lib\site-packages\wfastcgi.py
  7. At this point your settings in the Add FastCGI Application dialog should look like this:
  8. With the Add FastCGI Application dialog box still open, under the "General" section, click on the "Environment Variables" line, then click the gray "..." button that appears next to (Collection) on the right-hand side of the line. This opens the EnvironmentVariables Collection Editor dialog.
  9. In the EnvironmentVariables Collection Editor dialog, click "Add"
  10. In the "Name properties" section on the right, click the input box to the right of "Name," remove the "Name" text that is already in the input box, and enter DJANGO_SETTINGS_MODULE (note that this MUST be entered in ALL CAPS)
  11. Click the input box to the right of "Value" and enter foo.settings
  12. Click "Add" again and enter the following:
    Name: PYTHONPATH
    Value: C:\apps\foo
  13. Click "Add" again and enter the following:
    Name: WSGI_HANDLER
    Value: django.core.wsgi.get_wsgi_application()
  14. At this point you will have three environment variables:
    1. NOTE: All of these settings are CASE-SENSITIVE. They must be entered with exactly the case indicated here in order to work.
  15. Click "OK" to close the EnviromentVariables Collection Editor
  16. Click "OK" to close the Add FastCGI Application dialog

Create and Configure a New IIS Web Site

Next we need to create a new web site in IIS for the Django application, and add a Handler Mapping to the newly created web site so that requests coming to the IIS web site are passed off to the Django application for processing.
  1. Open IIS Manager
  2. On the left-hand side under Connections, expand the tree under the server name by clicking on the arrow to the left of the server name
  3. Right-click on the Sites folder and click "Add Website ..."
  4. For the site name enter foo
  5. For the physical path, type the following:
    C:\apps\foo
  6. For the purposes of this example configuration, change the Port to 81, since the Default site is running on port 80. For a real-world application you'll likely want to use name-based virtual hosting by adding bindings and run the site on port 80.
  7. You may leave the "Host name" blank. At this point the Add Website dialog should look like this:
  8. Click "OK"
Next, we'll add a FastCGI handler mapping to this site so the requests coming into the site will be processed by the Django application.
  1. In IIS Manager, expand the Sites folder on the left-hand side and click on the foo site
  2. On the right, double-click "Handler Mappings"
  3. On the right, under "Actions," click "Add Module Mapping"
  4. In the "Request path" box enter an asterisk: *
  5. Click the arrow on the right-hand side of the "Module" box and select "FastCgi Module"
    1. NOTE: Make sure to select FastCgi Module, NOT CgiModule
  6. In the "Executable" box, enter the following:
    C:\virtualenvs\foo\Scripts\python.exe|C:\virtualenvs\foo\Lib\site-packages\wfastcgi.py
    1. Note that the character after python.exe is a pipe (|), which is entered by pressing Shift-\ on your keyboard
  7. In the "Name" box, enter Django Handler (you can call this whatever you want; it's merely a friendly name for the module mapping)
  8. The Add Module Mapping dialog should now look like this:
  9. Click the "Request Restrictions" button and uncheck the "Invoke handler only if request is mapped to:" checkbox
  10. Click "OK" to close the Request Restrictions dialog
  11. Click "OK" to close the Add Module Mapping dialog
  12. When prompted "Do you want to create a FastCGI application for this executable?" click "No" since we created the application earlier.
    1. Note that you CAN have it create the FastCGI application for you from the module mapping, but the settings seem to be different and the end result isn't fully editable. I also detailed how to create the FastCGI application to be as thorough as possible with all the various pieces involved.
  13. You will now see the Django Handler listed in the Handler Mappings for the foo website:
At this point everything should be working, so verify by loading the application in a browser.
  1. Open a browser on the server
  2. Browse to http://localhost:81
  3. You should see the following:
This is the same result as when we ran the Django test server earlier, but now the requests are being handled by IIS.

Configure Django and IIS to Serve Static Files

Now we have things more or less working but we aren't quite done. If you browse to the Django admin page (http://localhost:81/admin), you'll see something like this:
The reason things look a little strange is because IIS doesn't know where things like the CSS and image files for the Django application are located. In the Django world these assets are referred to as "static files," and we need to perform a couple of additional steps for IIS to be able to serve these files. This isn't an issue during development since Django has a static file server built into it, but in production we'll want IIS to handle serving these files.

How Django handles static files is a bit of a topic in and of itself, so for further information please consult the Django documentation (https://docs.djangoproject.com/en/1.9/howto/static-files/). In this guide we'll focus on a specific set of steps to get things working without getting into too many of the behind-the-scenes technical details, the different philosophies on where static files can or should be placed, or the various and sundry ways in which this aspect of Django applications can be addressed.

Add a STATIC_ROOT Setting to the Django Settings File

First, your Django application's settings file needs to include a STATIC_ROOT setting, and this setting is not in Django applications by default. The STATIC_ROOT setting is used when Django's collectstatic management command is run to determine where to place the collected static files.

In our sample foo application, we'll tell Django to collect the static files in a static directory located in the same directory as the settings.py file. (You can ultimately put your static directory wherever you wish, but this is where we'll be putting it for the purposes of this example.)

To accomplish this, do the following:
  1. Using a plain text editor such as Notepad, or Idle if that was installed when you installed Python, open the file C:\apps\foo\foo\settings.py
  2. Scroll to the bottom of the file, or use the find feature of your text editor, and find the STATIC_URL setting
  3. Above the STATIC_URL setting, add the following setting:
    STATIC_ROOT = os.path.abspath(os.path.join(BASE_DIR, 'foo', 'static'))
    1. The placement of the STATIC_ROOT setting in the settings.py file doesn't matter, but putting it right next to the STATIC_URL setting is typical and keeps all the settings related to static files in one place.
    2. The STATIC_ROOT setting we're using as our example will put the static files in C:\apps\foo\foo\static
  4. Save the settings.py file

Run the collectstatic Management Command

Now that Django knows where to put the static files, we can run the collectstatic management command. This command takes all the static files for your project, including the static files used by the Django admin and other packages in your virtual environment, and puts them under one static directory, the location of which is the STATIC_ROOT setting we added above, so they can be served by the web server.
  1. Open a Command Prompt
  2. Activate the application's virtual environment:
    C:\virtualenvs\foo\Scripts\activate.bat [then press Enter]
  3. Navigate to the application's root directory:
    cd C:\apps\foo [then press Enter]
  4. Run Django's collectstatic management command:
    python manage.py collectstatic [then press Enter]
  5. Django will then tell you the directory into which it's going to copy the static files, and ask you to confirm since this process overwrites any existing files in the directory. Type yes at this prompt and press Enter.
  6. The static files will then be copied to the directory indicated by the STATIC_ROOT setting.

Add a static Virtual Directory to the IIS Web Site

Now we have all our static files in one place, but we still haven't told IIS where they're located. We'll accomplish this by adding a static virtual directory to the IIS web site for our application.

Note that the name of the virtual directory in IIS must match the value of the STATIC_URL setting in the Django application's settings.py file, absent the beginning and trailing slashes. For our sample application we're using a value of /static/ for the STATIC_URL setting, so the name of the virtual directory in IIS will be static.

Create a virtual directory in IIS as follows:
  1. Open IIS Manager
  2. On the left-hand side under "Connections," expand the server's tree
  3. Expand the "Sites" folder
  4. Expand the foo web site. At this point, things should look like this:
  5. Right-click the foo web site and click "Add Virtual Directory"
  6. In the Add Virtual Directory dialog, enter the following values:
    Alias: static
    Physical path: C:\apps\foo\foo\static
  7. Click "OK." You'll see the static virtual directory appear under the web site.

Configure Handler Mappings for the static Virtual Directory

The last step is to tell IIS which handler to use to serve files out of the static virtual directory. Since at this point our Django Handler is set to serve all files for our application, even with the static virtual directory in place the Django Handler will still be attempting to serve these files.

IIS already has a static file handler active in the web site, but it's down further in the list of handlers than the global Django Handler we configured, so to get IIS to serve the static files under the static virtual directory we'll move the static file handler to the top of the handler list on the static virtual directory.
  1. Open IIS Manager
  2. Expand the server tree
  3. Expand the foo web site
  4. Click on the static virtual directory
  5. Double-click the "Handler Mappings" icon. You'll see the following list of handler mappings:
  6. On the right-hand side under "Actions" click on "View Ordered List ..."
  7. You'll now see a list of the handler mappings in the order in which they're run, with the Django Handler at the top.
  8. Click on the StaticFile handler at the bottom of the list, then click "Move Up" under "Actions" on the right-hand side.
    1. Note at this point you may receive a warning that by moving handlers you are detaching this virtual directory from the inheritance of the parent's handler mappings. This is what we want to do, so if you get this warning click "Yes."
  9. Continue to click "Move Up" until the StaticFile handler is at the top of the list of Handler Mappings.
With the StaticFile handler at the top of the list of Handler Mappings, requests that come into the static directory will be handled by the StaticFile handler as opposed to being processed by the Django Handler, so IIS will serve the static files now.

Confirm that everything is working by going to http://localhost:81/admin in a browser. You should now see the CSS being applied to the page.

Additional Concerns with Static Files

  1. Don't include the directory to which the STATIC_ROOT setting is pointing in your application's code repository; these files should not be tracked and versioned from that location, and the Django development server handles static files for you by serving them directly from their various locations. The collectstatic management command should be run as a normal part of the deployment process for production instances of your application.
  2. For complete information about Django's handling of static files, and production deployment of static files, please refer to the following section of the Django documentation:
    1. https://docs.djangoproject.com/en/1.9/howto/static-files/
    2. https://docs.djangoproject.com/en/1.9/howto/static-files/deployment/
At this point your Django application should be playing nicely with IIS.

Additional Information

The information in this section is merely for reference, and to retain some of the research that was performed in the preparation of this guide even though it ultimately may not have been used as part of the final configuration steps.

Resources/Further Reading

Interesting things I came across while preparing this guide, in no particular order:

Additional Avenues of Exploration


Install WFastCGI Gateway for IIS and Python 3.4

Installing WFastCGI for IIS through the IIS Manager is completely optional. It's simply an alternate way of installing Python (in this case Python 3.4 specifically), and also provides the wfastcgi.py file that is required to configure a handler in IIS for Python applications.

If you perform these steps, Python 3.4 will be installed at the root of the C: drive, and this distribution of Python 3.4 will also place the wfastcgi.py file in the C:\Python34 directory. You can then leverage this file or copy it into your application's directory to configure the IIS handler.
  1. Open IIS Manager
  2. Click on the server in the list under "Connections" on the left-hand side
  3. At this point you may see a dialog box prompting you to install Microsoft Web Platform:
    1. If you see this dialog box, click "Yes." This will open the download page in a web browser.
    2. If you do not see this box, under "Actions" on the right-hand side of IIS Manager, click on "Get New Web Platform Components." This will open the download page in a web browser.
    3. You will be prompted to launch the Microsoft Web Platform application every time you launch IIS Manager, so if you see this dialog box after you have already installed the Web Platform Installer, feel free to check "Do not show this message" and then click "No."
  4. On the Microsoft Web Platform web page, click the "Free Download" button.
  5. After the download completes, located the downloaded file and double-click to run it. As of the time of this writing the file is called wpilauncher.exe
  6. If you see a Security Warning dialog, click "Run" to run the installer.
  7. After the installation completes, the Web Platform Installer 5.0 application will launch.
  8. In the Web Platform Installer 5.0 application, in the search box on the top right, type "fastcgi" (without the quotes) and press Enter.
  9. In the search results, locate "WFastCGI 2.1 Gateway for IIS and Python 3.4" and click the "Add" button.
  10. At the bottom of the Web Platform Installer application, click "Install."
  11. In the Prerequisites dialog that appears, click "I Accept" to accept the terms of the license.
  12. The WFastCGI extension will be downloaded and installed. When the installation completes, click "Finish" to close the installation dialog, and then click "Exit" to close the Web Platform Installer.
Note that doing installing the WFastCGI Gateway and Python 3.4 in this way does not automatically configure the FastCGI and Handler Mappings in IIS for you. It does install a default Python FastCGI application, but this will not necessarily automatically work with your application, and at a minimum you would still need to add a Handler Mapping for your application.

Thursday, July 7, 2016

Exporting and Converting iPhone (iOS 9) Wallpaper Files

I found myself needing to export and convert iPhone (iOS 9) wallpaper files from an old iPhone, and the source photos for these files no longer exist. As with most things Apple this turned out to be a horrendous pain but I was determined not to let Apple win. To paraphrase Ferris Bueller, "If I'm gonna get busted,  it is not gonna be by a company like that."

Although getting here was pretty time-consuming, the end result is simple. I did this on Windows but it should work on a Mac as well, provided you can install Pillow and run some Python.

  1. Connect the iPhone to your computer
  2. Backup the iPhone using iTunes
  3. Grab iExplorer and run it, allowing it to connect to the iPhone
  4. In the Backup Explorer, navigate to HomeDomain -> Library -> Springboard
  5. In that directory you'll see HomeBackground.cpbitmap and LockBackground.cpbitmap files, along with .jpg thumbnails of the files. Export these files from iExplorer to your computer.
  6. The .cpbitmap files are worthless (except to the iPhone) on their own, so finally you need to run a Python script to convert the .cpbitmap files to a usable image file.
I had to mess with some of the various Python scripts I found online to get it working properly (this one came closest), so if you're reading this at a future date or using a different version of iOS, I won't be surprised if it needs further tweaking.

Hope that helps others retrieve their wallpaper files!

Tuesday, June 21, 2016

Fixing DPI Scaling Issues in Skype for Business on Windows 10

My setup for my day job these days is a Surface Pro 4 and either an LG 34UC87M-B or a Dell P2715Q monitor, depending on where I'm working. This is a fantastic setup, but some applications have trouble dealing with the high pixel density and don't scale appropriately.

One case in point is Skype for Business. For some reason it scales correctly as I move between the Surface screen and the external monitor when I use the Dell, but on the LG monitor Skype is either massive on the external monitor, or tiny on the Surface screen.

After a big of digging around I came across a solution that worked for me, which is to change a setting in Skype's manifest file (who knew there was one?). On my machine the file is here:
C:\Program Files\Microsoft Office\Office16\LYNC.EXE.MANIFEST

And the setting in question is this:
<dpiAware>True/PM</dpiAware>

Which I changed to this:
<dpiAware>False/PM</dpiAware>

Note that you'll probably have to edit the file as administrator in order to be able to save it.

Once I made that change, saved the file, and restarted Skype, it now scales correctly for both the Surface screen and the external LG monitor.

Thursday, June 16, 2016

Installing Python 3 and Django on Dreamhost

I'm working on a side project for a friend that has zero budget and since I'm already paying for Dreamhost (my favorite and only choice for shared hosting) and totally underusing it, I figured I'd use this project as my opportunity to figure out how to install Python 3 and Django since A) Dreamhost lets you do install things even on their shared platform (one of the reasons I love them!), and B) the default Python on Dreamhost is Python 2.7.3.

1. Install Python 3.5.1


This part's really simple and Dreamhost even documents it nicely on their knowledge base, but for the sake of one-stop shopping I'll include the steps I performed here. Note that I'm leaving the default Python 2.7.3 in place since I'm not sure what havoc it would wreak to blow it away, so Python 3.5.1 will be installed in such a way that it'll be run as python3. (For the curious, you can also read more about Python at Dreamhost in general.)

Before going any further, if you haven't enabled SSH access for your Dreamhost user you need to do that first since if you can't log in you can't install anything.
  1. Log into the Dreamhost web panel
  2. Click on "Users" and then "Manage Users"
  3. Find the relevant user and click the edit button
  4. Under "User Type" choose "Shell user," and I'd recommend also choosing /bin/bash as the shell type
  5. Scroll to the bottom of the page and click "Save Changes"
Next, we'll install Python 3.5.1.
  1. Go to https://www.python.org/downloads/release/python-351/ and get the URL for the gzipped source tarball.
  2. SSH into your Dreamhost server
  3. In your user's home directory, wget the tgz file using the URL you grabbed in step 1:
    wget https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tgz
  4. Extract the Python source:
    tar xvf Python-3.5.1.tgz
  5. cd into the extracted source directory:
    cd Python-3.5.1
  6. Configure Python 3.5.1 with a prefix flag so as not to stomp on anything that's already installed:
    ./configure --prefix=$HOME/opt/python-3.5.1
  7. Run make and make install:
    make && make install
  8. Add the new Python installation to your path:
    export PATH=$HOME/opt/python-3.5.1/bin:$PATH
  9. Update your bash profile:
    . ~/.bash_profile
  10. Verify that everything's working (this should output Python 3.5.1):
    python3 -V
If that all looks good at this point, we're ready to proceed and create a virtualenv in which we'll install Django.

2. Create a virtualenv


Python 3 ships with pyvenv, but since it works a bit differently than virtualenv which is what I'm used to using, I decided to install virtualenv and use that. Feel free to use pyenv here if that floats your boat.
  1. Use pip3 to install virtualenv:
    pip3 install virtualenv
  2. Create and/or cd into the directory in which you want to put your virtualenv. I tend to put mine in ~/.virtualenvs, so I did this:
    mkdir ~/.virtualenvs
    cd ~/.virtualenvs
  3. Create a virtualenv to use for our Django project (since we installed virtualenv with pip3 the default Python environment will be Python 3.5.1):
    virtualenv my_django_project
  4. Activate your virtualenv:
    source my_django_project/bin/activate
  5. Confirm everything looks correct at this point (this should output /path/to/virtualenv/bin/python3):
    which python3
If we're still on track we can proceed to install Django.

3. Install Django (and your other requirements)


At this point since we have Python 3.5.1 installed and have successfully created a virtualenv, we can do the usual installation of requirements using pip3, which typically will come from a requirements.txt file. For the sake of illustration we'll go through installing Django into our virtualenv and making sure it works, but pip3 install -r requirements.txt will be the more typical way of installing all the packages you need.
  1. With your virtualenv activated, install Django:
    pip3 install django
  2. Verify everything is working by starting a Python interpreter:
    python3
  3. Import Django to make sure it's in your virtualenv (this shouldn't throw any errors):
    >>> import django
Now that we have verified that our virtualenv is working and sane, we can proceed to the rest of the configuration to get a Django project running successfully in this environment on Dreamhost.

4. Enable Passenger on Your Web Site


Rather than the gunicorn or some of the other WSGI servers you may be familiar with, on Dreamhost you can use either Passenger or FastCGI as the application server for Django applications.

For this post I'll be using Passenger, a tool with which I was not familiar until I saw information about it on Dreamhost. Passenger is apparently more common in Ruby on Rails setups, but it serves the same purpose and falls in the same place in the stack as any of the other WSGI tools with which you may have experience. You can read more about Dreamhost's use of Passenger here and here.

Some Notes About Passenger


Worthy to note about using Passenger on Dreamhost is that once Passenger is enabled, all requests that come into your domain will be handed off to Passenger by Apache. Passenger will first look see if there are any files matching the request in your domain's /public directory (and note that the files must be in the /public directory or subdirectories under /public specifically), and if it can't find an appropriate file, Passenger then tries to handle the request itself.

To provide a couple of concrete examples, let's assume you have a file called hello.html in your domain's /public directory, and a request comes in for http://yourdomain.com/hello.html. Here's what happens next:
  1. Apache hands the request off to Passenger immediately. This is important to note since Apache is out of the picture entirely other than serving as a proxy to Passenger.
  2. Passenger looks for a file called hello.html in your domain's /public directory. Since it finds one, it returns that page to the requester.
Now let's assume a request comes in for http://yourdomain.com/some/django/url. Here's what happens in that case:
  1. Apache hands the request off to Passenger.
  2. Passenger looks for an appropriate file in /public. This time it doesn't find one.
  3. Passenger then attempts to handle the request itself, and since later we'll be configuring Passenger to serve up our Django application using WSGI, at this point Django would be processing the request.

Enable Passenger


In order to run a Django application on Dreamhost we'll need to enable Passenger on the domain on which we're going to run our Django application.
  1. Log into the Dreamhost web panel
  2. Click "Domains" and then click "Manage Domains"
  3. Locate the domain on which you'll be hosting your Django application and click "Edit"
  4. Scroll down and under "Web Options" you'll see a checkbox labeled "Passenger (Ruby/NodeJS/Python apps only)." Check that box.
  5. Scroll to the bottom of the page and click "Change Settings."
That's it for enabling Passenger. Next we'll throw our Django application at Dreamhost and see what sticks.

5. Create a Django App


For the purposes of this post we'll be creating and configuring a Django application directly on the Dreamhost server. Obviously if you have an application that's already written that you want to run on Dreamhost you'd be uploading your application's files as opposed to doing things from scratch on the server, but this should illustrate some of the nuances of what you need to do with your own Django application to get it running on Dreamhost with Passenger.

  1. ssh into the Dreamhost server
  2. Activate the virtualenv you created above:
    source ~/.virtualenvs/my_django_project/bin/activate
  3. cd into the root directory of the web site on which you'll be running your Django application. Note that this is not the aforementioned /public directory under your web site's directory, but the root of the web site itself.
  4. Start a new Django project using django-admin.py:
    python3 django-admin.py startproject my_django_project
  5. Do a quick gut check by firing up the Django development server:
    cd my_django_project
    python3 manage.py runserver
At this point you should see the Django development server start up without any errors. We're nearly there! The last step is to configure this newly created Django application to work with Passenger. But before we do that, a quick note about static files.

Serving Static Files and the Static Root Setting

It wouldn't be Django if we didn't have to stop and say a word about serving static files. Luckily this isn't anything out of the ordinary, but given how Passenger works you do need to be aware of how to set your STATIC_ROOT to make sure everything comes together smoothly.

  1. Create a static directory under your domain's public directory, i.e. you'll wind up with ~/yourdomain.com/public/static
  2. Go into your Django project's settings file and add a STATIC_ROOT setting:
    STATIC_ROOT = os.path.dirname(BASE_DIR) + '/public/static/'
With that you should be all set to run collectstatic and have everything work properly. Now let's move on to configuring Passenger.

6. Configure Passenger


In order to have Passenger load up our Django application, we need to create a passenger_wsgi.py file in the root of our domain. As with all WSGI files this is what tells the application server, in this case Passenger, how to load up the application so it can handle requests.

In the root of your domain, create a passenger_wsgi.py file with the following contents:


With that file in place, you should be able to hit http://yourdomain.com and see the default "Congratulations" Django page. Congratulations indeed! You now have a Django application running on Python 3.5.1 on Dreamhost.

A common error you might run into is something like "no module named foo" or "no module named foo.settings." If you get something along these lines it means your paths are incorrect, so make sure that your Django project is in a directory directly inside your domain's root, not in the /public directory. If that isn't the issue, double-check all the paths in the passenger_wsgi.py file.

Additional Passenger Considerations


It's worth noting that if you change your passenger_wsgi.py file you need to run pkill python to reload the Django application since Passenger loads your application into memory (which is one of its advertised advantages since it makes things pretty zippy).

I also saw note of having to run touch tmp/restart.txt but in my experience I didn't have to do that, so I'm not sure if that's more of a Rails thing or if I simply didn't yet run into situations in which that was necessary. I'm noting it here in the event pkill python doesn't do the trick.

Final Notes


There you have it. You now have the latest Python and Django running on Dreamhost. At this point you can load your own Django project, set up a database (note that Dreamhost only supports MySQL, and even though I much prefer Postgres, MySQL works just fine), and do all the usual Django stuff.

Enjoy!

Monday, June 13, 2016

Installing PyGame with Python 3.5 on Windows 10

Quick installation notes for PyGame with Python 3.5 on Windows 10 since the available installers didn't work for me, the symptom being the apparently highly frequent "pygame module has no attribute init" error when trying to call pygame.init() after importing pygame.

First if you've run any PyGame installers previously, make sure to clean up anything they installed. The easiest way to do this is to run the installer again and choose the uninstall option, but you may also want to check the Lib/site-packages directory under your Python installation to make sure there aren't any remnants.

Next, go here and download the appropriate .whl file for your environment.

Next, open a command prompt and run:
pip install wheel

Finally, browse to the directory where your downloaded .whl file is located and run:
pip install pygame_file_name_here.whl

Restart your command prompt to be safe, and at that point you should be able to do this in a Python interpreter without getting errors:
import pygame
pygame.init()

Hope that saves someone else some head against desk time.

Saturday, December 5, 2015

New Project: "Learning Django REST Framework" Book

Quick announcement that I'm writing a book! Learning Django REST Framework will be published by Packt Publishing around the middle of next year. Excited!

Sunday, November 1, 2015

Python 2.7, Django, Apache, and Gunicorn on CentOS 6.5

I'm working on a little Django project for KPTZ Radio in Port Townsend and since this project has to talk to a serial relay board from a specific server that has other things running on it, I've been going through the process of installing Python 2.7 on CentOS 6.5, along with configuring Django, Apache, and Gunicorn.

Since I'm a lot more used to dealing with Nginx and Gunicorn on Ubuntu, getting this all up and running correctly took a lot more trial and error than I thought it would, but I finally got it figured out so figured I'd share since I found a lot of either incomplete or misleading information about this as I searched for solutions.

Installing Python 2.7

Your first question is probably why I'm not installing Python 3. In the case of this particular project, pyserial was not (when I first started the project) Python 3 compatible, so rather than fight that battle I decided to use Python 2.7.

The problem with Python 2.7 is on CentOS 6.5, Python 2.6.6 is the default, and since there's other Python-related stuff running on the server already I didn't want to run the risk of screwing anything else up, so I had to install Python 2.7 as an alternate Python installation. Luckily there were a couple of resources from people who had already been through this so it wasn't an issue. Here's the steps I took on a fresh CentOS 6.5 VM I was using to do some trial runs before doing everything on the production server (do all these as the root user).

  1. yum -y update
  2. yum -y groupinstall "development tools" --skip-broken
  3. yum -y install wget gcc gcc-c++ make httpd-devel git vim zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel
  4. wget https://www.python.org/ftp/python/2.7.10/Python-2.7.10.tgz
  5. tar xvf Python-2.7.10.tgz
  6. cd Python-2.7.10
  7. ./configure --prefix=/usr/local --enable-shared LDFLAGS="-Wl,-rpath /usr/local/lib"
  8. make && make altinstall
  9. python2.7 -V (to confirm it's working)
  10. cd ..
  11. wget --no-check-certificate https://pypi.python.org/packages/source/s/setuptools/setuptools-18.2.tar.gz
  12. tar xvf setuptools-18.2.tar.gz
  13. cd setuptools-18.2
  14. python2.7 setup.py install
  15. cd ..
  16. curl https://raw.githubusercontent.com/pypa/pip/master/contrib/get-pip.py | python2.7 -
  17. pip2.7 install virtualenv

Create a User to Own the Project

Depending on how you want to do things this could be considered optional, but I created a user to own the project files (again as root):
  1. useradd -m -s /bin/bash/someuser

Create a Python virtualenv and Install the Django Project

Next we'll create a Python 2.7 virtualenv, grab the Django code, install the Django project's requirements, and do a couple of other configuration things for the Django app.
  1. sudo su - someuser
  2. mkdir ~/.virtualenvs
  3. cd ~/.virtualenvs
  4. virtualenv foo --python=python2.7
  5. cd foo
  6. source bin/activate
  7. cd ~/
  8. git clone foo
  9. cd foo
  10. pip install -r requirements.txt
  11. python manage.py runserver (just to make sure things are working at this point)
  12. python manage.py collectstatic
  13. python manage.py migrate

Configure Upstart to Start the Gunicorn Process When the Server Boots

I suppose on the next version of CentOS this will be done with systemd but thankfully on CentOS 6.5 we can still use Upstart. Note that if you're familiar with Upstart on Ubuntu the syntax is quite different on CentOS -- thanks to my good friend and former coworker Brandon Culpepper for pointing that out before I lost my mind.

First, we'll do a quick test to make sure everything's working at this point:
  1. sudo su - 
  2. cd /home/someuser/foo
  3. /home/someuser/.virtualenvs/foo/bin/gunicorn --workers 4 --timeout 60 --bind 0.0.0.0:8000 foo.wsgi:application
  4. Hit Ctrl-C to kill the process if you don't see any errors.
Next we'll create the upstart file:
  1. vim /etc/init/foo.conf
  2. Put the following in the foo.conf file and save it:
    description "Gunicorn process for foo app"

    start on started sshd
    stop on shutdown

    script
      cd /home/someuser/foo
      /home/someuser/.virtualenvs/foo/bin/gunicorn --workers 4 --timeout 60 --log-level debug --bind 0.0.0.0:8000 foo.wsgi:application
    endscript
  3. start foo (to make sure the upstart process works)
  4. ps -wef | grep python (you should see some python processes running under your virtualenv)

Create Apache Virtual Host for the App

There's a bunch of ways to set up Django apps with Apache. In my early days with Django I would have used mod_wsgi but since I'm way more used to Nginx and Gunicorn these days, I figured I'd set up Apache in similar fashion and have it proxy to Gunicorn.
  1. vim /etc/httpd/conf/httpd.conf
  2. Uncomment the NameVirtualHost *.80 line if it isn't already uncommented
  3. Add a new VirtualHost section at the bottom of the Apache config file:
    <VirtualHost *:80>
      ServerName whatever
      DocumentRoot /home/someuser/foo

      # serve static files from Apache
      RewriteEngine on
      RewriteRule ^/static/.* - [L]

      # proxy everything else to the gunicorn process
      ProxyPreserveHost on

      RewriteRule ^(.*)$ http://127.0.0.1:8000$1 [P]
      ProxyPassReverse / http://127.0.0.1:8000/
    </VirtualHost>
  4. apachectl restart
At that point you should be all set! Hope that helps people who are in this same or a similar boat save some time.