Wednesday, October 27, 2010

Apache Error "Document Root Doesn't Exist" on Red Hat Enterprise LinuxWhen the Document Root DOES Exist

I upgraded one of our Red Hat Enterprise Linux VMs to Tomcat 7.0.4 tonight, did a bit of Apache reconfiguration, and when I restarted Apache I got a "document root doesn't exist" error even though in fact the document root does exist. (Trust me Apache, it's there.)

I double-checked the owners and permissions of all the directories in question and everything was identical to how things are set on another RHEL VM in this cluster on which I haven't upgraded Tomcat and done the reconfiguration yet. I was at a bit of a loss, so I googled around a bit and the prevailing sentiment seemed to be this was related to either A) a config file copied from a Windows box and the line breaks were throwing things off (wasn't applicable in my case), or B) the fact that SELinux was enabled.

If you've been around the Red Hat flavors of Linux long enough you'll remember that SELinux used to be absolutely horrible. For years the very first thing you had to do on Red Hat and Fedora to get anything working at all was turn SELinux off, and for a long time I believe it was even Red Hat's recommendation under their breath to just shut it off. It's gotten better over the years, and honestly stays out of the way to the point where I'd almost forgotten about it.

Since I didn't have anything else to try, however, I went into /etc/sysconfig/selinux and changed SELINUX=enforcing to SELINUX=disabled, restarted the server, and voila the complaining from Apache went away.

What I still don't get is A) why this isn't occurring on my other RHEL box with the same setup, and B) why it just started happening now. The only potential weirdness here is that my document root is a symlink, but again, it's been that way since I setup up these boxes originally and it hasn't been an issue.

So if you run into this same problem the fix (at least until I have more information about why it's happening) is to disable SELinux, but if anyone has more ideas about why this might be happening I'd love to hear them.


awest said...

Matt, I've run into the selinux wall several times installing ColdFusion 9 on CentOS with Apache 2.2.x. The issues were all related to selinux not allowing the Apache connector to communicate with resources on the system.The best way I've discovered to get CF and Apache working is to not hook up Apache during the installation of CF. Instead, I temporarily disable selinux and run the commands to manually hook CF to Apache post install.After install of CF and the Apache connector I stop Apache and CF and run a few selinux commands that a) set the Apache connector to run in the same selinux security context and all other Apache modules b) set selinux to allow scripts to connect to Apache.After doing these things I put selinux back into enforcing mode and reboot the OS. After restart I check that CF resources are properly going through Apache.Perhaps your issue with Tomcat is related. It may be that something in selinux changed from OS update to update and now selinux (in enforcing mode) is stopping file actions. The selinux log file was a big help for me in resolving the Apache + CF connector issues. Maybe there's something in there on your RHEL system.

Matthew Woodward said...

Thanks for the info Aaron. In this case it's just Apache not seeing a directory that exists so it doesn't even get to the level of the AJP proxying I'm doing between Apache and OpenBD, but that's good info to have. Honestly it doesn't get to the Tomcat level at all; I just have a docroot set in Apache and until I turned off SELinux, Apache couldn't see it. My first guess was permissions/ownership issues but on another VM in this cluster the permissions and ownership are the same (tomcat:tomcat) and Apache saw it just fine. All this leads me to the conclusion you mentioned, which is that a recent update to RHEL (many of which were security related) caused a change of some sort. When I did some work on another VM I had the same issue so that seems to reinforce this idea. I'll check out the selinux logs as you suggested. Thanks!

denstar said...

It depends on how you edited the config files. If you do it wrong (don't ask me how to do it right =]), the files are no longer under the Apache context in SELinux.You don't need to turn off SELinux, you just need to to add the stuff that shows up in /var/log/audit/audit.log (or using audit2allow) back to the security context. I always have to google how to do it. Something like this tho, I think:chcon -t httpd_var_lib_t /path/to/dirFreaking SELinux. It's Good Stuff tho. =)

Matthew Woodward said...

Thanks--great to know. I'll look into that before deciding to leave it off. ;-)