Skip to main content

Open Source Bridge - The Return of Command-Line Kung Fu

Hal Pomeranz (@Hal_Pomeranz)
  • Independent consultant
  • SANS faculty fellow
    • author, track lead for Sec506:Linux/Unix Security
    • instructor for SANS forensics classes
      • teach a lot of people *nix over the years, see people struggling
      • Windows people aren't happy that all the forensics are done from the Linux command line
  • blogs
Simple Output Redirections
  • output one way, errors another
  • two different output streams (stdout, stderr)
    • make >/tmp/build.log 2>/tmp/build.errors
  • getting stderr out of your way
  • output and errors together to the bit bucket
    • make distclean >/dev/null 2>&1
  • can also use >> to append instead of overwrite
Add /dev/tcp/... Goodness
  • command output to network (bash only, disabled in older versions of ubuntu)
    • df >/dev/tcp/
    • "netcat without netcat"
    • useful for offloading data without adding any additional software
  • or a simple port checker
    • echo >/dev/tcp/
    • echo >/dev/tcp/
    • this throws error message if the port isn't open (returns nothing if it is open)
    • e.g. if/then/else on command line
      • echo >/dev/tcp/host/22 && echo live port || echo dead port
      • this still shows error output and redirecting output (2>/dev/null) doesn't fix this unless you use parens--parens cause things to happen in subshell
    • put for loop around this and you get a port scanner
    • for ((i=1; $1 < 1024; i++)); do (echo >/dev/tcp/host/$i) 2>/dev/null && echo $i/tcp live; done
      • this spawns a new subshell for each loop iteration
      • can do the output redirection on the entire loop so a new subshell doesn't get spawned each time
      • for ((i=1; $1 < 1024; i++)); do echo >/dev/tcp/host/$i && echo $i/tcp live; done 2>/dev/null
  • practical uses
    • security/crime incident responses
    • want to capture a bunch of information about the status of the local machine, but can't write anything to the local machine because that would taint the state of the machine from a forensics standpoint
    • need to fake out the script command to write the data to a remote system
Fun With FIFOs
  • first in first out or named pipe
  • can make a fifio using mkfifo
  • pipe that looks like a file
  • script command wants to write to a file
  • can use a fifo to fake out the script command--send output elsewhere
  • mkfifo /tmp/fifio
  • cat /tmp/fifo > /dev/tcp/host/port &
  • script -f /tmp/fifo
  • from this point on, everything run in the terminal on the source machine is sent over the wire to another machine
    • mirrors to the other machine keystroke by keystroke
  • when dealing with disk images, need to do string searches to hunt down evidence from disk images
  • can use strings command to search
  • disk images are getting larger and larger
  • strings would have to do two passes to find ASCII strings, then unicode strings
  • can use the fifo to split the strings command and do both passes at once
    • cat /tmp/fifo | strings -a -t d -e l | gzip > /tmp/strings.unicode.gz (this gets unicode strings)
    • tee command splits input (like a t-joint in plumbing)
    • e.g. cat diskimage.dd | tee /tmp/fifo | strings -a -t d | gzip > /tmp/strings.ascii.gz
    • can keep stacking fifos and tee commands to split as many times as you want
Kill! Kill! Kill!
  • kill processes by name
    • pkill sshd
  • or perhaps more selectively
    • pkill -P 1 sshd
      • -P is parent id--process ID 1 is init, so this would kill the sshd that was started at boot
  • kill processes by user
    • pkill -u user
  • killling process associated with a particular port--not built into kill and pkill
    • kill `lsof -t -i :22`
    • -t only outputs the PIDs
  • unmount that volume
    • kill `lsof -t /home` -- kills all processes under the /home mount point
Killing by Start Time is Hard
  • timestamps on /proc aren't related to proc start time
  • pkill can only kill oldest (-o) or newest (-n) proc
  • lsof has no options to select process start time
  • ps -eo pid, comm, start_time is useless
    • if something is more than a few months old, you just get a starting year
    • completely inconsistent
  • ps does let you get at a couple of different time values
  • if you list etime (elapsed time) from ps, still irregular output, but irregular in a sensible fashion
    • get either mm:ss, hh:mm:ss, or days-hh:mm:ss
    • need to canonicalize the format to get the fields in a consistent order
      • change all the hyphens and colons to whitespace so awk can deal with it better
    • ps -eo pid,comm,etime | sed 's/[-:]/ /g' | awk '{print $1, $2, $6, $5, $4, $3}'
      • awk will output null for fields that don't exist
      • will cause issues if there are hyphens or spaces in the command name
      • with everything in a canonical order can finally parse the time
    • ps -eo pid,comm,etime | sed 's/[-:]/ /g' | awk '{print $1, $2, $6, $5, $4, $3}' | awk '{print $1, $2, ($3 + $4 * 60 + $5 * 3600 + $6 * 86400}'
      • this gives pid, command name, and seconds of elapsed time
  • comment from audience cat /proc/1/stat -- 11th field is the start time as unix epoch (?) -- need to verify; states this in the docs
  • convert epoch time to normal date/time
    • date -d @epochtime
Got the touch
  • use touch (as root) to manipulate timestamps at will
    • touch -t 201001010000 /tmp/testing
    • stat /tmp/testing
    • this sets last access/modify to whatever date you give it, but at this point change date is the timestamp when touch was executed
  • aside -- !$ gives you the argument from the last executed command
  • can use touch to set date on something for comparison purposes
  • obviously can be used by the bad guys to modify timestamps on files
  • ls -lrt will sort ascending on last modified date so the most recently modified stuff is right above your command prompt
  • what about ctime?
    • could go into the inode with a hex editor and change things that way
    • can get specific tools to handle this
    • debugfs on ext file system
      • debugfs -w -R 'set_inode_field /tmp/testing ctime 201001012222' /dev/mapper/elk-root
      • if you run debugfs it will show the ctime as what it was set to with debugfs
      • if you run stat, it'll show the real time still because there's caching going on at the OS level
      • how do you flush the cache?
        • echo 2 >/proc/sys/vm/drop_caches (can pass in 1, 2, or 3 to flush file, directory, or both)
Stumper Question #1
  • searching a directory structure
  • want to find files containing a particular string
  • only want to look in ASCII text files
  • find /etc -type f | xargs file | grep ' text' | sed 's/:[^:]*$//' | xargs grep -l mystring
Stumper Question #2
  • two directories
  • each will have some files in common and some unique
  • matching files may have different names between directories
  • create a list of unique files from both dirs
  • would have to use checksums due to matching files not necessarily having the same name
  • find dir1 dir2 -type f | xargs md5sum | sort
    • shows which checksums are the same, but the lines would be different as far as the unique command is concerned
  • find dir1 dir2 -type f | xargs md5sum | sort -u -k1,1 | awk '{$1=""; print}'
    • sort can be used to selectively uniqueify on columns
Slides at


Popular posts from this blog

Installing and Configuring NextPVR as a Replacement for Windows Media Center

If you follow me on Google+ you'll know I had a recent rant about Windows Media Center, which after running fine for about a year suddenly decided as of January 29 it was done downloading the program guide and by extension was therefore done recording any TV shows.

I'll spare you more ranting and simply say that none of the suggestions I got (which I appreciate!) worked, and rather than spending more time figuring out why, I decided to try something different.

NextPVR is an awesome free (as in beer, not as in freedom unfortunately ...) PVR application for Windows that with a little bit of tweaking handily replaced Windows Media Center. It can even download guide data, which is apparently something WMC no longer feels like doing.

Background I wound up going down this road in a rather circuitous way. My initial goal for the weekend project was to get Raspbmc running on one of my Raspberry Pis. The latest version of XBMC has PVR functionality so I was anxious to try that out as a …

Running a Django Application on Windows Server 2012 with IIS

This is a first for me since under normal circumstances we run all our Django applications on Linux with Nginx, but we're in the process of developing an application for another department and due to the requirements around this project, we'll be handing the code off to them to deploy. They don't have any experience with Linux or web servers other than IIS, so I recently took up the challenge of figuring out how to run Django applications on Windows Server 2012 with IIS.

Based on the dated or complete lack of information around this I'm assuming it's not something that's very common in the wild, so I thought I'd share what I came up with in case others need to do this.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Assumptions and CaveatsThe operating system is Windows Server 2012 R2, 64-bit. If another variant of the operating system is being used, these instructions may not work properly.All of the soft…

Setting Up Django On a Raspberry Pi

This past weekend I finally got a chance to set up one of my two Raspberry Pis to use as a Django server so I thought I'd share the steps I went through both to save someone else attempting to do this some time as well as get any feedback in case there are different/better ways to do any of this.

I'm running this from my house (URL forthcoming once I get the real Django app finalized and put on the Raspberry Pi) using I don't cover that aspect of things in this post but I'm happy to write that up as well if people are interested.

General Comments and Assumptions

Using latest Raspbian “wheezy” distro as of 1/19/2013 (’lll be using Nginx ( as the web server/proxy and Gunicorn ( as the WSGI serverI used heavily as I was creating this, so many thanks to the author of that tutorial. If you’re looking for more details on …