Tuesday, June 1, 2010

Open Source Bridge - The Return of Command-Line Kung Fu

Hal Pomeranz (@Hal_Pomeranz)
  • Independent consultant
  • SANS faculty fellow
    • author, track lead for Sec506:Linux/Unix Security
    • instructor for SANS forensics classes
      • teach a lot of people *nix over the years, see people struggling
      • Windows people aren't happy that all the forensics are done from the Linux command line
  • blogs
Simple Output Redirections
  • output one way, errors another
  • two different output streams (stdout, stderr)
    • make >/tmp/build.log 2>/tmp/build.errors
  • getting stderr out of your way
  • output and errors together to the bit bucket
    • make distclean >/dev/null 2>&1
  • can also use >> to append instead of overwrite
Add /dev/tcp/... Goodness
  • command output to network (bash only, disabled in older versions of ubuntu)
    • df >/dev/tcp/foo.example.com/9999
    • "netcat without netcat"
    • useful for offloading data without adding any additional software
  • or a simple port checker
    • echo >/dev/tcp/
    • echo >/dev/tcp/
    • this throws error message if the port isn't open (returns nothing if it is open)
    • e.g. if/then/else on command line
      • echo >/dev/tcp/host/22 && echo live port || echo dead port
      • this still shows error output and redirecting output (2>/dev/null) doesn't fix this unless you use parens--parens cause things to happen in subshell
    • put for loop around this and you get a port scanner
    • for ((i=1; $1 < 1024; i++)); do (echo >/dev/tcp/host/$i) 2>/dev/null && echo $i/tcp live; done
      • this spawns a new subshell for each loop iteration
      • can do the output redirection on the entire loop so a new subshell doesn't get spawned each time
      • for ((i=1; $1 < 1024; i++)); do echo >/dev/tcp/host/$i && echo $i/tcp live; done 2>/dev/null
  • practical uses
    • security/crime incident responses
    • want to capture a bunch of information about the status of the local machine, but can't write anything to the local machine because that would taint the state of the machine from a forensics standpoint
    • need to fake out the script command to write the data to a remote system
Fun With FIFOs
  • first in first out or named pipe
  • can make a fifio using mkfifo
  • pipe that looks like a file
  • script command wants to write to a file
  • can use a fifo to fake out the script command--send output elsewhere
  • mkfifo /tmp/fifio
  • cat /tmp/fifo > /dev/tcp/host/port &
  • script -f /tmp/fifo
  • from this point on, everything run in the terminal on the source machine is sent over the wire to another machine
    • mirrors to the other machine keystroke by keystroke
  • when dealing with disk images, need to do string searches to hunt down evidence from disk images
  • can use strings command to search
  • disk images are getting larger and larger
  • strings would have to do two passes to find ASCII strings, then unicode strings
  • can use the fifo to split the strings command and do both passes at once
    • cat /tmp/fifo | strings -a -t d -e l | gzip > /tmp/strings.unicode.gz (this gets unicode strings)
    • tee command splits input (like a t-joint in plumbing)
    • e.g. cat diskimage.dd | tee /tmp/fifo | strings -a -t d | gzip > /tmp/strings.ascii.gz
    • can keep stacking fifos and tee commands to split as many times as you want
Kill! Kill! Kill!
  • kill processes by name
    • pkill sshd
  • or perhaps more selectively
    • pkill -P 1 sshd
      • -P is parent id--process ID 1 is init, so this would kill the sshd that was started at boot
  • kill processes by user
    • pkill -u user
  • killling process associated with a particular port--not built into kill and pkill
    • kill `lsof -t -i :22`
    • -t only outputs the PIDs
  • unmount that volume
    • kill `lsof -t /home` -- kills all processes under the /home mount point
Killing by Start Time is Hard
  • timestamps on /proc aren't related to proc start time
  • pkill can only kill oldest (-o) or newest (-n) proc
  • lsof has no options to select process start time
  • ps -eo pid, comm, start_time is useless
    • if something is more than a few months old, you just get a starting year
    • completely inconsistent
  • ps does let you get at a couple of different time values
  • if you list etime (elapsed time) from ps, still irregular output, but irregular in a sensible fashion
    • get either mm:ss, hh:mm:ss, or days-hh:mm:ss
    • need to canonicalize the format to get the fields in a consistent order
      • change all the hyphens and colons to whitespace so awk can deal with it better
    • ps -eo pid,comm,etime | sed 's/[-:]/ /g' | awk '{print $1, $2, $6, $5, $4, $3}'
      • awk will output null for fields that don't exist
      • will cause issues if there are hyphens or spaces in the command name
      • with everything in a canonical order can finally parse the time
    • ps -eo pid,comm,etime | sed 's/[-:]/ /g' | awk '{print $1, $2, $6, $5, $4, $3}' | awk '{print $1, $2, ($3 + $4 * 60 + $5 * 3600 + $6 * 86400}'
      • this gives pid, command name, and seconds of elapsed time
  • comment from audience cat /proc/1/stat -- 11th field is the start time as unix epoch (?) -- need to verify; states this in the docs
  • convert epoch time to normal date/time
    • date -d @epochtime
Got the touch
  • use touch (as root) to manipulate timestamps at will
    • touch -t 201001010000 /tmp/testing
    • stat /tmp/testing
    • this sets last access/modify to whatever date you give it, but at this point change date is the timestamp when touch was executed
  • aside -- !$ gives you the argument from the last executed command
  • can use touch to set date on something for comparison purposes
  • obviously can be used by the bad guys to modify timestamps on files
  • ls -lrt will sort ascending on last modified date so the most recently modified stuff is right above your command prompt
  • what about ctime?
    • could go into the inode with a hex editor and change things that way
    • can get specific tools to handle this
    • debugfs on ext file system
      • debugfs -w -R 'set_inode_field /tmp/testing ctime 201001012222' /dev/mapper/elk-root
      • if you run debugfs it will show the ctime as what it was set to with debugfs
      • if you run stat, it'll show the real time still because there's caching going on at the OS level
      • how do you flush the cache?
        • echo 2 >/proc/sys/vm/drop_caches (can pass in 1, 2, or 3 to flush file, directory, or both)
Stumper Question #1
  • searching a directory structure
  • want to find files containing a particular string
  • only want to look in ASCII text files
  • find /etc -type f | xargs file | grep ' text' | sed 's/:[^:]*$//' | xargs grep -l mystring
Stumper Question #2
  • two directories
  • each will have some files in common and some unique
  • matching files may have different names between directories
  • create a list of unique files from both dirs
  • would have to use checksums due to matching files not necessarily having the same name
  • find dir1 dir2 -type f | xargs md5sum | sort
    • shows which checksums are the same, but the lines would be different as far as the unique command is concerned
  • find dir1 dir2 -type f | xargs md5sum | sort -u -k1,1 | awk '{$1=""; print}'
    • sort can be used to selectively uniqueify on columns
Slides at http://www.deer-run.com/~hal/

No comments: