Article -- Listing IPs using regular expresions

Listing IPs Using Regular Expressions

I ran across an interesting situation recently. I have client with a web based application which is accessed almost exclusively by local systems with 192.1.168.X addresses and a few remote users. They wanted to be able to list remote IPs in the apache logs without seeing all the details and filter out local IPs from the list.
This turns out to be very easy to accomplish using regular expressions from th Linux command line. I will try to assume in this article that you are a Linux beginner. Typically I use Debian but most distributions should be almost the same.
First start with a basic listing of your apache access log file.
cat /var/log/apache2/access.log

And you get somthing like this.

192.168.1.1 - - [11/May/2006:13:54:33 -0500] "GET /cgi-bin/week.cgi?focusday=2006-6-10 HTTP/1.1" 200 22478 "http://bcpc.no-ip.com/cgi-bin/mainweek.cgi?focusday=2006-6-8" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20050922 Firefox/1.0.7 (Debian package 1.0.7-1)" 192.168.1.1 - - [11/May/2006:13:54:33 -0500] "GET /cgi-bin/week.cgi?focusday=2006-6-11 HTTP/1.1" 200 22476 "http://bcpc.no-ip.com/cgi-bin/mainweek.cgi?focusday=2006-6-8" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20050922 Firefox/1.0.7 (Debian package 1.0.7-1)" 192.168.1.1 - - [11/May/2006:13:54:34 -0500] "GET /cgi-bin/daily.cgi?focusday=2006-6-8&calfilter= HTTP/1.1" 200 18620 "http://bcpc.no-ip.com/cgi-bin/mainweek.cgi?focusday=2006-6-8" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20050922 Firefox/1.0.7 (Debian package 1.0.7-1)" 192.168.1.1 - - [11/May/2006:13:59:00 -0500] "GET /cgi-bin/title.cgi?searchtype=Keyword&searched=Search&browserwidth=912&changed=undefined HTTP/1.1" 200 4922

By using the grep command and regular expressions I can narrow this down a little.

grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' /var/log/apache2/access.log

Now I get a more readable list.

192.168.1.1
192.168.1.5
70.249.173.61
70.249.173.61
70.249.173.61
192.168.1.13
192.168.1.13
192.168.1.1
192.168.1.5

Now all that is left is to filter out the local addresses.

grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' /var/log/apache2/access.log | grep -v 192.168.1

Now I get a list more like this.

203.57.95.63
203.57.95.63
70.141.52.159
70.128.2.255
70.249.173.61

The [0-9] tells grep to match only digits from 0 to 9. The {1,3} says that between 1 and 3 numbers can be a group before the period. If I wanted just to match 2 digit numbers between 5 and 8 I would use '[5-8]\{2\}'. Notice that you need to include a backslash other wise grep will try to look for back slashes. The grub switches that were used are -o to show only the part of the line matches our regular expression and -v which tells grub to show everything that was not a match.
Next article I'll show you how to hide duplicate IPs.