awk command: pick columns, filter rows, add them up

Syntax

awk 'condition { action }' file

Each line is split into columns on whitespace, and the action runs on matching lines.

Try it first

Take the first column, the IP address.

$ cat logs/access.log
10.0.0.1 /index 200 512
10.0.0.2 /missing 404 128
10.0.0.1 /index 200 640
10.0.0.3 /boom 500 96
10.0.0.2 /index 200 512
$ awk '{print $1}' logs/access.log
10.0.0.1
10.0.0.2
10.0.0.1
10.0.0.3
10.0.0.2

$1 is the first column, $2 the second. A comma between them prints a space.

$ awk '{print $1, $3}' logs/access.log
10.0.0.1 200
10.0.0.2 404
10.0.0.1 200
10.0.0.3 500
10.0.0.2 200

When lines have different numbers of columns, $NF, the last one, is handy.

$ awk '{print $NF}' logs/access.log
512
128
640
96
512

Filtering rows

A condition in front of the braces limits which lines are acted on.

$ awk '$3 == 200 {print $1}' logs/access.log
10.0.0.1
10.0.0.1
10.0.0.2

Leave out the action and matching lines print as they are.

$ awk '$3 == 200' logs/access.log
10.0.0.1 /index 200 512
10.0.0.1 /index 200 640
10.0.0.2 /index 200 512

Numbers can be compared, which is something grep cannot do.

$ awk '$4 > 500 {print $2, $4}' logs/access.log
/index 512
/index 640
/index 512

For text, put the pattern between slashes.

$ awk '/index/ {print $1}' logs/access.log
10.0.0.1
10.0.0.1
10.0.0.2

Totals and counts

END { ... } runs once, after every line has been read.

$ awk '{sum += $4} END {print sum}' logs/access.log
1888

Each line adds $4 to sum, and the total is printed at the end. For a line count, print NR.

$ awk 'END {print NR}' logs/access.log
5

Since NR is the current line number, it also skips headers.

Changing the separator

CSV needs -F.

$ awk -F, 'NR > 1 {print $2, $3}' users.csv
ada web
linus infra
grace infra

NR > 1 drops the header row.

When you actually reach for it

awk is what you type when the columns and a condition matter at the same time.

The rule of thumb against cut is short. Extracting only, use cut; conditions or arithmetic, use awk.

Things that trip people up

Wrap the script in single quotes. $1 means something to the shell too, so double quotes let the shell get there first. Always write awk '{print $1}'.

Extra spaces do not matter. awk treats a run of whitespace as one separator. Logs padded out to line up still give you the third column as $3.

It leads straight into counting.

$ awk '{print $1}' logs/access.log | sort | uniq -c | sort -nr
      2 10.0.0.2
      2 10.0.0.1
      1 10.0.0.3

Pick a column with awk, then let sort and uniq do the tallying.

A missing file says so.

$ awk '{print $1}' nothing.log
awk: nothing.log: No such file or directory