Premature Optimization

In the last few days I’ve been busy doing allot of web application performance benchmarks. For some of these benchmarks, it was important to simulate a real end-user’s average effective bandwidth (real-world download / upload speed from the end user’s perspective).

Since we used a fast, clean LAN for our benchmarks (you can’t get consistent numbers otherwise), we had to find a way to somehow limit the bandwidth of our LAN so that it would resemble “real Internet” throughputs. Since it took some research and reading to find the best way to do that, I’ve decided to write a quick HOWTO to describe what we’ve done.

Recent Linux kernels have built-in network traffic shaping capabilities. Those capabilities, in combination with the userland tool ‘tc’ can be used to set a bandwidth limit on one of your network interfaces, and even on the traffic incoming to a specific port.

First, you’ll need to compile the following features into your kernel, or compile them as modules (as I have done) and make sure you load them. In the 2.6.23 menuconfig:

Networking --->
  Networking Options --->
    QoS and/or fair queueing --->
      [*] QoS and/or fair queueing
      <M> Class Based Queueing (CBQ)
      <M> Universal 32bit comparisons w/ hashing (U32)

(you might want to enable other options here as modules as well for future use)

If you need to, recompile your kernel and reboot. On many distributions these options might be enabled by default though, so you don’t have to worry about recompiling and rebooting.

Next, you’ll need the userland tool to control your traffic shaping enabled kernel. This tool is ‘tc’ which is a part of the iproute2 package (on Gentoo: emerge sys-apps/iproute2). iproute2 is a set of powerful tools to control TCP/IP networking features in Linux - far more powerful than traditional tools like ‘route’, ‘ifconfig’, etc.

Once you have iproute2 installed, make sure tc works by running it as root (you’ll have to use root to preform the following steps):

# tc
Usage: tc [ OPTIONS ] OBJECT { COMMAND | help }
       tc [-force] -batch file
where  OBJECT := { qdisc | class | filter | action | monitor }
       OPTIONS := { -s[tatistics] | -d[etails] | -r[aw] | -b[atch] [file] }

Now we can start with the real work. To throttle down networking traffic incoming on eth0 to TCP port 80, you’ll have to run the following commands:

# tc qdisc add dev eth0 root handle 1: cbq avpkt 1000 bandwidth 100mbit
# tc class add dev eth0 parent 1: classid 1:1 cbq rate 128kbit \
  allot 1500 prio 5 bounded isolated
# tc filter add dev eth0 parent 1: protocol ip u32 \
  match ip protocol 6 0xff match ip dport 80 0xffff flowid 1:1

Now all traffic going to TCP port 80 on eth1 should be throttled down to 128kbit per second.

If you want to revert your network settings back to normal, this command should do the trick:

# tc qdisc del dev eth0 root

That’s it! If you’re one of those people who want to know what every command does before you run it as root on your server (sheesh!), here’s some words of explanation.

First - some terminology out of the tc(8) man page:

QDISCS
qdisc is short for ‘queueing discipline’ and it is elementary to understanding traffic control. Wenever the kernel needs to send a packet to an interface, it is enqueued to the qdisc configured for that interface. Immediately afterwards, the kernel tries to get as many packets as possible from the qdisc, for giving them to the network adaptor driver.

A simple QDISC is the ‘pfifo’ one, which does no processing at all and is a pure First In, First Out queue. It does however store traffic when the network interface can’t handle it momentarily.

CLASSES
Some qdiscs can contain classes, which contain further qdiscs - traffic may then be enqueued in any of the inner qdiscs, which are within the classes. When the kernel tries to dequeue a packet from such a classful qdisc it can come from any of the classes. A qdisc may for example prioritize certain kinds of traffic by trying to dequeue from certain classes before others.

FILTERS
A filter is used by a classful qdisc to determine in which class a packet will be enqueued. Whenever traffic arrives at a class with sub-classes, it needs to be classified. Various methods may be employed to do so, one of these are the filters. All filters attached to the class are called, until one of them returns with a verdict. If no verdict was made, other criteria may be available. This differs per qdisc.

It is important to notice that filters reside within qdiscs - they are not masters of what happens.

Now, what exactly the commands we gave mean?

# tc qdisc add dev eth0 root handle 1: cbq avpkt 1000 bandwidth 100mbit

Create a new root queueing discipline of type CBQ on interface eth0.
“bandwidth 100mb” is the bandwidth of the underlying physical interface. “avpkt 1000″ is the average size of packets. The qdisc needs both for it’s internal calculations.

“handle 1:” is kind of an alias for the qdisc we’ve just created.

CBQ stands for “Class Based Queueing” which is a common hierarchical classful queueing discipline that implements both shaping and prioritizing capabilities.

This is the root queueing discipline for eth0, and all non-filtered traffic will go into it.

# tc class add dev eth0 parent 1: classid 1:1 cbq rate 128kbit \
  allot 1500 prio 5 bounded isolated

Add a child CBQ queueing class under the “1:” qdisc we’ve just created on eth0, and name it “1:1″. Limit the rate to 128kbit per second.

“allot 1500″ specifies that the qdisc can take up to 1500 bytes off the queue on each round.

“prio 5″ specifies the priority of this queue (the lower the better - so packets that go into a queue with “prio 1″ will go first). In our case it is meaningless since we only have one queue - but “prio” is mandatory so we must specify it.

“bounded” and “isolated” are important here because they mean that the class will not borrow bandwidth from other inactive sibling queues - this is crucial when you want to throttle bandwidth down.

# tc filter add dev eth0 parent 1: protocol ip u32 \
  match ip protocol 6 0xff match ip dport 80 0xffff flowid 1:1

Creates the actual filter that will queue all packets going to TCP (”match ip protocol 6 0xff”) port 80 (”match ip dport 80 0xffff”) to dqisc “1:1″ which is our throttled queue. “u32″ is the filter type, and according to the docs it is “the most advanced filter available in the current implementation”, whatever that might mean.

That’s pretty much it. I hope this helps anyone - and if you have any comments or corrections, drop me a line below.

Most of the knowledge comes from the Linux Advanced Routing & Traffic Control HOWTO by Bert Hubert (and others). It’s a long and quite in-depth document, but is a very interesting read if you have the time.

If that’s not enough, the next best place to look for information would be ‘man tc’