Wanpipe Load Balancing with TEQL
--------------------------------


Introduction
------------

There are several ways of loadbalancing in Linux. One of the easiest and
straightforward ways is 'TEQL' - "True" (or "trivial") link equalizer.
Like most things having to do with queueing, load sharing goes both
ways. Both ends of a link may need to participate for full effect.

However in real-world situations, users control only one side
of the link, and telco the other.  Refer to diagrams below.

Imagine this situation:



                        +-------+   wan0   +-----------+
                    eth0|   A   |==========|   B       |eth0
        'network 1' ----|Wanpipe|          | Wanpipe   |---- 'network 2'
                        |       |==========|   or      |
			|       |          | Ext Router|
                        +-------+   wan1   +-----------+

or

                        +-------+       wan0 (ppp)    +-------+
                    eth0|   A   |=====================| Cisco |--- ISP1
    'local network' ----|Wanpipe|                     |       |
                        |       |========             +-------+
                        +-------+        | wan1  
				         | (fr)           
					 |            +-------+
					 =============|  Bay  |--- ISP2
	                                              |       |
						      +-------+


Therefore, we will concetrate on our side of the network (Router A),
and how to setup TEQL with multiple wanpipe devices, for greater
than T1 throughput. 



What is needed
--------------

1) Make sure that TEQL modules has been compiled
	modprobe -l | grep eql

   The module name is "sch_teql".

   If doesn't exist:
   	cd /usr/src/linux
	make menuconfig
            (network options-->QOS section--> enable TEQL as module)

	make modules
	make modules_install


2) Make sure that wanpipe has been installed and running.

   Refer to ftp.sangoma.com/linux/current_wanpipe/doc
   for further info.



Setup by Step Configuration
---------------------------

0) Configure ALL wanpipe interfaces with
   DYN_INTR_CFG option set to YES.
	
   This option will bring the wanpipe network 
   interface down, if the line goes down !!!  
   
   Once the interface goes down, it will be
   temporarily removed from the routing table until it comes
   back up.  
   
   While one interface is down, TEQL will force all traffic to 
   the working interface.
  

1) Start all wanpipe devices
	
	wanrouter start


2) Load a TEQL kernel module
	
	modprobe sch_teql

	or

	modprobe sch_teql max_equalizers=N

		Where N=number of eql devices


3) Setup queueing disciplines on each wanpipe interface
   using iproute2 utility called 'tc'. 
 
    	tc qdisc add dev wan0 root teql0
    	tc qdisc add dev wan1 root teql0


   Interfaces wan0 and wan1 could be running any protocol
   WANPIPE supports (PPP, Frame Relay, CHDLC ...)
   
   The device teql0 is basically a roundrobbin distributor 
   over wan0 and wan1, for sending packets. No data ever comes in 
   over an teql device, that just appears on the 'raw' wan0 and wan1.
	
   
4) Set the ip address of teql interface.  Note, the
   ip address used is ARBITRARY, it is fake.

   Reason for this is that packets will be forwarded through
   this ROUTER, thus packets will never originate from this
   box.  
   
   This means, however, that pinging from this machine
   will not be possible (your local address if fake). 

  	
	ip addr add dev teql0 10.0.0.0/31
    	ip link set teql0 up
    	ip route del 10.0.0.0/31
    	ip route add default dev teql0
	

   IMPORTANT:   The default route must be on teql0 device.

5) Goth router A and B need to have return path filtering turned off, 
   because they will otherwise drop packets destined for ip addresses 
   other than their own:
   
    	echo "0" > /proc/sys/net/ipv4/conf/wan0/rp_filter
    	echo "0" > /proc/sys/net/ipv4/conf/wan1/rp_filter



6) Load Balancing Issues

  There is the nasty problem of packet reordering. Let's say 6
  packets need to be sent from A to B - wan0 might get 1, 3 and 5. wan1
  would then do 2, 4 and 6. In an ideal world, router B would receive
  this in order, 1, 2, 3, 4, 5, 6. But the possibility is very real that
  the kernel gets it like this: 2, 1, 4, 3, 6, 5. 
  
  The problem is that this confuses TCP/IP. While not a problem for 
  links carrying many different TCP/IP sessions, you won't be able 
  to bundle multiple links and get to ftp a single file a lot 
  faster, except when your receiving or sending OS is Linux :), 
  which is not easily shaken by some simple reordering.



FAQ
---

1) Question

    I cannot get a single connection incoming or (downloading from here)
    to download at greater that 175KB.

   Answer
   
    OK that sounds about right.   The reason you are not getting download 
    throughput of greater than 175KB is because your telco is not load balancing 
    down to you.   The telco is sending the download stream over a single link.  
    However if you do multiple downloads you do see, a greater throughput. 

    In load balancing you can only control the outgoing direction.  The incoming can 
    only be controlled from the other end (telco end) of the link. 



2) Question  

    My main concern is redundancy, it seems as if one line
    goes down the entire connection is dead. I assume this is because
    I cannot route the ips originally assigned to me on the first
    T1 over the new T1.  Or am I mistaken.  Should I be able to route
    say 216.133.91.193-222/27 over either line regardless of which are
    up or down?

  Answer

    The TEQL determines the state of the link via state of
    the network interface.  Thus, if the link is down, the
    network interface must also come down and vice versa.

    Wanpipe drivers have this option; to use the network 
    interface state to show state of the link below.

    Thus,enable the dynamic interface config (DYN_INTR_CFG) 
    option in /usr/sbin/wancfg configurator.  
    (or directly in wanpipe1.conf).
    
    This option will bring the network interface down, if the line
    goes down !!!  Once the interface goes down, it will be
    temporarily removed from the routing table until it comes
    back up.



Further questions or comments can be emailed to:

Nenad Corbic <ncorbic@sangoma.com>
