Motivation for a Hybrid Testbed

When designing a new network protocol, there are several possibilities to evaluate and validate it. In general these are theoretical analysis, simulation, evaluation through emulation or virtualization, and the direct measurement in a real world testbed. All these evaluation methods are different concerning their level of abstraction in relation to the real application. A mathematical analysis has got the highest level of abstraction and, in descending order, is followed by simulation, emulation, virtualization, and finally reproduction in a real world testbed. The use of simplifying quantitative models leads thereby to a deviating behavior of the experimental setup. The more parameters remain unconsidered in such a model, the larger will be the inaccuracies in the evaluation [KotNewGra04]. Besides the different level of abstraction, all methods mentioned above have also different trade-offs in regard to applicability, repeatability, maintainability, scalability, and scenario creation. For a broader discussion on the advantages and the disadvantages of the different evaluation methods we refer to [ZimGunWen07].

Before we present our solution to study wireless mesh network, we will shortly review the software development process to point out the advantages of our approach. Basically, as known from software engineering, the software development process is iterative. It comprises developing, distributing, and testing. In the developing step the realizations of new network protocols and tools starts. Implementation and debugging are made. Subsequently, in the distributing step, the installation of the implementation and its validation are done to ensure a correct distribution among all testbed nodes. In the final step, the functionality and performance of the implementation have to be tested and evaluated respectively. In addition, if any failure occurs in one of these steps, debugging information has to be collected and analyzed. Preferably, the environment for developing and testing should be as close to reality as possible generally achieved best by utilizing real hardware and standard software.

All in all, the iterative software development process is a complex and labor-intensive undertaking. In particular, the distribution of new software versions is a challenging task, since new versions have to be distributed among all nodes of the testbed frequently. A hybrid testbed that combines a real testbed and a virtualized environment offers a solution to the problems mentioned above. The hybrid testbed allow us to benefit from the advantages of a real testbed but, at the same time, to avoid its disadvantages.

The separate tasks of the software development process are distributed over the virtualized environment and the real testbed. On one hand, the virtualized environment takes over the developing task (implementation and debugging), the validation, and the functionality testing. On the other hand, performance evaluation takes place in the real testbed, since a high degree of realism is of great concern to this step. Thus, there is no need for an accurate emulation of a wireless medium, as for example in [JudSte05], since the virtualized environment is used only for software development and functionality testing. Network performance evaluation is solely done in the real testbed. However, for scenario creation or testing special cases some emulation might be of help (see Realization).

To illustrate the benefit of the virtualized environment for the software development process consider for example that booting a real mesh router may take up to 2 minutes whereas booting the kernel of a virtualized machine only takes about 20 seconds. The same holds for compiling. As the real mesh routers are equipped with hardware which is appropriate for routing and forwarding only, compiling on the mesh routers is hardly bearable. In contrast, the virtualized environment may come up with high-end hardware to enable fast cycling through the software development process.

Besides a more efficient software development process a hybrid testbed offers a more flexible scenario creation as well. As virtualization allows fast and flexible setup of an almost arbitrary number of nodes, scenarios with many or few nodes, potentially leaving and joining the network, are easy to achieve. Moreover, even complicated scenarios in which we realize arbitrary WMN architectures (see WMN Architecture) are possible by defining which mesh routers operate as gateways or which mesh client have their routing functionality enabled. Thus, any mesh protocols, e.g., routing protocols or channel assignments algorithms, can be easily evaluated in different mesh architectures.