Restore

Motivation for a Hybrid Testbed

After having designed a new network protocol, e.g, an autoconfiguration protocol, there are typically several possibilities to evaluate and validate it. The approaches of evaluation and performance analysis of network protocols can be classified into five categories. These are theoretical analysis, simulation, evaluation through emulation or virtualization, and the direct measurement in a real world testbed. All these evaluation methods are different concerning their level of abstraction in relation to the real application. A mathematical analysis has got the highest level of abstraction and, in descending order, is followed by simulation, emulation, virtualization, and finally reproduction in a real world testbed. The use of simplifying quantitative models leads thereby to a deviating behavior of the experimental setup. The more parameters remain unconsidered in such a model, the larger the inaccuracies will be that can occur in the evaluation.

Besides the real testbed environment, two other approaches are close to reality: emulation and virtualization. Within the emulation concept, there are two approaches, the network emulation and the environment emulation. When using network emulation, simulated components can communicate with network protocols realized in the real world. In environment emulation, real network protocols are embedded into the simulation environment. In the latter case, the simulation has to provide the same environment as a particular operating system (OS) for the network protocol implementation. Thereby, the simulated components in an environment emulation behave as they would do in a real world environment. Thus, this type of emulation is closer to reality compared to the network emulation. However, environment emulation is bound to the imitation of a particular existing OS. Generally, such an imitation is rather difficult to achieve.

In contrast to emulation, the competing virtualization approach replicates a whole machine or a particular OS. Therefore, there is no need to simulate a machine, OS, or network. Moreover, it is possible to create several virtual machines on a single host system. Each of these machines may run a separate OS and, hence, represents an entire computer system. By coupling several virtual machines over the network, a whole virtual network of virtual machines can be created. Nonetheless, the most important advantage of virtualization is the fact that the development of the software can be done on the real machine, tested on the virtual network of virtual machines, and later be installed without any modifications on the real testbed.

The development process of a testbed will be reviewed to point out the advantages of the approach used in the UMIC-Mesh. Basically, this process can be divided into two phases. In the first phase, the testbed is designed and realized. Considering the UMIC-Mesh, that is to design and realize a semi-managed WMN. Nevertheless, this step necessarily includes decisions on hardware and software. Obviously, the most important software decision to be made is the choice of the OS. For this purpose, Linux and Unix derivatives are widely used. Subsequently, in the second phase, the realization of network protocols and tools starts. As it is known from software engineering, this process is an iterative one and comprises developing, distributing, and testing. The step of developing a protocol or tool consists of the implementation and its debugging. Furthermore, the distributing step includes the installation of the implementation and the validation of this installation to ensure a correct distribution among all testbed systems. In a final step, the functionality and performance of the implementation have to be tested and evaluated respectively. In addition, if any failure occurs in one of these steps, debugging information has to be collected and analyzed. Preferably, the environment for developing and testing should be as close to reality as possible—generally achieved best by utilizing real hardware and standard software components.

All in all, the iterative software development process is a complex and labor-intensive undertaking. In particular, the distribution of new software versions is a challenging task, since new versions have to be frequently distributed among all systems of the testbed. A hybrid testbed that combines a real testbed and a virtualization environment offers a solution of the problems mentioned above. That means, the separate tasks of the software development process are distributed to the real part and the virtualization one. On the one hand, the virtualization environment takes over the developing task (implementation and debugging), the validation, and the functionality testing. On the other hand, functionality testing and performance evaluation takes place in the real testbed. Thus, there is no reason for emulating a wireless network interface, i.e., the physical and the data link layer. The following sections describe in detail the software, hardware, and virtualization component of the hybrid testbed.