UniMe Development Methodology
In the following, we propose some recommendations gathered from the experience matured during the RESERVOIR and VISIONCLOUD projects with particular focus on testbed development and management. These recommendations aim to improve the integration methodology, the usability of the Testbed, the use of the components and the tests. A general advice is to use Linux OS best practices, such as packages, daemons, logs, etc., for components development.
Components Packaging: OS specific packages (e.g. ebuild+source code in the case of Gentoo, deb or rpm packages otherwise) have to be provided by developers. Proper packaged components allow clean installation and uninstallation, taking care of setting proper paths, avoiding to leave dirty stuff behind and taking care of dependencies. This allow to move among different components releases more easily, in particular when candidate releases need to be tested, or the system needs to be reverted to stable releases.
Components configuration: It is important to provide a detailed description of each configuration file needed by the module and of the configuration files of third party software used by the module
Components Daemons: OS daemons have to be created for the components with at least start and stop options. Each component needs to take care of his running instances, which means that it has to manage OS pids, avoiding having multiple running instances.
Components logs: each component needs to take care of his own log, avoiding to have an always growing file. Moreover, it is required the possibility to compile a module in DEBUG mode in order to produce a verbose log file and to provide useful feedbacks to developers if the module does not work properly. For example, if the module needs an executable to be stored in a specific file system position (such as /usr/vision/bin/kvm) and such a component is not properly installed than this error has to be written in the log file.
Components tests: Before the commit of a software module, developers have to self-test it given that it is not possible to provide the root password of the testbed machines. Moreover, the commit of the software to svn has to be performed at fixed date every 1 or 2 week giving testbed administrators 1 or 2 days to upgrade, test and provide feedback to developers. A set of trivial automatic tests need to be provided for after installation verification. Developers have to provide information on how to verify if a module is correctly installed and runs properly. Moreover, it is requested that a module can be executed in a DEBUG mode that allows testbed administrators to run dummy tests to verify module functionalities.
Components information: An high level description of module functionalities has to be provided: information on paths, ports, processes, dependencies.
System Framework : a system framework with a given directories structure, where components need to be installed and with a DB for components configurations can give a more reliable and robust system. The directories structure allows to have a common environment for all components. The configuration DB can have components common configuration parameters, which avoid to have repetitions of parameters, and components specific parameters. The dir structure and the configuration DB can be provided as a template to developers, which need to adhere to it. Also, a set of management scripts to start, stop, check the status of the whole system should be provided as well (this should be done by the integration partner).
Continuous Integration: Integration f2f meetings, proved to be fundamental for the success of the project. They should be planned to be every 2-3 months and intensified close to important dates (e.g. Reviews).
System regression tests: Automated end-to-end tests should be created in order to test basic functionalities of the system, for example using one of the use cases.
Tools: The use of tools such as trac, skype sessions and weekly teleconferences it's of fundamental importance. In particular to always have a quick interaction with partners, skype availability should be at almost all time during working hours.
Use of Virtual Machines: once the architecture is defined and clear, you should try to embed as many components as possible within a VM, exploiting therefore the virtualization benefits. Advantages in such approach were:
- Install once - use everywhere: the VM could be copied to multiple sites and reconfigured for the needs of each site. This avoids the hassle of installing the components at each site, and provides a more uniform environment across different sites;
- Infrastructure flexibility: The infrastructure itself became more flexible, because its scope can be changed according to the current needs. For example, VMs for implementation or testing can be used during development phases, while stable VMs can be used in demonstrations. Also, the testbed can be partitioned by having multiple VMs running in parallel, allowing the different partners to have their own working environment to develop, experiment and test, without the need to share it, and therefore speeding up the developing life- cycle;
- Robust infrastructure: Fault tolerance and load balancing solutions could be implemented to make the infrastructure more resilient against failures.
For what concern t he development and testing work-flow, it can be organized according to the following phases:
- Development phase
- Integration testing phase
- Stabilization phase and global release
About the Development Phase a set of development VMs with tested project software stack will be provided. New versions of VMs with baseline code will be released as new integration tests are performed. Problems encountered by either developers or test bed owners will be logged as through a ticketing system. Scheduled test bed progress calls will be used to track progress on tickets.
During the Integration Testing phase, the new code will be submitted to the testbed manager for testing. Together with such a code also documentation for dependencies/libraries, tag for release version and results of single component tests have to be provided. In order to accomplish these operations, a set of testbed machines will be dedicated. The testbed manager will open tickets for detected issues.
In the stabilization phase deals with the propagation of the tested changes in the development VMs and in the machines used for demos.