Home made testbed

Architecture

We chose a relatively simple architecture:

• we have one NFS/DHCP/PXE/Rsync/etc server
• it is also a router/firewall:

• • the mesh network is protected from outside with very restrictive rules
  • but it can have a public access
  • the shorewall scripts are carefully configured
• all the mesh nodes are connected in Gigabit Ethernet to the firewall/server so that it can retrieve an updated image, mount NFS directories for a management purpore

Testbed installation

This could be quite tricky to maintain identical nodes (same packages, same environment). Hopefully, we have all the tools dedicated to clusters which can do most of the job.

After having hesitated longly, we chose to use the debian voyage with the following automatic procedure:

• PXE boot: a symbolic link in the pxelinux.cf either forces to update the OS or to boot the local compact flash
• I used the systemimager procedure to bootstrap the node but:
  • the install failed (grub fails, and much more details are boring …)
• thus, I rewrote almost from scratch the systemimager script:
  • sfdisk is your friend (be careful, you could have to force the CF type since the kernel does not automatically detect the correct characteristics)
  • first parition = the OS
  • second partition = temporary files we will download from our server via scp (you just have to get a private RSA key to login without problems…)
  • dd with the whole CF, using only the first X bytes (the MBR, partition table, etc) but without the second primary partition
  • you remove via ssh the script in pxelinux.cfg to force a local boot for the next time

and .. that’s all, you just have to reboot: no problem with grub, etc. since you did a dd with all the stuff!

This method works pretty well: you just have to change pxelinux.cfg (ln -s C*** XXX) to force a re-installation the next time the node reboots..

Obviously some tricks are required and require much time :-)

• conversion of IP address in hexa to have uniqueness
• uniqueness of image filenames
• creation of the script to update the image on the server and to maintain different images
• the correct sizes for the dd (or you will erase everything…)
• deactivating the bad modules, finding the reasons for a crash..
• chrooting for some details, etc.
• and debuging a node with minicom is quite tricky since to reboot it requires much time!