Our FPGA-based robotics and vision research began in the summer of 2004 with a simple question. How do we enable real-time vision on very small, autonomous vehicles? Extensive work with small, ground-based robots and small UAVs had already been performed in BYU's Magicc Laboratory. However, the embedded microcontrollers in use had neither the I/O capabilities nor the processing power to handle real-time image processing.

We had at our disposal a number of FPGA development boards and decided to mount one of them on an existing robot platform. We then connected a tiny Micron image sensor to the system and developed a platform that would allow us to retrieve and process images.



This simple experiment immediately led to two graduate classes that used the platform as a basis for experimentation with small, autonomous robots with on-board vision. The first class was an attempt to play small-size robot soccer using only on-board vision. The second was a class on robotic vision where students learned a lot about obstacle detection and avoidance for use in robot navigation.

In the fall of 2004 we attempted to play two on two robot soccer using on-board vision. Robot soccer had been senior project option for undergraduate students for many years at BYU. However, all vision processing was done off-board on computer workstations and all the images were taken from an overhead camera. Moving the cameras and the processing to the robots themselves posed significant challenges.

In order to facilitate localization of the robot on the playing field, we devised a color scheme that gave each section of the field a unique color arrangement. The ball was simply a white golf ball. With this arrangement we were able to play crude robot soccer.



During this class, vast improvements were made to the robot platform. A custom daughter board was created that allowed us to connect a variety of devices to the robot, such as an electronic compass, ball sensors, a wireless modem, and, of course, the camera. Additionally, a great deal was learned about the FPGA and camera capabilities. We also learned about the Xilinx development tools, embedded processors, and image processing.

Here are some of the technical characteristics and features of the robot platform at the end of this class:

The robot was next used as the platform for a class on robotic vision. The students of the class were divided into four teams and each team was given one of the robots to work with. The core of the class was a series of challenges that each team was to compete in. The challenges are outlined below.

By the end of the robotic vision class of winter 2005, a tremendous amount of infrastructure had been built for this robot platform. Many of us were beginning to see the potential of this platform and were convinced that a platform like this is in fact the ideal solution for putting on-board vision on many small autonomous vehicles.

The existing FPGA board had several weaknesses. For example, it was far too large for many vehicles already in use at BYU. It was also very power hungry and used the slowest Virtex-II FPGA available. It was clear that a new FPGA board was needed. After a search of the available FPGA development boards, none was found that met the needs of the target application.

It was decided early in 2005 that we would develop our own very small, low-power, FPGA development board that would allow us to put on-board vision on a wide range of the small autonomous vehicles in use at BYU. The significant work already done with the existing robot platform was a tremendous help in developing the specification for the new board.

One of the features that would be essential for the board was its independence from the cameras and devices to which it would be connected. In other words, the board could not be designed for a specific camera model or sensors. Instead, all platform dependent hardware would be connected via one or more daughter boards. This would allow us to use the same board for a wide variety of autonomous vehicles and other applications.

The board was developed over the spring and summer terms of 2005. Construction of the first prototype was delayed by the availability of Xilinx's new Virtex-4 FX FPGAs. The first prototype was obtained in early November 2005. We were very happy to discover after significant testing that the board worked extremely well and only minor changes would need to be made for the final version.

The resulting board was named Helios after the Greek god of the sun who had the power to give vision and to take it away.

The first significant use for Helios was in the new Robot Racers senior project. In this senior project, for undergraduate seniors, students designed a robot based on a small model R/C truck that would navigate a complex race track using only on-board vision. Various obstacles would be placed on the course, including cones, jumps, and hairpin turns. The vehicles were required to operate completely autonomously, with no kind of external processing or feedback.

We were all pleasantly surprised with the results. These undergraduate students, with only limited digital design, image processing, and artificial intelligence experience were able to design complete robotic systems that could navigate the course at speeds much greater than anybody expected. It is powerful evidence that Helios is a very capable platform, truly giving vision to autonomous vehicles. See the video below to view a portion of the competition.

For more information on this and other senior projects in the department, visit the senior projects home page.