CanberraUAV Outback Challenge 2016 Debrief
Please note that this HTML document is an Internet Explorer-friendly version of
Andrew Tridgell's original HTML document published on the ARDUPILOT blog website
at http://discuss.ardupilot.org/t/canberrauav-outback-challenge-2016-debrief/12162
CMAC Ed.
The members of CanberraUAV are home from the Outback Challenge and life is starting to return to normal after an extremely hectic (and fun!) time preparing our aircraft for this years challenge. It is time to write up our usual debrief acticle to give those of you who weren't able to be there some idea of what happened.
For reference here are the articles from the 2012 and 2014 challenges:
http://diydrones.com/profiles/blogs/canberrauav-outback-challenge-2012-debrief
http://diydrones.com/profiles/blogs/canberrauav-outback-challenge-2014-debrief
Medical Express
The Outback Challenge is held every two years in Queensland, Australia. As the challenge was completed by multiple teams in 2014 the organisers needed to come up with a new challenge. The new challenge for 2016 was called "Medical Express" and the challenge was to retrieve a blood sample from Joe at a remote landing site.
The back-story is that poor Outback Joe is trapped behind flood waters on his remote property in Queensland. Unfortunately he is ill, and doctors at a nearby hospital need a blood sample to diagnose his illness. A UAV is called in to fly a 23km path to a place where Joe is waiting. We only know Joes approximate position (within 100 meters) so first off the UAV needs to find Joe using an on-board camera. After finding Joe the aircraft needs to find a good landing site in an area littered with obstacles. The landing site needs to be more than 30 meters from Joe (to meet CASA safety requirements) but less than 80 meters (so Joe doesn't have to walk too far).
The aircraft then needs to perform an automatic landing, and then wait for Joe to load the blood sample into an easily accessible carrier. Joe then presses a button to indicate he is done loading the blood sample. The aircraft needs to wait for one minute for Joe to get clear, and then perform an automatic takeoff and flight back to the home location to deliver the blood sample to waiting hospital staff.
That story hides a lot of very challenging detail. For example, the UAV must maintain continuous telemetry contact with the operators back at the base. That needs to be done despite not knowing exactly where the landing site will be until the day before the challenge starts.
Also, the landing area has trees around it and no landing strip, so a normal fixed wing landing and takeoff is very problematic. The organisers wanted teams to come up with a VTOL solution and in this
they were very successful, kickstarting a huge effort to develop the VTOL capabilities of multiple open source autopilot systems.
The organisers also had provided a strict flight path that the teams have to follow to reach the search area where Joe is located. The winding path over the rural terrain of Dalby is strictly enforced, with any aircraft breaching the geofence required to immediately and automatically terminate by crashing into the ground.
The organisers also gave quite a wide range of flight distance and weather conditions that the teams had to be able to cope with. The distance to the search area could be up to 30km, meaning a round trip distance of 60km without taking into account all the time spent above the search area trying to find Joe. The teams had to be able to fly in up to 25 knots average wind on the ground, which could mean well over 30 knots in the air.
The mission also needed to be completed in one hour, including the time for spent loading the blood sample and circling above Joe.
Together these requirements meant that teams had to build an aircraft that could fly for a long time at quite high speed. You can see just how high from this graph of required airspeed to complete a 60km mission in 50 minutes at various air speeds:
This challenge is really pushing the edge of what the current generation of VTOL aircraft can do.
CanberraUAV
As we have done for the last two challenges CanberraUAV put in a huge effort to try to give us the best chance of completing the mission. That effort started in mid 2015 when the challenge was announced. At that point the rules made it clear that the most suitable aircraft would likely be a quadplane or a helicopter, but both solutions presented quite a high level of difficulty.
We were not at all sure a quadplane could land accurately in a 25 knot wind. The quadplane code for ArduPilot hadn't been written yet, so we had no way to test it, and we'd never built a quadplane before.
The helicopter option seemed attractive, but helicopters are well known for their high degree of mechanical difficulty. Helicopters are really in a constant state of trying to tear themselves to pieces, and building one with the speed and endurance for this challenge that can also reliably takeoff and land at a remote site with no pilot supervision was a real challenge.
To maximise our chances (and also because two aircraft is twice the fun!) we decided to develop both helicopters and quadplanes. We launched a large development effort to develop the quadplane code for ArduPilot while also working on adding many new features to the existing helicopter code in order to be able to meet the unusual requirements of the Outback Challenge.
For our helicopter work we were fortunate to be sponsored by Gaui with a steep discount on two of their new GX9 airframes. The GX9 gave us an extremely capable airframe that is perfect for the sort of long range high speed autonomous flight we needed in the challenge.
For the quadplane we went with the same base airframe that we used successfully in the 2014 challenge, which is a 2.7 meter VQ Porter with a DLE35 petrol engine. The porter normally flies with a takeoff weight of around 7kg, but once we converted it to an OctaQuadPlane and had all the additional equipment needed for it to complete the OBC mission it ended up at about 15kg. That is pushing up against the limits of what the airframe can handle. To cope with the weight we setup ArduPilot to limit the g-forces by limiting the maximum bank angle to 45 degrees and using large radius turns. That turned out to present a few problems in the search area, as in high wind the turn radius took us closer to the geofence than we would really have liked which meant very careful waypoint positioning and lots of pre-mission SITL testing in simulated high winds.
The 2.7m porter is really a brute force approach to the challenge. It is not an aerodynamically sleek at all, so it just uses the power of the 35cc petrol engine and a large fuel tank to give it the endurance it needs. The attraction of the Porter is that it has a huge open space for electronics inside, making it extremely easy to work on. That is perfect when doing lots of experimentation with new equipment.
We went with an OctaQuad quadplane after losing one of our prototype aircraft when an ESC failed:
building-flying-and-crashing-a-large-quadplane
we are very glad we did as we suffered from motor failures several more times in the lead up to the challenge, but we never lost two motors at once with this aircraft. We did lose two motors at once on one of our backup aircraft (also a VQ Porter) which resulted in the loss of the airframe. When we lose two motors on the OctaQuad we have sufficient control left to maintain stability but not enough to keep the descent rate below the threshold at which the ground impact is more than the airframe can handle.
Development and Testing
Something that many teams that are new to the Outback Challenge do not realise is just how much time it takes to prepare for the challenge. We have a dozen people in CanberraUAV and we had 4 or 5 people out at the flying field doing flight tests every weekend for months leading up to the challenge.
A lot of the testing was in perfecting the new quadplane code. We developed a number of simulation systems first to get the basics right, and also developed a number of smaller test aircraft which were invaluable in validating new code before we flew it on the primary aircraft.
CanberraUAV has always had a policy of publicly releasing everything we do while we are developing it and that allowed us to also benefit from feedback from users of quadplanes all over the world, including other Outback Challenge teams. The aim of the challenge isn't really to win some money, it is to advance the state of the art of amateur UAV technology so developing openly is a natural choice.
We were delighted to find that many other Outback Challenge teams chose to use ArduPilot and cooperate with us on development in the lead up to the challenge. Of the ten teams who ended up competing in Dalby eight of them were running ArduPilot. One was using Paparazzi (the amazing Delftacopter from the Netherlands) and the final team was
using a custom autopilot system built by a pair of very talented developers from Canada.
During the development phase we added many hundreds of changes to ArduPilot to extend its capabilities to meet the demands of the challenge. Each of these was reviewed and accepted by the ArduPilot development team as part of the normal ArduPilot development process. We made an effort to make the changes as generic as possible so they are useful to anyone using ArduPilot for advanced missions.
Reliable Communications Link
One of the keys to the 2016 challenge was communication links. In the 2014 challenge the aircraft remained in line of sight, although it was quite a long way from home (about 6km or so). That allowed us to use two radios that both relied in direct line of sight communication with the aircraft, a 5.8GHz Ubiquity ethernet bridge and a 900MHz RFD900.
For the 2016 challenge the aircraft needed to land at the remote site, which meant line of sight was unlikely to be retained. That means you need to use some sort of communications link that could be reliably retained without a direct link.
We decided to solve that in two ways. As we were developing two aircraft (the helicopter and the quadplane) we used one of them as a radio relay for the other. The idea is that the relay aircraft would stay well above the search area (at 600 foot AGL) so that it would retain a direct line of sight link with the ground station and it would relay telemetry data from the retrieval aircraft. We setup the helicopter and quadplane so either could fulfil either role, which gave us the flexibility to choose the most suitable aircraft on the day for the blood sample retrieval task.
The second approach we took was to use two 3G modems per aircraft to provide a higher bandwidth link. We used two modems so we could have one on the Telstra network and another on the Optus network. On the ground stations (two ground stations, one per aircraft) we also had two 3G modems, along with an RFD900x 900MHz telemetry radio.
The networking setup turned out to be much more complex than we initially anticipated. The reason for the complexity is we wanted to avoid a single point of failure anywhere in the network. The rules of the Outback Challenge mean that if you lose telemetry for more than 10 seconds then your aircraft has to start returning home, so reliability is absolutely critical.
To achieve that we had a "network box" containing two RaspberyPi3 boards running MAVProxy to distribute the telemetry data between the two ground station machines. That meant that if one laptop locked up and had to be rebooted then the other laptop would retain control of both vehicles. It also meant both laptops could use both 3G links, with MAVProxy automatically aggregating the links into a single telemetry stream.
We also needed to cope with other more complex failure scenarios. For example, when the retrieval aircraft is on the ground it seemed likely it would lose 3G as the local towers would be out of view. At the same time the relay aircraft would be at maximum range for the 900MHz link, so that may be down. To retain link in that period we needed to be able to bi-directionally relay telemetry from the retrieval aircraft on the ground to the relay aircraft over the 900MHz link and then forward that over either 3G link to the two ground stations. That required setting up additional links in between each of the network nodes.
As usual we made all of our configuration open. You can see for example the MAVProxy setup script for the primary retrieval GCS instance here:
In that script you can see the IP addresses of the two cloud servers we used to relay the MAVLink and image system UDP packets between the nodes. Use of cloud servers for UDP forwarding was required as the 3G modems did not provide public IP addresses, and also provided us with a convenient way to monitor the connectivity of the system.
We wrote a simple UDP proxy system which ran the total of 20 UDP network ports (ten proxies with 2 ports each) to run the network. These 20 UDP ports resulted from all the combinations of imagery system and MAVLink telemetry forwarding used in our setup to achieve the level of redundancy we needed. Using two separate cloud servers meant that if one went down we could still complete the challenge.
All of the start-up scripts and configuration files for all of the nodes in the network are available in our cuav repository. For example, you can see all of the start-up scripts and configuration for the first of the RaspberyPi3 boards here:
Mesh Network
The 2nd part of the networking solution was the 900MHz link. The RFD900 series of radios from RFDesign really set the benchmark for point to point telemetry in UAVs, but they only support a single aircraft. There have been patches for the RFD900 that support multiple ground stations with a single aircraft, but none that include the necessary packet forwarding logic to support multiple aircraft with one acting as a radio relay for the other.
To address this we used the new RFD900x radios which are based on the newer EZR32 radios. This gives a 32 bit arm chip in a radio with even better RF capabilities than the RFD900. We extended the firmware for the RFD900x to include a simple form of TDM based mesh networking, allowing for efficient operation of the 900MHz link with relaying between aircraft.
This simple mesh network worked extremely well and greatly increased our confidence in our ability to retain telemetry throughout the mission.
Travelling to Dalby
After all the preparation the time finally came to travel up to Dalby for the challenge. As our team is quite large we travelled in 4 cars plus the van that James had converted for use as our ground station. It is two days drive to Dalby, and we had some nervous moments as we weren't sure if the vans LPG tank could get us between LPG filling stations. We stopped overnight in Coonabarrabran then continued on to Dalby the next day.
The competition was held at the Dalby Model Aircraft Club, a truly great flying field just south of the township of Dalby. It was wonderful finally meeting up with all the other teams and getting a chance to look over their aircraft and discuss the solutions each team had come up with to the 2016 challenge.
Scrutineering
The first formal part of the challenge was the scrutineering, where the judges carefully inspect all aircraft to ensure they meet the challenge rules and are safe to fly.
One of the scrutineering tests that was particularly interesting was testing the geofence. We'd had that in previous years too, but in past years with normal fixed wing aircraft it was a simple matter of carrying the aircraft over the fence and demonstrating it entered a termination state (all control surfaces at maximum to enter a spin).
This year with the VTOL component the organisers wanted teams to demonstrate that all the VTOL motors as well as any forward motor also cut as you crossed the fence. This meant carrying the plane over the geofence with all 8 VTOL motors going and the forward petrol motor also running. We had never tried that before and as it inevitable with anything that you try for the first time it turned out we had a bug which caused the VTOL motors to keep spinning (slowly) after crossing the fence. That meant we failed the initial test and needed to try again.
After a few minutes of debugging we fixed the bug in the code and successfully passed the scrutineering. We also pushed out the change so that all teams using ArduPilot could benefit from the fix.
The names of the 10 teams were then drawn out of a hat to determine start order. As luck would have it CanberraUAV came out as the last team drawn out, which had its advantages and disadvantages. An advantage is we get to see how other teams go with the challenge before we needed to fly. A disadvantage is it meant we would more likely to fly during the predicted bad weather on the 2nd day of the competition.
We were also finally told the actual course we would need to fly and given the approximate location of Joe. We then had just one evening to fully prepare for the actual mission. We stayed up late that night developing the exact mission waypoints and testing scenarios in the SITL simulator. We had extended SITL to include the full camera and
imaging system we were going to use, along with using the real radios (both 3G and RFD900x) inside the simulator. That gave us a very realistic simulation system that allowed Stephen and myself (the two GCS operators) to be well prepared for the mission.
Competition Day 1
The competition organisers have done a really good write up of the flights of each of the teams that flew on the first day of the competition:
https://uavchallenge.org/2016/09/28/day-2-of-medical-express-2016/
Given we were drawn last from the hat we weren't expecting to be flying at all on that day, but as is so often the case the Outback Challenge threw up some surprises. A number of teams were quickly eliminated due to crashes and others elected to go to the back of the queue as their aircraft were not ready. The result was that right at the end of the first day we got our opportunity to fly.
We had setup our GCS van right on the southern end of the field to give the best possible radio link to the search area, with a 6 meter high mast to hold our three antennas.
After some fiddling to get the required NMEA feed to the organisers computers (for geofence compliance checking) we got through our pre-flights and got the first of our aircraft, the Porter quadplane, into the air.
It took off perfectly, starting the petrol motor automatically at 3m above the ground and transitioning to forward flight at 12m. Despite the seemingly perfect takeoff a later examination of the logs showed that the rear-left-bottom VTOL motor had failed part way through the takeoff. It wasn't noticeable at all while watching the takeoff or in the video as the stabilisation automatically compensated for the motor loss. We are still trying to work out why the motor failed. The logs show that all 8 VTOL motors worked fine for all the rest of the flight.
The plane headed off into the main mission at about 50 knots. We were deliberately flying it a bit slower than usual to minimise stress on the engine. We were also running the engine quite rich to maximise its reliability, which reduced its power somewhat.
After that we finished the pre-flight of the GX9 helicopter and it took-off as well. The heli takeoff was initially in stabilize mode, changing to AUTO once the pilot was ready. Our heli pilot Greg likes to check the aircraft is flying well before engaging AUTO, and as the challenge rules only require that one of the aircraft do an automatic takeoff we decided to make that the quadplane.
We had setup the two aircraft to use a vertical separation of 200 feet, so the quadplane flew at 400 foot AGL while the helicopter flew at 600 foot. Both were using the automatic terrain following code in ArduPilot to track the small terrain changes in the area around Dalby.
It took about 15 minutes for the Porter quadplane to reach the search area, after doing a nice fly past of the Dalby flying field. The helicopter followed close behind.
While I didn't realise it at the time, the helicopter navigation tuning was clearly a bit off, and spectators noticed that it "porpoised" a fair bit in turns. That may have been due to some of the weight changes with the recently built new undercarriage. I was concentrating on the quadplane ground station control so I didn't notice as otherwise it would have been quite simple to adjust for that during the flight.
Joe Search
The quadplane reached the search area with no issues and started its search for Joe. A few minutes later the helicopter also reached the search area and started its slow circuits at 600 foot to act as the radio relay.
This year we were using an updated version of the CanberraUAV imaging system we initially developed for the 2012 challenge. The system uses a 1280x960 PtGrey Chameleon machine vision camera mounted to point straight down from the aircraft. The camera is setup to capture images at 3.75 frames per second. Those images are processed by an onboard image recognition system that selects objects that it finds in each image based on a scoring system. Objects that are above a score threshold are sent as thumbnails to the GCS to be displayed in a "mosaic" form. The thumbnails are also overlaid on the map.
We had made a number of enhancements to the imaging system for this year. In particular we had adjusted the recognition system to be able to find much smaller objects. The organisers had told the teams that Joe would be standing up this year, which meant he would be much smaller when viewed from above, and he was not wearing the bright high visibility clothing that he had on in previous years. In our testing we used clothing that blended in well with the landscape, and ensured we could find Joe even with poorly contrasting images.
We also added a number of user interface improvements that gave the operator the ability to measure distances to possible landing sites on the ground, to place markers on the map based on image features and to "fly around" the area by showing sequences of images showing the objects at different angles.
The result was a much improved interface which made it much easier to find Joe and plan a remote landing.
The search itself was a complex clover-leaf pattern that allowed us to see objects on the ground from many angles and to cancel any systematic measurement errors from the camera and autopilot. In our pre-competition testing we had found we could land within one meter of the correct position based solely on the imaging system.
You can see a video of what the user interface looks like in this video:
The effort to improve the imaging system to cope with smaller object really paid off. It turned out that the view from a height of 230 feet Outback Joe looked like a small white blob. You can see the view from our on-board camera in this video:
the first white object you can see in the top left corner at the start of of the video is Outback Joe. The blue box around him is from our automatic image recognition system, which has flagged Joe as an object to show the GCS operator as a likely target.
If you watch the rest of the video you will see that the image recognition system found a lot of objects in the search area. Most of them had a much lower score than the recognition of Joe, but we did need the human input of the GCS operator to ensure we had the right object before landing.
After the mission the organisers confirmed that our estimate for the location of Joe was within 3m of their reference GPS position of Joe, which is within the typical margin of error for GPS in Australia.
Helicopter Crash
Part way through the search and shortly after we found outback Joe the GCS announced that the engine had stopped on the helicopter, and Stephen, who was operating the helicopter ground station, watched helplessly as the aircaft plunged to the ground and crashed. At the time we had no idea why the helicopter engine had stopped, but later examination of the logs and the aircraft leads us to believe that most likely cause is the engine overheated. We still don't know for sure why that happened, although it may be that the mixture was a bit too lean for the warmer weather in Queensland as compared to our testing in Canberra. Our helicopter lead developer Greg is still performing a detailed tear-down and examination of the engine.
The helicopter crashed quite close to the organisers tents, which must have been quite an experience. The helicopter weighed about 14 kilograms and has a 1.8 meter blade span which makes it quite an sight to see it plunging out of the sky and crashing 19 meters from where you are standing.
There was remarkably little damage given it hit the ground at 20 meters per second from a height of 600 foot. It came down level, and most of the impact was absorbed by the new heavy undercarriage. Greg thinks he may have it flying again soon.
Retrieving the blood sample
With the helicopter down we knew we were not eligible for the grand prize (as all aircraft have to come home safely for that), but the organisers kindly let us continue with the quadplane to retrieve the blood sample from Joe.
We had already found Joe and were preparing for the landing when the helicopter went down. So after we got permission to land we were able to complete the landing within a few minutes.
The landing itself went perfectly. Although were were confident of the landing accuracy of the quadplane we erred on the side of caution and asked it to land 42 meters from our estimated position for Joe. The number of points you get increases the closer you get to Joe, but if you come closer than 30 meters you are disqualified. So 42 meters gave us plenty of room for error.
After the landing the organisers measured how far we were from Joe and it turns out we landed 42.6 meters from Joe, which was nice to hear. They then loaded the blood sample and pressed the button to tell the plane that it could takeoff after one minute.
We were quite surprised to find that we maintained the telemetry link all the time the aircraft was on the ground at the remote site. All 3 links (the two 3G links and the RFD900x link) went down at least once while the aircraft was landed, but we never lost all 3 links at once. We were particularly surprised that the RFD900x link stayed up for most of the time the aircraft was on the ground. The mesh networking code had automatically switched to point-to-point mode when
the helicopter crashed and it kept the link despite having to travel through hundreds of meters of trees. It really is a testament to how well RFDesign has done with these radios.
We did get a huge amount of lag on the Optus 3G link though, with some packets arriving at the GCS over 80 seconds after they left the aircraft. Our telemetry code was designed to cope with link lag, but not lag of that magnitude and we will need to make some adjustments in our future developments.
After the blood sample was loaded we told the Porter that it could continue the mission, and it did a perfect automatic takeoff again, and started its long trip home.
With all of the delays from the helicopter crash and the long search over Joe we flew a total of 61 kilometres for the mission, and landed just past one hour into our mission time. That cost us a few points, but still left us with a good overall lead.
After landing the organisers retrieved the blood sample from the Porter and were quickly got our equipment packed up within our allotted 15 minutes of tear-down time. The challenge was over for us for another two years!
Final Scores
The final teams who had elected to go to the back of the queue flew tne next day, but unfortunately none of them made it to the search area. This left us as the only team to have successfully landed at the remote site and bring the blood sample home. We are delighted with that achievement, although the crash of our helicopter does dampen things a bit.
We're really looking forward to what the organisers come up with next time. They were not expecting any team to complete the challenge this year, so our completion means they need to come up with something new for 2018. It is bound to be a big challenge again!
Thanks
We'd like to offer a huge thank you to everyone who helped our with our 2016 OBC effort. A huge thank you to our sponsors Gaui, Halo Blades and Hobbyking, and a special thank you to all our friends at the Canberra Model Aircraft Club who have had to put up with all the crazy aircraft we have been flying (and crashing) over the last 18 months.
Also a huge thank you to all of the developers and users of ArduPilot who make this effort possible and such a fun community to be part of.
Thank you to our photographer Darrell Burkey for the great photos of the team (many of which are featured above), plus to the competition organiser photographers who took most the remaining photos above.
Finally thanks to the OBC challenge organisers for the huge effort they put in to run the challenge. The Outback Challenge has inspired a whole generation of UAV developers.
Back Row: Jack Pittar, Stephen Dade, Paul Tridgell, Andrew Tridgell, Greg Oakes, Darrell Burkey, Paul Williamson
Front Row: Grant Morphett, Peter Barker, Felix Williamson, Matthew Ridley, James Pattison
