SSC16–IX–05

Supporting the Flock:Building a Ground Station Network for Autonomy and Reliability

Kyle Colton, Bryan KlofasPlanet Labs

346 9th Street, San Francisco CA 94103; 858–304–0302kyle.colton@planet.com

ABSTRACT

Planet Labs is on a mission to image the whole earth every day. This requires a large orbital constellationand a distributed, autonomous ground station network. The network currently includes 11 geographicallydiverse sites, many with multiple antenna systems. Antennas systems are deployed with uniform equipmentto simplify interfaces and ease operations. Additionally, tools have been developed for automated monitoringand remote troubleshooting of RF chains. Predictive modeling tools help to plan for future need as the orbitalconstellation grows. Lessons learned and ideas for automation and monitoring are also discussed.

A NEW IMAGE OF THE EARTHEVERY DAY

Planet Labs is on a mission to image the whole worldevery day and make global change visible, accessible,and actionable. We strive to improve the world byproviding a medium-resolution rapidly-updating dataset. Recently our data has been used to raise aware-ness about illegal mining in a natural reserve[1]. Inresponse to the recent fires in Alberta [2] and earth-quake in Ecuador [3], we released our imagery of theareas to aid government and humanitarian relief ef-forts.

As of May 2016, we have successfully launched 133Dove satellites from 11 different launch vehicles andthe International Space Station, and we are rapidlyscaling operations to meet our mission. Our cur-rent constellation consists of fifty-five 3U CubeSats.Planet has built the largest constellation of commer-cial earth imaging satellites, and our Doves imageanytime they are over land. This constellation gener-ates massive amounts of image data that we need tobring to the ground in a quick and efficient manner.In addition, we collect normal housekeeping teleme-try, update software, and refresh the on-board sched-ules.

MEETING THE NEEDS OF AGILEAEROSPACE

Agile aerospace is a new set of principles based on theagile software development concepts including rapiddesign iteration, a continuous improvement process,and easy scalability. As a result of Planet’s agile ap-

proach, recent ISS deployments have dramatically in-creased our constellation of satellites (or “Flock ofDoves”) to fifty-five on-orbit CubeSats. Our Dovestake approximately 650 passes per day on 33 antennasystems spread across our 11 active ground stationsites. The average amount of picture data downlinkedper day is approximately 550 GB, with a maximumof 777 GB [4]. Once these latest satellites finish com-missioning and begin operations, downlink numberswill rise further.

Our distributed ground station network provides sev-eral benefits; it reduces the impact of any one fail-ure, serves multiple orbital planes, reduces the timebetween image capture and downlink, and spreadsthe load of satellite downlinks more evenly across theglobe. At the same time, the multiple sites increasecomplexity of the network and result in a global,“always-on” ground station network that is main-tained, monitored, developed, and expanded by a rel-atively small team of five people.

Autonomy and reliability are necessary. Passes andtasks are scheduled by a computer model, faults aredetected and cleared as much as possible by the indi-vidual stations, and those not automatically resolvedare escalated to the on-call engineer. We rely onsoftware tools to aid in managing a network capa-ble of supporting the current and future constella-tion.

Building Uniform Ground Stations

The first Dove CubeSats were launched in April 2013,and there have been 14 satellite hardware revisions

Colton 1 30th Annual AIAA/USUConference on Small Satellites

since then that have included new processors, bet-ter persistent data storage, newer cameras, and moreadvanced optics. Even with these hardware changesthe link interface has been consistent throughout thistime period. Aside from better antennas and a recentbaud rate change (from 24 Mbaud to 60 Mbaud) thatrequired a hardware upgrade, all of the changes havebeen software.

On the ground station side of the link, there havebeen several hardware upgrades and countless soft-ware updates during the past three years. Hardwarechanges include new internet firewall appliances andethernet switches for increased network security, newtransceivers with reduced EMI and spurious signals,GPS time synchronization, and new computer serverswith more processing power and remote managementfeatures.

Figure 1: Planet’s Ground Station Network.

Our ground station network includes 11 active globalsites. While Figure 1 shows that most of our groundstation sites are located in the United States, wedo have sites scattered around the world. As dis-cussed previously [5, 6], most CubeSats are usingamateur radio frequencies, which are licensed for non-commercial use worldwide. Since we are a commercialcompany based in the United States, Planet acquiredcommercial licenses for our satellites [7]. Workingwith the different frequency licensing organizationsaround the world can be challenging and time con-suming.

Uniformity, however, aids in licensing. Licenses oftenrequire knowledge of the hardware, and by keeping itsimilar from site to site, we can begin the licensing ex-ploration and application process in advance.

Tracking, Telemetry, and Command

The Dove satellites use UHF for TT&C and orbit de-termination [9], so the constellation benefits greatlyfrom having frequent and global UHF link opportuni-ties. The Planet Labs UHF ground station uses sim-ple, inexpensive COTS components to support theneed for diverse coverage at a reasonable cost andwith easy maintenance. They use standard Yagi an-

tennas positioned by a Yaesu rotor for both uplinkand downlink. Figure 2 shows the antennas on a typ-ical UHF ground station.

Figure 2: Example UHF Station.

The only non-COTS component in our ground sta-tions is the custom-built SpaceTalker transceiver. Itis based on a Texas Instruments CC1110 wirelessMCU, which has both a UHF transceiver and mi-crocontroller unit in a single chip. The server com-municates with the transceiver over USB [8]. Boththe ground SpaceTalker radio and space hardware useextremely similar architecture and interfaces, increas-ing the system’s predictability and reliability. Figure3 shows a block diagram of our UHF systems.

Figure 3: UHF TT&C Block Diagram.

Colton 2 30th Annual AIAA/USUConference on Small Satellites

High-Speed Downlink

In addition to the UHF capabilities of the Planet LabsSpaceTalker radio, there is another CC1110 chip withupconversion to S-band. This is our high-speed up-link radio for uploading commands and new softwareto the constellation. We use 8.2 GHz for our high-speed data downlink. The encoding scheme is DVB-S2, which allows for different modulations, forward-error correction settings.

While developed for satellite television applications,the DVB-S2 standard includes a Generic Stream En-capsulation (GSE) protocol which allows any digi-tal data to be transmitted within DVB-S2 frames.Within the GSE, we use Internet Protocol format-ting. Combined with the on-board processor, whichruns vanilla Ubuntu, this formatting allows us to se-curely connect with our satellites and run commands,scripts, or anything that can be done on a normalcomputer on the ground. During nominal operations,the encrypted link is used for downlinking picturesand logs.

On the ground station side, we have three types ofdishes spread amongst our six S/X-band sites. All ofour dishes are designed to 29 dB/K or better. Figure4 shows a block diagram of our high-speed downlinkstations. Note the similarity with our UHF TT&Cstations.

Figure 4: High-Speed Downlink BlockDiagram.

Automating the Downlink

At a fundamental level, our satellites have a very sim-ple ConOps: when over land, point down and takepictures. Occasionally, the Doves take pictures ofthe moon for optical calibration, desaturate wheels,or turn on the star camera for attitude calibration.When a downlink session is scheduled, the Dove turnson the X-band transmitter and tracks the ground sta-tion as it passes under the satellite. Whenever thesatellites are not doing the above, they charge bat-teries in a low-drag state.

This basic ConOps cries out for a fully automatedsystem; Planet Labs has fifty-five satellites in orbitand will be scaling rapidly in the next year. PlanetLabs planned for satellite and ground station automa-tion from the beginning in order to allow the Flockto scale as necessary. Our operations team consistsof five “spaceship captains” responsible for commis-sioning new satellites, anomaly resolution on existingsatellites, and building tools to improve on-orbit op-erations. A high degree of autonomy is necessary forsuch a small team to support the on-orbit constella-tion.

These same automation concepts are applied to theground station network. The ground station team,which also consists of five people, does not have thetime nor the ability to manually control each of theground stations during its downlink sessions. Just be-fore each downlink session, the ground station down-loads the pass schedule, satellite commands (or tasks)to run during the pass, and new orbital elements fromMission Control for antenna pointing.

After the pass, pictures are uploaded to Amazon WebServices (AWS), a cloud-computing platform. Satel-lite and ground station telemetry and pass logs aresent back to Mission Control. The Mission Controlwebsite, also hosted on AWS, provides the Space-ship Captains and ground station engineers up-to-date information on each satellite and ground station.Statistics generated for each pass include tasks run,gigabytes downloaded, commands uploaded, satellitehealth, and ground station status.

Mission Control is written mostly in Python, andschedules approximately 350 passes per day on theUHF TT&C systems and 300 X-band passes per day.Mission Control is built by an in-house team of threesoftware engineers. We found that none of the ex-isting products fit our needs, and most were closed-source products. As we scale up our constellationto hundreds of satellites in space, open access to the

Colton 3 30th Annual AIAA/USUConference on Small Satellites

source code is necessary for new features and bugfixes.

FAULT DETECTION, NOTIFICATION,AND CLEARING

Planet’s ground stations use many open source pack-ages for fault detection. Often, these software pack-ages are designed for IT professionals for website andserver monitoring but can be applied to other scalablenetworks, such as ground stations.

Much of the fault detection and notification is builtaround an open source network monitoring service.This service runs on the ground station servers andcontinually monitors the health and status of thehardware and software. When a fault is detected, thelocal machine notifies the cloud-based central server,which then alerts our internal messaging system andan external incident notification service. The ser-vice is configured to notify the “on-call” engineer,handle acknowledgements, and escalate alerts whenneeded.

While the ground station team strives for 100 percentuptime, at any given time some ground stations canbe out of service due to equipment failure. It is muchless expensive to build for 90% uptime versus 100%uptime, so the network is purposely slightly undercapacity, and therefore more fault-tolerant. Satellitetasks are also robust to a single station failure; Mis-sion Control tracks acknowledged commands and willautomatically re-run incomplete or unacknowledgedcommands during the next pass. In addition, eachfailure gives us new insight into our next upgrades toimprove the network.

If a fault requires hardware to be adjusted or fixed,we do have “remote hands” contractors for each sitethat are able to work with us to diagnose and fixissues from afar.

In the event of power-loss to a site, the hardware is alldesigned to come back up automatically in a knowngood state and resume activities by polling MissionControl and rescheduling the next passes.

REMOTE TROUBLESHOOTING

While it is easiest to to address an issue directly on-site, visits take time. We have built tools to reducethat time and personnel cost by solving issues fromafar. This both allows us to clear some issues imme-diately and to provide instruction to our local helpor plan for when we do need to visit the site.

One of our most basic tools is a network-based powerswitch. From the switch, we can power-cycle com-ponents remotely and view current consumption. Itis instrumental for testing the system in various con-figurations. For example, a quick power cycle of theLNA in the system can tell us if it is still operat-ing as expected. If there is no appreciable differencein system performance, we can quickly narrow in oninvestigating a fault before or at the LNA.

The Planet Labs open source radio also gives us in-valuable tools in troubleshooting [10]. At the mostbasic level, we can poll the RF power received bythe TI CC1110 receiver. Combined with our abilityto remotely power parts of the receive and transmitchains, we can quickly test the functionality of indi-vidual components.

Using the RF power at the receiver, we can rotate an-tennas and collect measurements at all azimuth an-gles to get a noise-survey of a site. These surveyscan give us quick insights to how an antenna is per-forming. A normal site, as seen in Figure 5, will havevery uniform plots with a bit of variation in noiseas the antenna rotates. We expect variation at thelowest elevations, so absence of noise can be just asuseful an indicator as too much noise in any givendirection.

Figure 5: Example Noise Survey.

We have used these tools to identify local sourcesof interference that have hampered our link perfor-mance, and as an additional troubleshooting optionin the transmit chain. For example, at one site anamplifier was continuously biased. The problem man-ifested itself as increased noise at the site, impactingoperations. Figure 6 shows surveys from other anten-

Colton 4 30th Annual AIAA/USUConference on Small Satellites

nas at the site that we used to triangulate the sourceof the issue and subsequently fix it.

Figure 6: Faulty Amplifier Noise Survey

The noise survey program we wrote moves clockwisein azimuth before stepping in elevation and return-ing counter-clockwise in azimuth. Figure 7 shows amarked difference between the noise seen in clock-wise motion and counterclockwise motion. Revers-ing the direction of the program changed the pat-tern to reflect the new clockwise and counterclock-wise, enabling us to point our local contractor to aninternally-separated coaxial cable.

Figure 7: Motion-Dependent Fault.

Tracking with Multiple Systems

Tracking the same satellite with multiple systems ata single site is another very useful troubleshootingmethod. We can collect and store the receive poweron our X-band systems throughout a pass, then over-lay them at the end to get a rough comparison of thetwo systems. This also helps to differentiate betweenwhat anomalies are satellite specific, and what areground station specific. Co-tracking can also high-light minor errors in pointing; one of the systemsmay show a drop-off at the Time of Closest Approach(TCA), when the dish is required to move the fastestto keep up with the LEO satellite, and when precisepointing is most important.

The red and purple lines in Figure 8 show Carrier-to-noise-plus-interference ratios (CNI) on two nearbydishes tracking the same satellite. The dip in the reddish shows that it was not pointing as accurately atthe satellite during the middle of the pass. Link mar-gins for these same two dishes are shown with thegreen and blue lines. The step changes at the begin-ning and end of the pass are due to MODCOD/datarate changes.

Figure 8: Cotracking satellites.

Additionally with co-tracking, we can transmit com-mands up on one system and receive them via an-other system at the same site. This method veri-fies the entire link chain, and when combined withreceive power readings from the spacecraft and theground segment, can provide valuable informationabout both the receive and transmit chains of eachsystem. Lastly, if any portion of the chain is not func-tioning, it quickly becomes evident by co-trackingwith a known good site.

Colton 5 30th Annual AIAA/USUConference on Small Satellites

Tracking Known Objects

When nothing else is available, or when a system isstill in commissioning and not ready for active down-link, we can always fall back to using a known ob-ject, such as the sun, the moon, or one of the well-documented earth observation satellites such as Terraor Aqua.

In addition to monitoring RF performance using re-ceivers, we can monitor antenna movement, replaypast tracks, and check motion using remote camerasat each dish. The cameras are an easy way to get a setof eyes on the dish and equipment from afar and canwatch even when people cannot. The cameras givean independent check of hardware status and havehelped us with equipment errors more than once. Wecan try to be robust to failure, but troubleshootingand recovery methods are invaluable for getting op-erational quickly after an unforeseen problem. Beingagile on the ground is not about avoiding mistakes,rather it is about being able to quickly learn aboutproblems, gracefully recover, and institute proceduresto prevent future problems.

Building Tools for Troubleshooting

Since much of the equipment and software is the sameacross the network, we have built very standardizedtesting tools to automate tasks. Software is keptin centrally controlled repositories and can be re-deployed to any or all stations automatically, leavingus with a high degree of confidence about how eachsystem will perform and about what tools we can useat a site.

As much as possible, we design tools to be versatileand accept different configurations. Much of this isdone through abstraction layers. For example, inter-face code may reside in a library and be called uponby a service. Troubleshooting tools then call uponthe same library to run. By reusing production codein troubleshooting, we stay as close to the originalenvironment as possible. Additionally, since we useunix-like operating systems as our primary develop-ment, work, and production environments, we canuse many of the standard shell commands and nativetools to pipe outputs between our scripts or to expandupon the normal scope of the programs.

PREDICTIVE MODELING OFFUTURE NEED

Agile aerospace moves quickly; in an iterate-launchcycle, the longest lead time is often the launch. How-ever, licensing and building ground stations can belong-lead items. Simulations inform us on how muchcapacity to build and prevent us from over-building,a waste of both time and capital.

Orbit propagation tools generate both an estimate ofthe total number of Doves needed for our high ca-dence, whole-earth imagery, as well as the estimatedpass lengths over any given latitude. By knowing thetotal number of satellites needed and the data gen-erated by the Doves, we can estimate the necessarytotal contact time the ground station network mustsupply for any give location. New ground station lo-cations are then optimized for cost, data capacity,and location.

By understanding the operating cost with respect tothe data downlink capacity of each station, we canquickly generate a comparison cost to those providedby third party vendors. This calculation feeds backinto decisions regarding purchasing excess capacityeither in the case of a site failure or simply duringhigh-use time periods, like when commissioning newsatellites.

LESSONS LEARNED

Agile aerospace is a focus on iterative design to con-tinuously upgrade a product. While much of thehardware can be expensive and we do not have an“agile” dish, we can iterate around the expensivehardware and support it with an agile ground build.Control software and systems can improve over timewithout necessitating change of the larger hardwarecomponents. Additionally, external components canbe added to improve an already existing system. Forexample, we are currently deploying GPS time sourcesto many of our sites. Existing NTP time sources givesus adequate timing, but an upgrade to local stra-tum 1 time sources allows better orbit determination,link encryption, and positional knowledge onboardthe satellites. Time sources are just one example ofground systems that we can iterate and improve uponto make the system better as a whole.

Additionally, it is very helpful to keep a lab setup ofas much of the ground hardware as feasible. Not onlydoes this provide a better testing platform for the de-velopment of both the ground and the space systemsin the lab, it also provides fast access to a system for

Colton 6 30th Annual AIAA/USUConference on Small Satellites

investigating how things may have failed on a site.The lab is a far easier build and test environmentthan the field, so having the lab setup can also pro-vide information about how to streamline field instal-lations.

While we try to be agile, licensing and the physi-cal build can often take much longer than expected.We have found it very important to begin the licens-ing and contracting processes well before there is anyneed for the site. For that reason, modeling the fu-ture system requirements can be invaluable.

SUPPORTING AGILE AEROSPACEFROM THE GROUND

By focusing on design iteration, flexibility, and rapidchange we can emulate agile aerospace. Not everyaspect of a ground station can be agile, but by keep-ing those principles in mind, we can implement agileaerospace in other aspects of the network architec-ture. Software development and deployment can becentrally controlled and deployed to allow improve-ments as the system develops. Capacity can be mod-eled to plan coverage expansion as needed. Remotetroubleshooting tools can allow a central team to con-trol all aspects of a station from afar and manage anyissues that arise.

Planet Labs’ ground station network is built with theidea of easy expansion and extensibility. At everypoint, we reflect on what we can change to improvecapabilities while sustaining the system as a whole.The Planet Labs ground station network works dailyto support the on-orbit Flock and is ready to growand advance in parallel with our satellites as we scaleto a new image of the earth every day.

REFERENCES

[1] Planet Pulse. https://www.planet.com/pulse/illegal-gold-mine-encroaches-on-protected-rainforest/.

[2] Planet Pulse. https://www.planet.com/pulse/fort-mcmurray-wildfire/.

[3] Planet Pulse. https://www.planet.com/pulse/ecuador-earthquake-data-available-under-open-license/.

[4] Klofas, Bryan. “Planet Labs Ground StationNetwork.” Presented at the 13th Annual Cube-Sat Developers Workshop. San Luis Obispo, CA.April 2016.

[5] Matas, Attila. “ITU Radio Regulations Relatedto Small Satellites.” Presented at the 10th An-nual CubeSat Developers Workshop. San LuisObispo, CA. April 2013.

[6] Persaud, Sankar. “Regulatory matters concern-ing authorization of Small Sats: Common Mis-understandings about the FCC and Licensing.”Presented at the 2012 CubeSat Summer Work-shop. Logan, Utah. August 2012.

[7] FCC. “Guidance On Obtaining Licenses ForSmall Satellites.” Federal CommunicationsCommission Public Notice DA-13-445. March2013.

[8] Hallam, Henry. “Design and Results of thePlanet Labs Low-Speed Transceiver Radio.”Presented at the 2014 CubeSat Summer Work-shop. Logan, Utah. August 2014.

[9] Foster, Cyrus, Henry Hallam, James Ma-son. “Orbit Determination and Differential-DragControl of Planet Labs CubeSat Constellations.”arXiv:1509.03270v1. AAS 15-524.

[10] Hallam, Henry, Matt Ligon, Kiruthika Devaraj,Alex Ray, Ryan Kingsbury. “Open Source Low-Speed Transceiver Solution for CubeSats.” Pro-ceedings of the 13th Annual Summer Cube-Sat Developers’ Workshop. Logan, Utah. August2016.

Colton 7 30th Annual AIAA/USUConference on Small Satellites