Remotely Fixing and Preventing Mooring Issues

Alex Franks’s job is a big one. He is charged with fixing various issues that occur on OOI moorings, while they are hundreds and sometimes even thousands of miles away in the ocean. As an Engineer II at Woods Hole Oceanographic Institution (WHOI), Franks is intimately familiar with the mooring system controller software, which allows him to troubleshoot and fix instrumentation problems on OOI moorings, regardless of their location.

Franks has been working with electronics for over a decade and solving OOI mooring-related challenges since 2015.  Many examples exist of his innovative solutions. In 2020, for example, the satellite Internet service that was being used to send data from OOI moorings to WHOI servers was no longer a viable solution. The WHOI team faced the task of either finding a replacement system, or working with the then-current system. One easily implementable solution was to move to transmitting data through OOI’s Iridium radio antennas full time. There were downsides to this solution, however.  It would allow no margin of error, would consume more power, and still not be able to send data from all the instruments.

Franks figured out a better solution that would both keep costs manageable and continue to meet timely data transmission goals by modifying the Iridium file transfer portion of the mooring software to accommodate a new data transfer scheme. The new scheme used a feature of the computer program rsync, a fast and versatile file copying tool, called “diff”. Instead of using rsync to communicate with shore servers and determine the “delta” or change between the new instrument data on the mooring and the instrument data files on the WHOI server, he used one of the mooring’s onboard computers as an intermediary server to generate “diff” files against (delineating old from new data). These files were then generated and stored, and sent over the Iridium connection. Using this new configuration, Franks succeeded in sending the entire dataset of all instruments on the mooring, except one that was sent at a reduced sample rate. While transmission times can vary with weather conditions, this newly configured system sends data to the server every 20 minutes every other hour, reducing transmission times from 1440 minutes per/day to about 240 minutes per day.

[media-caption path="/wp-content/uploads/2022/04/DSC_0639-copy.jpg" link="#"]Waves in the North Atlantic can get pretty large, which makes it hard to conduct research at sea, especially in winter.  The waves and wind in the Irminger Sea also create challenges for ocean observing equipment in the water there year-round. Credit: ©WHOI.[/media-caption]

Franks also found ways to remotely manage mooring issues caused by weather and sea state by modifying software that controls wind turbines. Wind turbines play a critical role on OOI moorings, providing power to recharge the main system batteries.  At the Irminger Sea Array, where the sun is absent for months at a time (the moorings also utilize solar panels), these wind turbines are critical. Prior to Franks’ software fix, human input was required to disable the turbines to prevent them from spinning while wave heights were too great. Franks modified the software used to control the spinning of the turbines to read environmental data from the buoy itself and make automated  decisions in real-time that previously had to be done manually. The system now changes its configuration based on a variety of sensor inputs, which make for more immediate decisions to ensure the continued safe operations of the turbines. The software modifications not only help mitigate heavy sea damage to the turbines but saves power, as well. The software detects when the air temperature is above freezing and turns off the precipitation sensor heaters, conserving energy when possible. The software also has fail-safes in place for high or low voltage and to determine hydrogen concentration levels inside the electronics. An illustration of this software configuration is provided below.

[media-caption path="/wp-content/uploads/2022/04/Mooring-system-software-upgrade.png" link="#"] This new software configuration detects when the air temperature is above freezing and turns off the precipitation sensor heaters, and has fail-safes in place for high or low voltage and for hydrogen concentration levels inside the electronics .Credit: ©WHOI.[/media-caption]

Franks has also developed software improvements to the power system controller inside the OOI surface moorings.  His work ran the gamut from disabling operational bugs in the system to reducing power consumption to fixing software errors to increase reliability. During a year-long deployment in the Irminger Sea, part of the power system controller board failed. Franks installed a software patch remotely that was able to limit the level of charge coming from wind turbines and wrote a fail-safe feature for the system to disconnect all charging sources if the voltage approached dangerous levels.

The challenges are what keeps Franks enthusiastic about his job, “I just love trying to figure out a solution and it’s particularly rewarding to be able to remotely resolve issues with equipment deployed in the open ocean.”





Read More

Improving Remote System Response in Increasingly Hostile Oceans

Wind and Waves and Hydrogen, Oh My!

Improving remote system response in increasingly hostile oceans

This article is a continuation of a series about OOI Surface Moorings. In this article, OOI Integration Engineer Alexander Franks discusses the mooring software and details some of the challenges the buoy system controller code has been written to overcome.

Components of the OOI buoys working in concert make up a system that is designed for deployment in some of the most challenging areas of our world’s oceans. These systems collect valuable scientific data and send it back to Wood Hole Oceanographic Institution (WHOI) servers in near real time. Mechanical riser pieces (wire rope, and/or stretch hoses) moor the buoy to the bottom of the ocean. Foam flotation keeps the buoy above water in even the worst 100-year storm, while its masthead supports instrumentation and satellite radios that make possible the continuous relaying of data. The software controlling the system is just as important as the physical aspects that keep the system operating.

The system software has a variety of responsibilities, including setting instrument configurations and logging data, executing power schedules for instruments and parts of the mooring electronics, controlling the telemetry system, interfacing with lower-level systems including the power system controller, and distributing GPS and timing. The telemetry system is a two-way communication path, so the software controls data delivery from the buoy, but also provides operators with the ability to perform remote command and control.

[caption id="attachment_22938" align="alignnone" width="745"] Software flow diagram created by OOI Integration Engineer Alex Franks[/caption]

The unforgiving environment and long duration deployments of OOI moorings lead to occasional system issues that require intervention. Huge storms, for example, can build waves so high that they threaten wind turbines on the moorings. At the Irminger Sea Array, ice can accumulate so much as to drastically increase the weight of the masthead, and with subsequent buoy motion, risk dunking the masthead and instruments. Other mooring functions require constant attention. The charging system must be monitored to ensure system voltages stay at safe levels and hydrogen generation within the buoy itself is kept within safe limits. Two-way satellite communication allows operators to handle decision making from shore using the most up-to-date information from the buoy.

“Since starting in 2015 and following multiple mooring builds and deployments, I’ve realized that issues can rapidly arise at any time of the day or night. I started thinking about what the buoys can do for themselves, using the data being collected onboard,” Franks said.

One of the game-changing upgrades implemented by Franks was to read environmental data and make automated buoy safety decisions in real-time that were previously performed by the team manually. For example, previously, the team would need to monitor weather forecasts and decide preemptively whether changes to buoy operations were advisable. With recent software changes, the system can now change its configuration based on a variety of sensor inputs. These variables include system voltage, ambient temperature, hydrogen levels inside the buoy well, wind speed, and buoy motion (for sea state approximation). In addition to the software updates, the engineering team redesigned the power system controller. They added charge control circuits and the ability to stop the wind turbines from spinning. The software and electrical upgrades now provide redundant automated safeguards against overcharging situations, hydrogen generation, and turbine damage, maximizing buoy operability in harsh environments.

[caption id="attachment_22946" align="alignleft" width="650"] Onshore engineers are able to keep track of Irminger Sea buoys and instrumentation on this new new dashboard.[/caption]

With a largely independent system, operators also needed a way to easily monitor status of the buoys and instrumentation. The software team created a new shoreside dashboard that allows operators to set up custom alerts and alarms based on variables being collected and telemetered by the buoy. While the buoy systems can now operate autonomously, alerts and alarms maintain a human-in-the-loop component to ensure quality control.

As operations and management of the moorings have progressed, the operations team has found opportunities to fine tune how operators and the system handle edge cases of how the system responds to hardware failures and extreme weather.  In the past, sometimes conditions changed faster than the data being transmitted back to shore. This new sophisticated software automates some of the buoy’s responses to changing conditions in real time, which helps to ensure their continued operation even under challenging conditions. The decreased response time to environmental and system events using an automated system, coupled with the ability to monitor and interact remotely, has increased the reliability and survivability of OOI moorings.

Read More

Irminger Sea Array Overcomes Challenging Conditions to Provide Climate Insights

Deployed 140 miles east of the southern tip of Greenland and three miles south of the Arctic Circle, the Irminger Sea surface mooring floats on a cold empty sea named for a Danish naval admiral few people have heard of, in a location that few people could point to on a North Atlantic chart. The Irminger Sea is delineated less by coastlines or geographic basins and more by what is taking place within the deep ocean here, processes only visible with the aid of deep-sea instruments. To oceanographers and climate scientists the region is a confluence of ocean currents where heat carried from the topics gets extracted and cold water sinks into abyss like few other places worldwide and with climate-changing impacts.

Like most high-latitude oceans, storms are frequent and strong. Some storms migrate northeast from the mid-latitudes. Other storms are born here and then mature to impressively violent conditions influenced by the distant high mountains and massive Greenland icecap. Gale-force winds and steep-faced ocean waves spread east over a wide cone from the tip of Cape Farewell. The ice pack around Greenland ejects icebergs, some washing far out to sea where they threaten vessels. Cold air and sea spray build layers of heavy ice on exposed surfaces and instrument sensors. Other oceans can be found with higher waves, some have colder weather, but in few places do storms intensify so quickly, occur as often, and happen in a place so vital to planetary climate. Right where the storm forces are the strongest is also the perfect place for a tower packed with weather instruments.

The Irminger Sea mooring is designed to collect data in this stormy world where meteorological and ocean measurements, especially at the surface, are rare and hard to sustain. The mooring is recovered and a new one put in its place once a year, typically during the short summer month of July when weather conditions are calmest. At more than 4 meters high, the surface mooring tower is heavily instrumented with meteorological sensors and communication antennas, and the surface float is filled with data loggers and redundant computing elements and controllers that collect, store, and transmit data to shore. In total about four tons of floating equipment is anchored to the bottom by a 1.5-mile cable studded with dozens of instruments sampling the deep interior of this sea. To power everything, the buoy float is packed with rechargeable batteries, fueled by solar panels and wind turbines on the buoy tower. Strong winds are usually welcome because they rapidly re-charge the battery packs. Sometimes, however, these can be too much of a good thing.

[media-caption path="/wp-content/uploads/2021/12/Irminger-storm-waves.png" link="#"]Storm waves captured by the Irminger Sea tower camera #5 on 2019-03-19 at 09:01:00 UTC during a typical bad weather day.  Observations made by the WAVSS instrument (from 09:00 to 09:20 UTC) report significant wave heights around 5 m (16 ft) and maximum heights of 21 m (69 ft).  Records from a second accelerometer (MOPAK) report a wave ~5 m high passing at 09:01 UTC, possibly the same one in this photo.  About half-a minute later, at 09:01:30 UTC, the MOPAK recorded a wave >20 m (no image).[/media-caption]

A recent storm during October 18-19, 2021, was one such time. The mooring was battered by wind speeds exceeding 35 knots (gale force) for almost 24 hours, with some topping out above 50 knots. Heavy storm seas built up and stacked upon themselves for hours. At the storm’s peak, about one third of the highest waves were above 15 m (49 ft). Picture heaving an 8000 lb. surface mooring 80 ft up and down on a tilt-a-whirl ride that never stops. Waves this high can bring tons of water crashing down.  Towering waves were recorded, some reaching up to 20-25 m (66-82 ft), so high they approached the limits of our instruments.

The Irminger Sea continues to test our ability to “weather harden” instruments in stormy parts of the world. From November to March, daylight is fleeting, the sun hovers near the horizon and solar panels trickle out only a few milliamps. For the next few months, many of the instruments in the ocean interior and on the tower will continue to sample, each instrument powered by its own small battery. The Irminger surface mooring will communicate once each day, a tiny burst of data with vital signs, until spring returns and the sun revives the cold battery packs.

[media-caption path="/wp-content/uploads/2021/12/Ice-near-mooring-.png" link="#"]Ice near the Irminger Sea mooring 2019-04-02. Credit: @WHOI, Peter Brickley.[/media-caption]

The 2021 storm demonstrated, yet again, the challenges of working in the Irminger Sea. Yet, it also demonstrated the remarkable robustness of the OOI moorings in such extreme conditions. Ocean and meteorological measurements gathered by the Irminger Sea mooring during such storm events are extremely valuable for understanding oceanography and climate processes. Equally important is the invaluable experience gained that will drive continued improvement in the accuracy and durability of instruments deployed under such extreme conditions, with consequent increases in knowledge.

Written by Peter J. Brickley, PhD, Senior Engineer, AOPE Dept., Woods Hole Oceanographic Institution and OOI’s Coastal and Global Scale Nodes Observatory Operations Lead



Read More

Irminger Array Successfully Turned 8th Time

The Irminger 8 Team successfully wrapped up the eighth turn of the Global Irminger Sea Array on 26 August when the R/V Neil Armstrong docked in Reykjavik, Iceland. After a few days of demobilization, the 10 members of the science party were free to head home after showing proof of a negative COVID test 72 hours before boarding a flight back to the U.S.

Chief Scientist John Lund led the science party of 10 in completing all of the expedition’s objectives. Over the course of 26 days at sea, they recovered four moorings and deployed four new moorings in their place. The team also deployed three gliders—two Open Ocean and one Profiling—and recovered a glider that had been in the water since 2020 and whose battery supply was rapidly depleting.

[media-caption path=”” link=”#”]The Irminger Sea presents challenges of high winds, strong waves, and icebergs as shown here with the R/V Neil Armstrong in the foreground. Credit: drone video, Croy Carlin SSSG. [/media-caption]

One highlight of the trip was engaging in scientific outreach with a class of fourth graders. The team connected with the students while out on the open ocean via Zoom. The oceanographers aboard the ship each had a chance to share what it’s like being on an oceanographic voyage and explain the purpose of the different instruments and sensors on the arrays. Another highlight of the expedition was the OOI team’s ongoing collaboration with OSNAP (Overturning in the Subpolar North Atlantic Program). While OSNAP participants were not onboard the Armstrong as in the past, their shore-based presence was clearly in evidence.  Expert hydrographer Leah McRaven worked with the onboard team to adjust CTD (Conductivity, temperature, depth) sampling to ensure that new CTD equipment was calibrated and sampling properly.

The science team also added a novel twist to the regular shipboard sampling that supports field calibration and validation of the platforms and sensors in the arrays. During Irminger 8, the shipboard team worked with OOI’s onshore data team to make collected CTD data available online in near real-time. As an added bonus, McRaven shared her insights about CTD sampling in regular blog posts here.

The Irminger 8 Team took full advantage of being in this critical ocean region, which is sensitive to climate change. During transit from Woods Hole to the array, off the southeast coast of Greenland, the team deployed surface drifters and ARGO floats for the Greenland Freshwater Project, which is studying the impact of freshwater runoff from Greenland’s melting ice sheet on the North Atlantic and Arctic climate. The team also deployed a biogeochemical ARGO float for the Global Ocean Biogeochemistry Project, and took a series of CTD casts on behalf of OSNAP, to add to long term data collection efforts in this critical region. In addition, the team deployed two RAFOS floats for the Madagascar Basin Project to measure deep water circulation and 15 Sofar Spotter buoys to measure wind, wave, and temperature data.

“In the ideal, science is a collaborative process,” said Chief Scientist John Lund. “During transit time to and from the array, we were able to help our scientific partners get their equipment in the water. The data provided will help advance understanding of this critically important region, which is equally difficult to sample. The region has high winds, large, steep waves, strong currents, icebergs, and consequent equipment icing.”

Given the challenges of the ocean environment at these latitudes, the eighth turn of Irminger Array included equipment improvements. The newly deployed surface moorings included wind turbine modifications to help it withstand strong, volatile winds, and it also incorporated other structural modifications to strengthen the mooring, while easing refurbishment. Similarly, design modifications were made to the subsurface moorings to help ensure consistent, long-term data collection.

The team experienced some of these challenges of high winds and strong waves while on the cruise, but the rough conditions were compensated by the gorgeous scenery of the region. Added Lund, “One afternoon, the sun came out as the ship transited further up Prince Christian Sound. Everyone was awed by the beauty of the landscape. We saw glaciers, icebergs and the occasional whale.”

Prior to leaving the Sound, the team secured all the items for the transit to Reykjavik, the demobilization of the ship, and finally the journey home to Woods Hole.




Read More

CGSN Webinar 15 September

An informational webinar on the OOI Coastal and Global Scale Nodes (CGSN) will be presented on 15 September 2021 from 3:00-4:00 pm EDT. A presentation by the CGSN Team will be followed by a Q&A session.

Topics covered will include infrastructure making up the CGSN Arrays (Coastal Pioneer, Global Irminger Sea, and Global Station Papa), the current status of deployments, how to access near real-time data, and how to engage with the OOI.

Register here so you don’t miss out on the opportunity to meet the OOI CGSN Team and learn how you can work together.



Read More