The OOI Data Team continues to listen to data users’ feedback to refine and improve Data Explorer. Many of those improvements are reflected in the latest release of Data Explorer, version 1.2, which is now operational. Data Explorer was originally released in September 2020, and this latest version is the second round of improvements made by Axiom Data Science, working with the OOI Data Team.
This version makes more OOI data accessible online and brings new features for gliders and profilers. It is also now possible to search for cruise data in the tabular search interface. Once there, you can select specific cruises, see their data profiles, and have a three-dimensional view of where the samples were taken. Glider data are also now available online and searchable by time and location. Once you’ve identified a glider of interest, it is possible to map or plot the glider’s route and compare data collected with data from other sensors. With another click, you can compare sensor data with profiler information, then change the parameters on the screen to learn more.
All data can be downloaded in csv, GeoJSON, KML, and ShapeFile formats for future use.
Additionally, discrete sample data (chemical analyses of seawater collected during shipboard verification sampling) have been added. Water samples are collected during OOI cruises at multiple depths, and analyzed for oxygen (Winkler), chlorophyll-a fluorescence and pigment distribution, nitrate/nitrite, and potentially a full nutrient suite, total DIC (dissolved inorganic carbon) and total alkalinity, pH, and salinity. These data can be used to compare to in situ instrument data or CTD casts in order to ensure OOI data quality. It is now possible to use Data Explorer 1.2 to convert discrete dissolved oxygen sample data from milliliters to micromoles and create standard_name mapping for discrete sample data.
In response to users’ feedback, many defects found in version 1.1 have been fixed. A summary of all the new features and bug fixes is available in the release notes.
“Data Explorer is a tool that allows users to access, manipulate, and understand OOI data for use in their research and classroom,” said Jeff Glatstein, OOI Data Delivery Lead and Senior Manager of Cyberinfrastructure. “Users’ feedback has been—and will continue to be— extremely useful in refining Data Explorer to ensure it meets users’ need and expectations. We are holding regular open meetings as one way to ensure that we receive timely feedback and work with our users to meet their needs.”
An open OOI Town Hall previewing some of the new and special glider-related features was held on 24 August 2021, where user input was welcomed. A recording of the session is available here.Read More
On 24 August, OOI Data Lead Jeff Glatstein, Axiom Data Science Designer Brian Stone, and Axiom Data Science Coder Luke Campbell gave a preview of upcoming additions to Data Explorer that will help users access glider data. The presenters sought input from OOI’s user community to improve the platform to ensure it meets data users’ needs when it goes live in September 2021. You can see the demonstration of the upcoming Data Explorer changes in the video below and hear suggestions from OOI’s data users community.[embed]https://vimeo.com/592362218[/embed]
OOI is seeking input from its data users. All are welcome to attend and contribute to an OOI Data Users Town Hall: Special Glider Session. The Town Hall will take place on August 24 AT 3 PM Eastern. Simply click here to register. We look forward to hearing your ideas.
Since Data Explorer’s inaugural launch in October 2020, OOI has been working with users of Data Explorer to learn what features worked for them, which could be improved, and what could be added to optimize users’ experiences. A version update (1.2) to the Data Explorer is now under development for release in early September. Among the new features include enhancements to the display and user interaction with underwater gliders.
During an upcoming Data Users Town Hall, August 24 at 3 pm Eastern, the new beta features will be demonstrated with the goal of soliciting feedback and suggestions from glider experts to ensure the tool meets users’ needs.
Here is a brief summary of the features that will be reviewed:
1) visualizations of glider previews alongside static instrument previews
2) searchable map interface for visualizing and downloading glider and discrete cruise data
3) mapping interface for finding and visualizing glider and discrete sample profiles that are within range of the selected instrument
Please register, mark your calendar, and see you soon.
[media-caption path="https://oceanobservatories.org/wp-content/uploads/2021/03/Screen-Shot-2021-03-30-at-5.51.41-PM.png" link="#"]Researchers used echosounder data from the Oregon Offshore site of the Coastal Endurance Array to develop a new methodology that makes it easier to extract dominant patterns and trends.[/media-caption]The ocean is like a underwater cocktail party. Imagine, as a researcher, trying to follow a story someone is telling while other loud conversations are in the background of a recording. This phenomenon, known as the “Cocktail Party Problem,” has been studied since the 1950s (Cherry, 1953; McDermott, 2009). Oceanographers face this challenge in sorting through ocean acoustics data, with its mixture of echoes from acoustic signals sent out to probe the ocean.
Oceanographer Wu-Jung Lee and data scientist Valentina Staneva, at the University of Washington, teamed up to tackle the challenge in a multidisciplinary approach to analyze the vast amounts of data generated by echosounders on Ocean Observatories Initiative (OOI) arrays. Their findings were published in The Journal of the Acoustical Society of America, where they proposed a new methodology that uses machine learning to parse out noisy outliers from rich echosounder datasets and to summarize large volumes of data in a compact and efficient way.
This new methodology will help researchers use data from long time series and extract dominant patterns and trends in sonar echoes to allow for better interpretation of what is happening in the water column.
The ocean is highly dynamic and complex at the Oregon Offshore site of the OOI Coastal Endurance Array, where echosounder data from a cabled sonar were used in this paper. At this site, zooplankton migrate on a diurnal basis from a few hundred meters to the surface, wind-stress curl and offshore eddies interact with the coastal circulation, and a subsurface undercurrent moves poleward. The echosounder data offer opportunities to better understand the animals’ response to immediate environmental conditions and long-term trends. During the total eclipse of the Sun in August 2017, for example, echosounders captured the zooplankton’s reaction to the suddenly dimmed sunlight by moving upwards as if it was dusk time for them to swim toward the surface to feed (Barth et al, 2018).
Open access of echosounder datasets from the OOI arrays offers researchers the potential to study trends that occur over extended stretches of time or space. But commonly these rich datasets are underused because they require significant processing to parse out what is important from what is not.
Echosounders work by sending out pulses of sound waves that bounce off objects. Based on how long it takes for the reflected echo to come back to the sensor, researchers can determine the distance of the object. That data can be visualized as an echogram, an image similar to an ultrasound image of an unborn baby.
But unlike an ultrasound of a baby, when an undersea acoustic sensor records a signal, it may be a combination of signals from different sources. For example, the signal might be echoes bouncing off zooplankton or schools of fish.[caption id="attachment_20566" align="alignleft" width="350"] (A) Data used in this work were collected by a three-frequency echosounder installed on a Regional Cabled Array Shallow Profiler mooring hosting an underwater platform (200 m water depth) and profiler science pod located at the Oregon Offshore site of the OOI Coastal Endurance Array (red triangle). The symbols indicate the locations of all OOI echosounders installed along the coast of Oregon and Washington. (B) The transducers are integrated into the mooring platform (from left to right: 120, 200, and 38 kHz). The platform also hosts an instrumented profiler that traverses the water column above the echosounder from ~ 200 m to ~ 5m beneath the ocean’s surface. (Image credit: UW/NSF-OOI/WHOI-V15).[/caption]
“When the scatterers are of different size, they will reflect the sound at different frequencies with different strengths,” said Lee. “So, by looking at how strong an echo is at different frequencies, you will get an idea of the range of sizes that you are seeing in your echogram.”
Current echogram analysis commonly requires human judgement and physics-based models to separate the sources and obtain useful summary statistics. But for large volumes of data that span months or even years, that analysis can be too much for a person or small group of researchers to handle. Lee and Staneva’s new methodology utilizes machine learning algorithms to do this inspection automatically.
“Instead of having millions of pixels that you don’t know how to interpret, machine learning reduces the dataset to a few patterns that are easier to analyze,” said Staneva.
Machine learning ensures that the analysis will be data-driven and standardized, thus reducing the human bias and replicability challenges inherently present in manual approaches.
“That’s the really powerful part of this type of methodology,” said Lee. “To be able to go from the data-driven direction and say, what can we learn from this dataset if we do not know what may have happened in a particular location or time period.”
Lee and Staneva hope that by making the echosounder data and analytical methods open access, it will improve the democratization of data and make it more usable for everybody, even those who do not live by the ocean.
In the future, they plan to continue working together and use their new methodology to analyze the over 1000 days of echosounder data from the OOI Endurance Coastal and Regional Cabled Array region.
Lee, W-J and Staneva, V (2021).Compact representation of temporal processes in echosounder time series via matrix decomposition. Special Issue on Machine Learning in Acoustics. The Journal of the Acoustical Society of America.
Barth JA, Fram JP, et al. (2018). Warm Blobs, Low-Oxygen Events, and an Eclipse: The Ocean Observatories Initiative Endurance Array Captures Them All.Oceanography, Vol 31.
McDermott, J (2009). The Cocktail Party Problem.Current Biology, Vol 19, Issue 22.
Cherry EC (1953). Some Experiments on the Recognition of Speech, with One and Two Ears.The Journal of the Acoustical Society of America. Vol. 25, No.5.
OOI’s new data access and visualization tool, Data Explorer, has been operational for about six months now. During that time, OOI’s Development Team has been revising it to incorporate input from community users.
We’d like to give the OOI Community an opportunity to preview this next iteration and give us your thoughts. Please join OOI Data Lead Jeff Glatstein and members of the Data Explorer Development Team on 9 April 2021 at 2 pm Eastern. Register here. We will briefly show participants the revised tool and receive any feedback you may have. Our goal is to continually improve this tool to better meet your needs.
Look forward to seeing you in early April.Read More
To provide context and comparison for data collected by OOI instrumentation, OOI collects and disseminates data collected by shipboard underway sensors and from water samples from CTD casts. Shipboard underway data can be accessed by using username and password ‘guest’ on the OOI Alfresco Document Management System, organized by cruise. Each cruise folder contains a Ship Data folder in the format provided by the ship operators and a Water Sampling subfolder. The Water Sampling subfolder includes scanned and digitized versions of the CTD logs, as well as, discrete water sample analyses in the formats provided by the labs which conducted the analyses.[caption id="attachment_20259" align="alignleft" width="199"] Collecting water samples from the CTD rosette on the Pioneer 8 cruise aboard the R/V Neil Armstrong. ©WHOI.[/caption]
To make these data more easily accessible to the science community, we have developed a common template to provide a full set of discrete water sample data from a cruise. These “Discrete_Sample_Summary” spreadsheets include the details for each Niskin bottle fired on a CTD cast, the CTD instrument rosette data from the time of bottle closure, and the water sample data and quality flags based on World Ocean Circulation Experiment (WOCE) standards.
These CSV files with common data formats can easily be read and manipulated in MATLAB, Python, or other computing programs and languages. Because water analysis data are received at different times from different labs, these spreadsheets are updated as data become available. An accompanying README file contains version history, general notes, and a description of the quality flags. The original spreadsheets from labs, which may contain additional data and methodology, will also be posted.
An example of how to read and use this discrete sample data can be found in this Jupyter notebook. Discrete_Sample_Summary spreadsheets have been posted for the Regional Cabled Array cruises 6-10, the Coastal Endurance Array cruises 1-13, and the Global Irminger Sea Array cruises 1-6. We will continue to work on completing spreadsheets for past cruises as well as cruises going forward.[caption id="attachment_20261" align="aligncenter" width="640"] Comparison of dissolved oxygen data on the Washington Shelf Surface Mooring with water sampling data from Endurance Cruise 13. Data from Deployment 10 and Deployment 11 are plotted together, and overlap during 5-7 July.[/caption] Read More
As of today, accessing, visualizing, and integrating OOI data into research and classrooms is a whole lot easier. The Ocean Observatories Initiative launched its new data exploration tool – OOI Data Explorer version 1.0 on 5 October. Data Explorer allows users to search and download cabled, uncabled, and recovered data, compare datasets across regions and disciplines, generate and share custom data views, and download full data sets using ERDDAP.
The OOI Data Team worked with Axiom Data Science to develop a data exploration system that is both powerful and user friendly. Version 1.0 has already been beta tested over the past three months by a subset of OOI scientific data users. This group’s feedback has improved Data Explorer, making it ready for broader distribution and use.
A live demonstration of Data Explorer is scheduled for 21 October at 2 pm EDT. This timeframe will allow users to try their hands using Data Explorer and come to the demonstration ready with specific questions about accessing data or suggestions on functions that could further enhance the user experience. To reserve a spot at the live demo, please register here.
“Data Explorer version 1.0 is the culmination of a process of listening to OOI data users and responding by implementing easier, more efficient, and useful means to deliver OOI data to them,” explained Jeffrey Glatstein, head of OOI’s Cyberinfrastructure, who guided a team of data and visualization experts in developing the tool. “It is an excellent tool that has been refined over the test period, and which we will continue to refine it in response to users’ suggestions. While continued improvements will be made and more data continually added, Data Explorer version 1.0 offers a great new way to find, access, and use OOI data.”
To ensure that all OOI data users maintain access to data, OOI’s current data portal will remain accessible and functional until the foreseeable future. Once all data have been exported to Data Explorer, tested, and vetted, only then will a complete switch be made to this new innovative tool.
The developers caution that Data Explorer version 1.0 may still have a few bugs, and users are asked to report them so the tool can be improved.
Added Glatstein, “Data Explorer is really a community-driven tool and it will be exciting to see how we can refine and improve it to meet the needs of our community.”Read More