Home :: Infrastructure Status :: OOI Critical Metadata Review

OOI Critical Metadata Review

The OOI strives to provide quality datasets to the public that meet QA/QC standards set by the program and that are expected by the user community. To that end, the OOI Marine Implementing Organization (MIO) data teams have undertaken a thorough review of all “critical” metadata in the system, including instrument calibration coefficients, instrument deployment assignments, and deployment dates. The validation of these metadata is essential to ensuring that data products are properly calculated. Because the OOI Data Portal operates on a process-on-demand model, users should check to see if and when changes have been made to metadata to ensure that previously downloaded data are correct, or to re-download corrected data. This initial phase of the metadata review focused on verifying calibration coefficients and was completed at the end of December 2019. We are continuing to verify the historical deployment assignments/dates, and the results will be updated accordingly.

The audit was conducted via a combination of human-in-the-loop (HITL) processes and automated script development. The HITL process ensured that two sets of eyes verified each metadata product, whenever possible, while the scripts performed automated verification and generation of reports to pass back into the HITL workflow. Finally, the results were used to aid the evaluation of current processes and guide in adapting workflows to improve QA/QC efforts and communication to users.

As part of this review process, the widget below provides a list of instrument deployments for which calibration metadata have been corrected and determined to affect the calculated Data Products. This list also includes instruments (e.g., nitrate sensors) that have dependencies on data products (e.g., temperature, salinity, or pressure from a CTD) recorded by affected instruments.

The information in the widget can be used to help locate and download a table of relevant data-impacting changes.

OOI Metadata Widget

Expand the Metadata Change Log to see the fields included for each critical metadata change:

OOI Metadata Change Log Format (Expand)

  1. Array: name of the OOI Array (e.g., Pioneer, Endurance, Cabled, Global) that was affected;
  2. Platform: name of the mooring or seafloor installation (e.g., Axial Base Seafloor) that hosts the affected instrument;
  3. Node: name of the junction box or mobile platform (e.g., profiler or glider) that hosts the affected instrument;
  4. Instrument: name of the instrument (e.g., CTD) that was affected;
  5. RefDes: the OOI reference designator that encodes the array, platform, node, and instrument information (e.g., RS03AXBS-LJ03A-12-CTDPFB301). This code can be used to search the system via OOINet or the machine-to-machine (M2M) API to quickly find and download the affected data;
  6. Asset ID: the OOI unique ID number (UID) assigned to the affected instrument. The UID is used in the OOI Asset Management system to track specific instruments and to name calibration sheets, so it can be used to search for the modified file;
  7. Serial Number: the vendor serial number assigned to the affected instrument ;
  8. Deployment: the sequential deployment number of the affected instrument – a deployment is the period of time between the instrument going in the water until it is recovered. The deployment number can be used with the reference designator and UID to determine the time range of data affected by the calibration change;
  9. GitHub changeDate: the date when the critical metadata change was entered into the Asset Management GitHub repository;
  10. file: the name of the OOI calibration .csv file that was changed, based on the UID and vendor calibration date. The entry in the sheet reflects the name of the file as it exists now on GitHub. The original name and the change should show up in the file history;
  11. URL: a direct link to the altered calibration .csv file in the OOI Asset Management GitHub repository;
  12. changeType: the category of data affecting change that was made to the critical metadata (‘calibration coefficient resolution changed’, ‘missing file added’, or ‘file renamed’). These categories are used to generate annotations that are applied to data in the system;
  13. dateRangeStart: start of the relevant time range covered by the changed metadata;
  14. dateRangeEnd: end of the time range covered by the changed metadata; and
  15. annotation: text of the system annotation that will be applied to the science data stream for the instrument that had a critical metadata change. These annotations can be included in all future OOINet and M2M data requests, and are the primary means of communicating data issues and corrections to users.

If you have questions about the above information, datasets, or the metadata review process, please contact the HelpDesk at help@oceanobservatories.org.