saeon_netcdf_guidance

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
saeon_netcdf_guidance [2022/07/28 12:19] – lindsaysaeon_netcdf_guidance [2022/07/28 12:23] (current) – lindsay
Line 3: Line 3:
 The SAEON Open Data Platform allows for the storage, discoverability and download of multidimensional scientific data. Network Common Data Format (netCDF) is a file format designed for storing multidimensional data in the form of arrays, and is widely used in the atmospheric and oceanographic communities. The netCDF library has been developed by the [[https://www.unidata.ucar.edu/software/netcdf/|Unidata program]] and is freely available. NetCDFs are self-describing, so it is essential that certain conventions are followed when creating these files. SAEON follows the requirements set out by the [[https://cfconventions.org/|Climate and Forecast]] (CF) and [[https://wiki.esipfed.org/Category:Attribute_Conventions_Dataset_Discovery|Attribute Conventions Dataset Discovery]] (ACDD) conventions for netCDF metadata and data. NetCDF standardisation also promotes interoperability and allows us to add tools such as THREDDS data services to the data. The main components of a netCDF that should be standardized are dimensions, variables and attributes. The SAEON Open Data Platform allows for the storage, discoverability and download of multidimensional scientific data. Network Common Data Format (netCDF) is a file format designed for storing multidimensional data in the form of arrays, and is widely used in the atmospheric and oceanographic communities. The netCDF library has been developed by the [[https://www.unidata.ucar.edu/software/netcdf/|Unidata program]] and is freely available. NetCDFs are self-describing, so it is essential that certain conventions are followed when creating these files. SAEON follows the requirements set out by the [[https://cfconventions.org/|Climate and Forecast]] (CF) and [[https://wiki.esipfed.org/Category:Attribute_Conventions_Dataset_Discovery|Attribute Conventions Dataset Discovery]] (ACDD) conventions for netCDF metadata and data. NetCDF standardisation also promotes interoperability and allows us to add tools such as THREDDS data services to the data. The main components of a netCDF that should be standardized are dimensions, variables and attributes.
  
-===== NetCDF File Structure =====+==== NetCDF File Structure ====
  
 There are certain types of discrete sampling in atmospheric/oceanographic research, like point, time series, trajectory or profile, and each of these feature types have defined netCDF file structures. The US National Oceanographic Data Centre has developed [[https://www.nodc.noaa.gov/data/formats/netcdf/v1.1/|feature type templates and examples]], which are accepted widely for netCDF standardisation. Most importantly, each feature type template highlights the relationships set out for netCDF dimensions and variables. There are certain types of discrete sampling in atmospheric/oceanographic research, like point, time series, trajectory or profile, and each of these feature types have defined netCDF file structures. The US National Oceanographic Data Centre has developed [[https://www.nodc.noaa.gov/data/formats/netcdf/v1.1/|feature type templates and examples]], which are accepted widely for netCDF standardisation. Most importantly, each feature type template highlights the relationships set out for netCDF dimensions and variables.
  
-==== Global Attributes ====+=== Global Attributes ===
  
 This section contains metadata that pertains to the overall netCDF file. The following table contains global attributes derived from CF and ACDD conventions required by the SAEON ODP for multidimensional data in netCDF format. For additional resources you can refer directly to the official documentations of each convention. This section contains metadata that pertains to the overall netCDF file. The following table contains global attributes derived from CF and ACDD conventions required by the SAEON ODP for multidimensional data in netCDF format. For additional resources you can refer directly to the official documentations of each convention.
Line 52: Line 52:
 | lineage (RECOMMENDED)  | “The CTD raw data was converted to scientific units using SBEDataProcessing software (ver 7.26.7.114). The following modules (SBEDataProcessing) were then run on the converted data using the recommended default values: Align CTD, Filter, Loop Edit and Cell Thermal Mass. The data was then split into the up cast and the down cast. The downcast data was then bin averaged.”  | Information about how the data has been produced and processed, modified.  | | lineage (RECOMMENDED)  | “The CTD raw data was converted to scientific units using SBEDataProcessing software (ver 7.26.7.114). The following modules (SBEDataProcessing) were then run on the converted data using the recommended default values: Align CTD, Filter, Loop Edit and Cell Thermal Mass. The data was then split into the up cast and the down cast. The downcast data was then bin averaged.”  | Information about how the data has been produced and processed, modified.  |
  
-==== Dimensions ====+=== Dimensions ===
  
 NetCDF dimensions define the shape, grid or coordinate system of a variable. A dimension has both a name and a length. A dimension length is an arbitrary positive integer, except for cases where one dimension, at most, is UNLIMITED and can grow along that dimension. Once the data feature type has been identified, the US National Oceanographic Data Center [[https://www.nodc.noaa.gov/data/formats/netcdf/v1.1/|netCDF templates]] can be used as a guide for which dimensions to define in your file. NetCDF dimensions define the shape, grid or coordinate system of a variable. A dimension has both a name and a length. A dimension length is an arbitrary positive integer, except for cases where one dimension, at most, is UNLIMITED and can grow along that dimension. Once the data feature type has been identified, the US National Oceanographic Data Center [[https://www.nodc.noaa.gov/data/formats/netcdf/v1.1/|netCDF templates]] can be used as a guide for which dimensions to define in your file.
Line 60: Line 60:
 Something to keep in mind in terms of naming dimensions and variables is the netCDF coordinate variable. This is when a variable has the exact same name as a dimension and is most commonly used for coordinates such as lat, lon, depth and/or time. This may not necessarily have special meaning to the netCDF library, however visualisation software treats this in a special way. Something to keep in mind in terms of naming dimensions and variables is the netCDF coordinate variable. This is when a variable has the exact same name as a dimension and is most commonly used for coordinates such as lat, lon, depth and/or time. This may not necessarily have special meaning to the netCDF library, however visualisation software treats this in a special way.
  
-==== Variables ====+=== Variables ===
  
 Variables in a netCDF contain the parameters measured by an instrument. Variables in netCDF files can be one of six types (char, byte, short, int, float or double). Each variable has a data type, name and a shape which is defined by its specified dimensions. Variables in a netCDF contain the parameters measured by an instrument. Variables in netCDF files can be one of six types (char, byte, short, int, float or double). Each variable has a data type, name and a shape which is defined by its specified dimensions.
Line 70: Line 70:
 Note: The //long_name// attribute is defined to contain a long descriptive name of the variable which may, for example, be used for labelling plots. The //standard_name// attribute is the name used to identify the physical quantity and must be taken from the [[https://cfconventions.org/standard-names.html|CF standard name table]]. Parameters with no suitable standard_name should be described using the long_name attribute only. Note: The //long_name// attribute is defined to contain a long descriptive name of the variable which may, for example, be used for labelling plots. The //standard_name// attribute is the name used to identify the physical quantity and must be taken from the [[https://cfconventions.org/standard-names.html|CF standard name table]]. Parameters with no suitable standard_name should be described using the long_name attribute only.
  
-==== Variable Attributes ====+=== Variable Attributes ===
  
-=== Time coordinates ===+== Time coordinates ==
  
 Time data in a netCDF is represented as an integer representing an interval from some reference time. Time variables must include a clear units attribute, as there is no default value. The units attribute for a time variable is a string value in the format recommended by UDUNITS. Commonly these strings include “//days//”, “//hours//”, “//minutes//” or “//seconds//” //since// a specific date, time and time zone (e.g. “seconds since 1970-01-01 00:00:00 UTC”). Use Double data type. Time data in a netCDF is represented as an integer representing an interval from some reference time. Time variables must include a clear units attribute, as there is no default value. The units attribute for a time variable is a string value in the format recommended by UDUNITS. Commonly these strings include “//days//”, “//hours//”, “//minutes//” or “//seconds//” //since// a specific date, time and time zone (e.g. “seconds since 1970-01-01 00:00:00 UTC”). Use Double data type.
Line 90: Line 90:
 | comment (RECOMMENDED) | “....” | Miscellaneous information about the data, that cannot be described in any of the other available attributes. | | comment (RECOMMENDED) | “....” | Miscellaneous information about the data, that cannot be described in any of the other available attributes. |
  
-=== Horizontal coordinates ===+== Horizontal coordinates ==
  
 SAEON ODP uses the WGS84 coordinate reference system in decimal degrees when describing latitude and longitude. SAEON ODP uses the WGS84 coordinate reference system in decimal degrees when describing latitude and longitude.
Line 107: Line 107:
 | comment (RECOMMENDED) | “....” | Miscellaneous information about the data, that cannot be described in any of the other available attributes. | | comment (RECOMMENDED) | “....” | Miscellaneous information about the data, that cannot be described in any of the other available attributes. |
  
-=== Vertical coordinates ===+== Vertical coordinates ==
  
 Depth or height variables should be measured in SI units, most commonly “metres”. Pressure measurements should not be labelled as a depth or height variable, but as a separate parameter. Depth or height variables should be measured in SI units, most commonly “metres”. Pressure measurements should not be labelled as a depth or height variable, but as a separate parameter.
Line 125: Line 125:
 | comment (RECOMMENDED) | “....” | Miscellaneous information about the data, that cannot be described in any of the other available attributes. | | comment (RECOMMENDED) | “....” | Miscellaneous information about the data, that cannot be described in any of the other available attributes. |
  
-=== Geophysical parameters ===+== Geophysical parameters ==
  
 These variables contain the data collected by an instrument or sensor. These variables contain the data collected by an instrument or sensor.
Line 147: Line 147:
 | comment (RECOMMENDED) | “....” | Miscellaneous information about the data, that cannot be described in any of the other available attributes. | | comment (RECOMMENDED) | “....” | Miscellaneous information about the data, that cannot be described in any of the other available attributes. |
  
-=== Quality control flags (CONDITIONAL) ===+== Quality control flags (CONDITIONAL) ==
  
 If applicable to the platform, quality control flags can be self describing variables that show some sort of assessment to identify possible errors in the data. If applicable to the platform, quality control flags can be self describing variables that show some sort of assessment to identify possible errors in the data.
Line 163: Line 163:
 | comment (RECOMMENDED) | “....” | Miscellaneous information about the data, that cannot be described in any of the other available attributes. | | comment (RECOMMENDED) | “....” | Miscellaneous information about the data, that cannot be described in any of the other available attributes. |
  
-==== NetCDF manipulation tools ====+=== NetCDF manipulation tools ===
  
   * [[https://downloads.unidata.ucar.edu/netcdf/|NetCDF library software]]: //ncdump// can convert a netCDF binary file to CDL text.   * [[https://downloads.unidata.ucar.edu/netcdf/|NetCDF library software]]: //ncdump// can convert a netCDF binary file to CDL text.
  • saeon_netcdf_guidance.1659010795.txt.gz
  • Last modified: 2022/07/28 12:19
  • by lindsay