From Ocean Teacher Library
Formats for Real-time Data ExchangeContents |
Background
Virtually every instrument making observations produces data with a unique content and structure. In order for these to be usable, the data must be converted to more general formats. Unfortunately there is no single standard for data delivery although there has been some progress in recent years to converge on netCDF. However, the most well controlled deliveries are those associated with real-time data.
Most oceanographic data delivered in real-time are reported through the Global Telecommunications System (GTS) operated by the World Meteorological Organization (WMO). There are two types of data structures for reporting data.
The older form is called Traditional Alphanumeric Codes (TACs). These are character-based forms, whose structures are rigidly set and maintained by WMO committees. They have been in operation for decades. A comprehensive list of these is provided by WMO (see Manual on Codes, Volume I.1 )
Although TACs are relatively easy to learn, they are fairly inflexible to changing requirements, and place a heavy maintenance burden on encode and decode software.
TACs are being phased out by WMO in favour of Table Driven Codes (TDCs). The earliest TDC version, called Binary Universal Form for the Representation of meteorological data (BUFR- see Manual on Codes, Volume I.2 ) is based on a number of tables listing variables and setting the number of bits used to send the information. A character form, called CREX, was developed later (See the same Manual on Codes).
The advantage of BUFR is that a simple addition to a BUFR table allows new variables to be reported. This is a much simpler and flexible process than available for TACs. As well, BUFR does not require any particular ordering of data. It has been designed so that the first part of a BUFR message explains what data and in what order they are presented in the next part of the message. This property makes BUFR a self describing format and is an important aspect to its power. The main drawback to BUFR, though less so since computers are virtually always used in data handling, is that data are reported in binary.
It is becoming popular to report data in real-time through the Internet by placing updates on web sites. Other technologies also exist that allow data to be sent through a subscription-like service, or for users to regularly pull data from a site where data are made available. Data made available this way are often available only to restricted audiences or the availability is just not widely known. In a redevelopment of its data transmission systems, the WMO Information System (WIS) will support GTS-like operations for time-critical data and also support request-reply operations for data that have less time-critical characteristics.
Data Submission Pathways and Protocols
The GTS
The Global Telecommunications System (GTS) is the communications network operated by national meteorological services of countries. The rules and regulations of its operation are governed by the World Meteorological Organization (WMO) located in Geneva. It is the network by which most of the meteorological data that are collected by nations are exchanged between countries around the world. It is the chief source of data that are used in the national weather prediction models operated by country’s meteorological services. Through agreements with the Intergovernmental Oceanographic Commission, oceanographic data are also exchanged on the GTS
How the GTS works
Data are written into agreed data formats to form GTS messages. Messages are bundled singly or multiply into bulletins. These have a prescribed structure that must be met. For data transmitted in character codes, the structure of the bulletin conveys some information about what kind of data are contained inside, and from what region of the world the data originate. Data sent in binary form travel on the GTS under different bulletins that convey different information from the bulletins carrying character codes. For oceanographers, it is simplest to talk to a staff member of your local meteorological service to get advice. If this route is unavailable, contact with JCOMMOPS can be made (select “Contacts” in the right hand menu). Readers can get more information about JCOMM through the link provided below.
Observations that should or can go to the GTS
Oceanographic data may be sent in a more limited set of older character code forms and a very limited number of variables are handled. The table driven code forms permit more variables to be sent. The variables that can be sent are those that are listed in the BUFR tables in the Manual on Codes. Briefly these include temperature, salinity and current profiles, wave spectral data, along track measurements of temperature, salinity and currents, and surface observations of temperature, salinity, and some meteorological variables. BUFR permits a slightly expanded set of variables. A selection of these appear in BUFR templates (standardized data structures that use BUFR rules). If there are any questions of what type of data can be handled on the GTS, consult JCOMMOPS for advice (see link above).
Timeliness of observations for the GTS
For oceanographic purposes, there is an agreement that observations up to 30 days old can go onto the GTS. This delay period represents a common understanding of the time utility of observations contributing to real-time operations. In recent years, there has been both an emphasis on getting data distributed more quickly and success in doing this. The ideal is to provide the data as soon as they are available.
Who to contact
If you are intending to transform observational data into either TACs or TDCs, you will need to make contact with your country's national meteorological service to find out how to move the data to them, to determine what help they can provide in transforming your data into GTS compliant forms, to verify that your messages were built correctly, to get their advice on what bulletins should contain the data, and perhaps other considerations as well. If you don't know who to contact, you can always start with the JCOMM secretariat (see contact address) or with JCOMMOPS..
Data quality
Because of time constraint, it is seldom possible to carry out instrument calibrations, corrections to times for clock errors, or more than the most rudimentary position checks.
In most of the TACs, there is no way to indicate the quality of the data being distributed on the GTS. Usually, if quality control is carried out, observations that fail the tests are removed from the data stream going to the GTS.
If TDCs forms are used, it is possible to send both the observations and quality indicators. Data providers may choose to remove measurements that fail tests or simply set a quality flag indicating their poor quality, but send the complete set of measurements made.
Whether reporting is permitted or not, data providers are encouraged to do a minimal amount of checking to remove at least the most egregious errors. Internationally organized systems for certain types of data management have recommended procedures and these will be noted in appropriate documents.
Moving data to the GTS
Getting data ashore
The data should come from the offshore platform as quickly as possible. In some cases, the data come ashore through a telecommunications system such as Service Argos, Inmarsat, or perhaps a satellite-based internet connection and a transfer protocol like email or ftp.
For oceanographic data, it may be more convenient or acceptable for a cruise of short enough duration to bring all of the data ashore at the end of the cruise. While it is preferred to have the data distributed to the GTS in the shortest possible time after measurement, as long as the time between oceanographic observations and placing on the GTS is less than 30 days, the data are valuable in real-time.
It is often the case that some reduction in resolution (either the precision of the measurement, and/or the spatial or temporal resolution) is done. This is done to reduce the quantity of data that are sent through communications systems so as to reduce transmission costs. However, the GTS is capable of handling relatively high-resolution data, if they can be sent ashore. For example, sending XBT data at 1 m intervals from the surface to 800 m poses no difficulty for GTS transmission.
All code forms (TACs or TDCs) for ocean profiles have the ability to indicate if the depths reported for the observations are "selected" or "significant". A profile with selected depths is one where the depths at which observations are reported are selected independently of the shape of the profile. A profile with significant depths has used some algorithm to reduce the number of depths required while still reproducing the features of a profile to some pre-selected accuracy.
Getting data onto the GTS
For some kinds of data, such as from surface drifters that report using Service Argos, buoy operators need only give permission to Service Argos to distribute their data on the GTS. Then, with the necessary information to decode the communication from the buoy itself, Service Argos takes care of the rest. Service Argos may offer a similar service for data sent through the Iridium system.
For other kinds of data, what must be done depends on what facilities exist in your country. In some, the national ocean data centre will accept the data and do all of the necessary work to convert the data to the format required by the GTS. Normally they will also send the data to the national meteorological service for insertion onto the GTS. It is important to contact the data centre ahead of time to discuss formats for the data coming to the centre, and mechanisms to pass the data from the place where the data come ashore. The International Oceanographic Data and Information Exchange (IODE) Committee of IOC maintains a list of national co-ordinators for ocean data management. If your country has no such co-ordinator, contact IODE (see “contact us” in the above link).
Getting data from the GTS
Although this subject is outside of the scope of this series of documents, this is what you should do. If you are part of your national meteorological service, you should already know or easily find out what branch of your service manages GTS data.
If you are outside of a national meteorological service, you should contact your national service to ask if they can provide the data to you. You will need to discuss the format of the data extracted from the GTS and coming to you. You may need to write software to read the various data formats
If you are interested in oceanographic data, another possibility is to talk to the IODE for advice on who may already deal with real-time ocean data from the GTS.
Additional Resources
netCDF: http://www.unidata.ucar.edu/software/netcdf/
Subsections of this Article
No subsections available
Information about this article
Short title: Real-time Data Formats and Exchange
Description: An overview of formats used and methods for reporting data in real-time.
Expertise level: beginner
Author: bob.keeley
Approval status: approved
Approved by: bob.keeley
Last change: 2012-2-9
Subsection of: Exchanging Ocean Data in Real-time
Contact
If you have any direct comments or suggestions for the author of this page then please feel free to send an email to the author (listed above). For discussions on this page please use the discussions page.
