Blacknest BDS DataFile API

Date	2010-06-04
Version	1.2.4

Note that this is a "work in progress" document and is being continually updated.

Introduction

Seismic sensor data is stored and transfered in a number of different formats. In order to make the BDS system as simple and flexible to use as possible the BDS system stores all data in its own internal format, BDS. This format has been designed so that it can encapsulate all of the information from external data formats. It is an internal format not intended for external use. This allows it to be easily modified and extended as required to support other external data formats or for system requirements.

Note that although all data file formats will store the raw seismic data samples, most will not store all of the additional meta data information such as instrument responses, locations etc. The BDS data format can store the Meta data as well as the raw seismic sample data. However, it is expected that only basic Meta data will be included in the BDS data files for consistency checking purposes. The BDS system generally stores the channel Meta Data in a database. A user or data program will need the seismic sensor data and the meta data information. The BDS API provides both the Meta data and the seismic sensor data.

In order to simplify data file access and allow the easy creation of data converters the BDS system has a data file access API. This API provides a simple, common access to seismic data files of any format including the BDS Data file format.

Data File Model

The BDS Data File API is based on the following common data file model.

Data files may have an overall header defining meta data such as array and/or stations, time and other information.
The seismic data is split into blocks. These may be variable or fixed length. Each block has a time stamp associated with it.
Each data block may have special meta data information associated with it.
There may be a single channel or multiple channels of data in the file.
If there are multiple channels, these may be synchronously sampled or independently sampled.
Each channels data may be split into separate segments. Each segment has a set of data for a contiguous time period. There may be multiple segments due to missing blocks or other reasons.
Data is indexed by: Time, Network, Array or Station and Channel

Features

The key features of the BDS Data File API and the BDS Data file format are:

Keeps original data samples intact. No interpolation of original data samples.
Keeps original time stamps intact.
Keeps original block sizes with timestamps intact.
Data channels can multiplexed at the channel level or sample level.
Able to support synchronously or asynchronously sampled multiplexed data channels
Simple API to read/write file files.

The Bds Data File API

The Bds Data File API is implemented as a set of C++ classes. There is a parent class, Bds::DataFile, which defines the main API functions available independent of the actual file format. Derived from the Bds::DataFile class is a set of classes one for each of the supported data file types. These implement the format specific functions to implement the BDS Data File API.
The core API functions include:

Function	Description
BError open(BString fileName, BString mode)	Opens the data file in read or write mode
void close()	Closes the file

File Write Interface
BError setFormat(BString format)	Sets the output data format. This is to allow sub-formats for example IMS-1.0 and IMS-2.0 within the sample converter.
BError setInfo(DataInfo& dataInfo, BList<ChannelInfo>& channelInfos)	Provides information on the data. this is used to create file headers etc. The channelInfos list is normally provided from the Meta data database if available. Some formats require this information while others don't.
BError start(BUInt channel, BUInt segment);	Start outputting data for the given channel and segment. This will, generally, introduce a header into the output format.
BError writeData(DataBlock& data)	Writes a data block to the file
BError end();	Ends the current segment

File Read Interface
BError getFormat(BString& format)	Gets information on the data format
BError getInfo(DataInfo& dataInfo, DataFileOptions options, BList<BErrorTime>& errors)	Gets information on the data from the files data header and perhaps the data blocks. The options specify validation and processing options defined in an options list and also any blocks to ignore. Any Validation errors and warnings are listed in the errors list.
BError seekBlock(BUInt32 channel, BUInt32 segment, BTimeStamp time, BUInt32& blockNumber, BUInt64& sampleNumber, DataBlock& data)	Seeks to a particular data block given a time. This can operate on a single channel if a channel number is given or on multiple channels if channel number is 0
BError readData(BUInt32 channel, BUInt32 segment, BUInt32 blockNumber, DataBlock& data);	Reads a data block from the file. If the channel number is given, then it reads data from a given channel. Otherwise it will read a set of data from all of the channels.

Misc
BStringList getFormats();	retuns the list or formats this converter supports.

The API is able to handle single and multiple independent channel files and synchronously sampled multiple channel files.

Bds DataFile class

Bds Data File Converter Access

To simplify the choice of file converter to use the BdsDataLib provides a class named DataFormats. This class has following functions:

Function	Description
BError formatList(BList<BStringList>& formats);	Gets the complete list of supported formats
BError formatGet(BString format, DataFile*& dataFile);	Given a required format, returns a suitable data converter for access.

BdsServer Internal Data Handling

All requests for data go through the BdsServer's data API calls. The BdsServer is responsible for getting the appropriate sensor data out of the data files and then collating it for return to the user in the format requested. Some export data formats are channel multiplexed and some, such as the BKNAS format, are sample multiplexed.

Generally the BdsSystem treats each channel of sensor data separately. The user selects the channels of data they are interested in over a particular period of time. The actual sensor data channels may be stored in individual files or may be multiplexed together in a single file. They can be multiplexed on a channel or a sample basis in the files. In the case of synchronously sampled channels from a seismic array they are normally stored in a sample multiplexed fashion in a single file and flagged as synchronously sampled. Note that, for the time period of interest, there may be multiple data files per channel each having the data for a portion of the time. In the case of analogue tape digitised data, these separate data files may have a time overlap. Normally these are returned in separate segments per channel.

When a user selects a set of sensor data channels, the system will open all of the necessary files to access. The user can then read the data one channel's segment at a time or multiple channels at a time. When reading multiple channels at a time the system will return blocks of sensor data of fixed time period. There may be different numbers of samples per channel however.

Data File Collation

A few of the external file formats require samples output in synchronously sampled, sample multiplexed blocks. If the original data format is in synchronously sampled form this is easy to do. However, if the original data is in separate channels this is more difficult. Currently the BDS system will issue an error if this occurs. In the future it could colate the data performing sample interpolation if required.

Further Information

BDS MetaData information