Account
Blacknest BDS DataFile API
| Date | 2010-06-04 |
| Version | 1.2.4 |
Note that this is a "work in progress" document and is being continually updated.
Introduction
Seismic sensor data is stored and transfered in a number of different formats. In order to make the BDS system as simple and flexible to use as possible the BDS system stores all data in its own internal format, BDS. This format has been designed so that it can encapsulate all of the information from external data formats. It is an internal format not intended for external use. This allows it to be easily modified and extended as required to support other external data formats or for system requirements.Note that although all data file formats will store the raw seismic data samples, most will not store all of the additional meta data information such as instrument responses, locations etc. The BDS data format can store the Meta data as well as the raw seismic sample data. However, it is expected that only basic Meta data will be included in the BDS data files for consistency checking purposes. The BDS system generally stores the channel Meta Data in a database. A user or data program will need the seismic sensor data and the meta data information. The BDS API provides both the Meta data and the seismic sensor data.
In order to simplify data file access and allow the easy creation of data converters the BDS system has a data file access API. This API provides a simple, common access to seismic data files of any format including the BDS Data file format.
Data File Model
The BDS Data File API is based on the following common data file model.- Data files may have an overall header defining meta data such as array and/or stations, time and other information.
- The seismic data is split into blocks. These may be variable or fixed length. Each block has a time stamp associated with it.
- Each data block may have special meta data information associated with it.
- There may be a single channel or multiple channels of data in the file.
- If there are multiple channels, these may be synchronously sampled or independently sampled.
- Each channels data may be split into separate segments. Each segment has a set of data for a contiguous time period. There may be multiple segments due to missing blocks or other reasons.
- Data is indexed by: Time, Network, Array or Station and Channel
Features
The key features of the BDS Data File API and the BDS Data file format are:- Keeps original data samples intact. No interpolation of original data samples.
- Keeps original time stamps intact.
- Keeps original block sizes with timestamps intact.
- Data channels can multiplexed at the channel level or sample level.
- Able to support synchronously or asynchronously sampled multiplexed data channels
- Simple API to read/write file files.
The Bds Data File API
The Bds Data File API is implemented as a set of C++ classes. There is a parent class, Bds::DataFile, which defines the main API functions available independent of the actual file format. Derived from the Bds::DataFile class is a set of classes one for each of the supported data file types. These implement the format specific functions to implement the BDS Data File API.The core API functions include:
| Function | Description |
|---|---|
| BError open(BString fileName, BString mode) | Opens the data file in read or write mode |
| void close() | Closes the file |
| File Write Interface | |
| BError setFormat(BString format) | Sets the output data format. This is to allow sub-formats for example IMS-1.0 and IMS-2.0 within the sample converter. |
| BError setInfo(DataInfo& dataInfo, BList<ChannelInfo>& channelInfos) | Provides information on the data. this is used to create file headers etc. The channelInfos list is normally provided from the Meta data database if available. Some formats require this information while others don't. |
| BError start(BUInt channel, BUInt segment); | Start outputting data for the given channel and segment. This will, generally, introduce a header into the output format. |
| BError writeData(DataBlock& data) | Writes a data block to the file |
| BError end(); | Ends the current segment |
| File Read Interface | |
| BError getFormat(BString& format) | Gets information on the data format |
| BError getInfo(DataInfo& dataInfo, DataFileOptions options, BList<BErrorTime>& errors) | Gets information on the data from the files data header and perhaps the data blocks. The options specify validation and processing options defined in an options list and also any blocks to ignore. Any Validation errors and warnings are listed in the errors list. |
| BError seekBlock(BUInt32 channel, BUInt32 segment, BTimeStamp time, BUInt32& blockNumber, BUInt64& sampleNumber, DataBlock& data) | Seeks to a particular data block given a time. This can operate on a single channel if a channel number is given or on multiple channels if channel number is 0 |
| BError readData(BUInt32 channel, BUInt32 segment, BUInt32 blockNumber, DataBlock& data); | Reads a data block from the file. If the channel number is given, then it reads data from a given channel. Otherwise it will read a set of data from all of the channels. |
| Misc | |
| BStringList getFormats(); | retuns the list or formats this converter supports. |
The API is able to handle single and multiple independent channel files and synchronously sampled multiple channel files.
Bds Data File Converter Access
To simplify the choice of file converter to use the BdsDataLib provides a class named DataFormats. This class has following functions:| Function | Description |
|---|---|
| BError formatList(BList<BStringList>& formats); | Gets the complete list of supported formats |
| BError formatGet(BString format, DataFile*& dataFile); | Given a required format, returns a suitable data converter for access. |
BdsServer Internal Data Handling
All requests for data go through the BdsServer's data API calls. The BdsServer is responsible for getting the appropriate sensor data out of the data files and then collating it for return to the user in the format requested. Some export data formats are channel multiplexed and some, such as the BKNAS format, are sample multiplexed.Generally the BdsSystem treats each channel of sensor data separately. The user selects the channels of data they are interested in over a particular period of time. The actual sensor data channels may be stored in individual files or may be multiplexed together in a single file. They can be multiplexed on a channel or a sample basis in the files. In the case of synchronously sampled channels from a seismic array they are normally stored in a sample multiplexed fashion in a single file and flagged as synchronously sampled. Note that, for the time period of interest, there may be multiple data files per channel each having the data for a portion of the time. In the case of analogue tape digitised data, these separate data files may have a time overlap. Normally these are returned in separate segments per channel.
When a user selects a set of sensor data channels, the system will open all of the necessary files to access. The user can then read the data one channel's segment at a time or multiple channels at a time. When reading multiple channels at a time the system will return blocks of sensor data of fixed time period. There may be different numbers of samples per channel however.