Database Headers

Topics:

Overview
Constants
Type Definitions
Enumerations
File Header
Block Header
File Lock Header
Record Lock Header

Overview

Type definitions, constants, and data structures needed by the 32/64-bit database engine and network DB functions to create database files and device headers.

// Constants for dynamic data attributes used by the first byte of the
// block header Status member.
const __SBYTE__ gxBadBlock     = 'B'; // Bad Block
const __SBYTE__ gxDeletedBlock = 'D'; // Deleted Block 
const __SBYTE__ gxNormalBlock  = 'N'; // Normal Read/Write attribute 
const __SBYTE__ gxRemovedBlock = 'R'; // Removed Block

// This block was received from a remote device. Added in revision C
const __SBYTE__ gxRemoteDeviceBlock = 'V'; 

// Constants for block control characters used by the second byte of the
// block header Status member. Added in revision C to handle device
// control commands.
const __SBYTE__ gxAddRemoteBlock    = 'A'; // Add a remote block
const __SBYTE__ gxChangeRemoteBlock = 'C'; // Change a remote block
const __SBYTE__ gxDeleteRemoteBlock = 'D'; // Delete a remote block
const __SBYTE__ gxRequestFailed     = 'F'; // The client request failed
const __SBYTE__ gxCloseConnection   = 'L'; // Close Client/Server connection
const __SBYTE__ gxKillServer        = 'K'; // Shutdown Client/Server
const __SBYTE__ gxRequestBlock      = 'R'; // Requesting a block
const __SBYTE__ gxSendBlock         = 'S'; // Sending raw data block
const __SBYTE__ gxAcknowledgeBlock  = 'W'; // Acknowledge a data block

// Constants for file operations and stream position 
const int gxMaxNameLength     = 255;     // Max length of file names
const FAU gxStartOfFile       = (FAU)0;  // First byte in the file
const FAU gxCurrAddress       = (FAU)-1; // Indicates current location
const gxUINT32 gxCheckWord    = 0xfefe;  // Default synchronization word
const gxINT32 gxFSListCorrupt = (FAU)-1; // Free space list is corrupt

Type definitions common to all platforms:
__SBYTE__ - Type definition for 8-bit signed values.
__UBYTE__ - Type definition for 8-bit unsigned values.
__WORD__ - Type definition for native integer types.
__UWORD__ - Type definition for native unsigned integer types.
__LWORD__ - Type definition for native 32-bit signed integer types.
__ULWORD__ - Type definition for native 32-bit unsigned integer types.
__SWORD__ - Type definition for native 16-bit signed integer types.
__USWORD__ - Type definition for native 16-bit unsigned integer types.
__DPFLOAT__ - Type definition for native 64-bit double precision floating points types.

Platform/Compiler specific type definitions:
__LLWORD__ - Type definition for native 64-bit signed integer types.
__ULLWORD__ - Type definition for native 64-bit unsigned integer types.

Type definitions used for file operations and stream position:
FAU - File address unit physical file address type.
gxStreamPos - Stream position indicator.

Type definition used to represent CRC-32 checksum values:
gxChecksum - CRC-32 checksum type.

Enumerated integer constants used by the database engine to control file access, I/O operations, and identify file lock types.

enum gxDatabaseAccessMode { // gxDatabase access/open mode enumeration
  gxDBASE_READONLY,   // Open file with read access only
  gxDABSE_WRITEONLY,  // Open file with write access only
  gxDBASE_READWRITE,  // Open file with read and write access
  gxDBASE_CREATE,     // Create the file if it does not exist
  gxDBASE_NO_CREATE,  // Do not create the file if it does not exist
  gxDBASE_TRUNCATE,   // Truncate the file  
  gxDBASE_APPEND,     // Append to the file
  gxDBASE_SHARE,      // Enable file sharing  (Platform specific)
  gxDBASE_EXCLUSIVE   // Disable file sharing (Platform specific)
};

enum gxDatabaseOperation { // gxDatabase I/O operation codes
  gxDBASE_READ,         // A read was performed
  gxDBASE_WRITE,        // A write operation was performed
  gxDBASE_REWIND,       // A rewind operation was performed
  gxDBASE_NO_OPERATION, // No operation was performed
  gxDBASE_SEEK          // A seek operation was preformed 
};

enum gxDatabaseSeekMode { // gxDatabase seek mode enumeration
  gxDBASE_SEEK_BEG, // Seek starting from the beginning of the file
  gxDBASE_SEEK_CUR, // Seek starting from the current location
  gxDBASE_SEEK_END  // Seek starting from the end of the file
};

enum gxDatabaseLockType { // gxDatabase file/record lock enumeration
  gxDBASE_READLOCK,
  gxDBASE_WRITELOCK
};

enum gxDatabaseReclaimMethod { // gxDatabase Block reclamation methods
  gxDBASE_RECLAIM_NONE = 0, // Do not reclaim deleted blocks
  gxDBASE_RECLAIM_BESTFIT,  // Use the best fit reclamation method
  gxDBASE_RECLAIM_FIRSTFIT  // Use the first fit reclamation method
};

The gxFileHeader data structure is used to store information needed by the allocation functions, a signature and version number. When a new file is created the database engine reserves enough space, starting at file address 0, for the database file header. The database file header is composed of six fields and will vary in length depending on whether 32-bit or 64-bit file offsets are used.

struct gxFileHeader // Database file header information
{ 
  FAU gxd_fs_fptr;      // Address to first block of free heap space
  FAU gxd_eof;          // Address of byte after end of file
  FAU gxd_hs_fptr;      // Address of the start of the heap space
  FAU gxd_hb_fptr;      // Highest allocated block
  __SBYTE__ gxd_sig[gxSignatureSize]; // Signature used to ID a database file 
  FAU gxd_ver;          // Database library version number
};

Free space field:
The "free space" pointer is used by the allocation function to reclaim deleted blocks. The number of deleted blocks is maintained in a non-contiguous list. Each block in the list points to the next block in the list starting at a specified address. The "free space" pointer is used to store the file address of the block where the list starts. In order to prevent database files from becoming extremely fragmented due to numerous deletions, blocks marked deleted or removed will be reused by the allocation function. The database engine's allocation function checks the "free space" field before allocating new blocks. If the "free space" field is not empty, the allocation function will walk through the free space list looking for a deleted or removed block of the size to be allocated. This includes the size of the block data plus the block header. One of two methods can be used to reclaim deleted or removed blocks, the best-fit method or the first-fit method. The best-fit method works by scanning the entire free sp ace list until the best location to reuse a block is found. The first-fit method works by searching the free space list until the first block of the appropriate size is found. Both methods will reuse any unused portion of a reclaimed block. The unused portion is assigned a new block header (marked removed) and placed back on the free space list. NOTE: If the free space list becomes corrupt, the "free space" field will be marked corrupt by the gxFSListCorrupt integer constant and is no longer used.

End of file field:
The "end of file" pointer is used to mark the end of the file and locate where the file can be extended during block allocation. When an existing file is opened the database engine will compare this field to the true end of file. If the values do not match, this pointer will be adjusted if possible.

Start of heap field:
The "start of heap" pointer is used to store the address where the static storage area ends and the dynamic data area begins. The allocation of new blocks will always start at this address. The size of the static storage area is determined when the file is created. Any data stored in the static area will not be altered by the block allocation routines.

Highest block field:
The "highest block" pointer is used to store the address of the highest allocated block. This ensures that the database engine will always know where the first and last blocks are located in the file. This pointer is also useful for building descending order linear navigation functions.

Signature field:
The signature field is used to determine if the file is of the correct type when an existing file is opened. When a new 32-bit file is created the database engine writes the "GXDBASE" string to this field and reserves the eighth byte for revision letter changes. When a new 64-bit file is created the database engine writes the "GXDBASE64" string to this field and reserves the tenth byte for revision letter changes.

Version field:
A version number is used to indicate that changes have been made to the database engine without affecting the file format. Version numbers are used to conditionally perform certain operations based on its numerical value. This will ensure backward compatibility with previous versions of the database engine.

The gxBlockHeader data structure is used to mark the start of a block in the dynamic data area and is additionally used for block transfers between local and remote devices. Block headers can be broken down into two specific categories: database blocks and device blocks. Database block headers are composed of four fields and will vary in length depending on whether 32-bit or 64-bit file offsets are used.

Database Blocks:
A database block header is stored with each block allocated by an application and is used by the database engine to keep track of blocks stored in the dynamic data area. Applications can also use database block header information to obtain file locations and status information. When new blocks are allocated the database engine writes the block header and then reserves enough space to store the number of bytes requested. The application is responsible for writing the actual data in the block starting at the address following the block header.

Device Blocks:
Device block headers contain synchronization and control information. They are used to transfer blocks and to signal events between devices. When an application sends a block of raw data the header precedes the block and informs the receiver of the block size and the block status. After reception of a device header the receiver then waits for the block data and processes the data or signals/handles an event according to the status of the block.

// Database/Device Block Header 
struct gxBlockHeader
{
  gxUINT32 block_check_word; // Block synchronization marker
  gxUINT32 block_length;     // Block length (object length + overhead)  
  gxUINT32 block_status;     // First byte  = status of the block's data 
                             // Second byte = block control commands
                             // Third byte  = reserved for future use
                             // Fourth byte = reserved for future use
  FAU block_nd_fptr;         // Pointer to next deleted block
};

Check word field:
The "check word" field represents a 32-bit check word. Database blocks use this field for integrity checks and to maintain synchronization between the database engine and the application. Device blocks use this field to maintain synchronization between local and remote devices. The check word field to should always be set using the gxCheckWord integer constant. NOTE: If the check word value is changed, database files and device drivers will no longer be compatible with previous releases.

Length field:
The "length" field stores the total length of the block. Block lengths include the size of the object plus the total size of the block overhead determined by the by the database revision letter and whether the block is a database block or a device block.

Database Block Lengths:
Revision 0 database blocks account for the length of the object plus the size of the block header. Revision 'A' database blocks account for the length of the object plus the size of the block header and the size of the block's CRC checksum value. Revision 'C' database blocks account for the length of the object plus the size of the block header, record lock header, and the size of the block's CRC checksum value. The database engine uses this field to index the file block by block. The "check-word" field is used to ensure that the next block in sequence is a valid block. The length of the object or record can be calculated by subtracting the total size of the block overhead from the "length" field.

Device Block Lengths:
Device blocks are used to send and receive blocks to and from various I/O devices. NOTE: All device blocks will only account for the length of the object. Device blocks used to send and receive control signals will only account for the length of the block header as no additional data will be sent or received other then the block header itself.

Status field:
The "status" field stores the status of the dynamic data contained in the block. Only two bytes of the status field are used. The remaining two bytes of the status field are reserved for future use.

Database Block Status Information:
The status of a database block can be determined by one of three primary byte values stored in the first block of the status field: 'N' for normal (ASCII 78), 'D' for deleted (ASCII 68), or 'R' for removed (ASCII 82). The database engine uses a signed byte constant named gxNormalBlock to mark normal database blocks, gxDeletedBlock to mark deleted database blocks, and gxRemovedBlock to mark database removed blocks. A block marked 'N' for normal indicates that the database block is in use and cannot be reclaimed by the allocation function. A block marked 'D' for deleted means that the data in the block is still valid, but the block can be overwritten if needed. Marking deleted blocks 'N' for normal can restore deleted database blocks. A block marked 'R' for removed means that the data in the block has been removed and the block can be overwritten. Once a block is removed it can never be restored.

Device Block Status Information:
The status of a device block can be determined by one of several byte values stored in the second block of the status field. Block control characters were added in revision 'C' to handle device control commands. Device control commands allow blocks to be added, changed, removed, and requested by local and remote devices. Device control commands are also used to signal events between local and remote devices.

Next deleted block field:
Database blocks use the "next deleted block" field to store a file address to the next deleted or removed block, only if this block has been deleted or removed. Device block headers ignore this field by default. The total number of deleted database blocks is maintained in a non-contiguous list. Each deleted or removed database block in the list points to the next deleted or removed database block in the list starting at the head of the list.

The file lock header is used in revision 'B' and higher to allow applications to lock the entire file during a multi-threaded/multi-machine read or write operations. The advisory file locking scheme includes a lock protect member required to protect the lock values during multiple file access. NOTE: The lock protect member is required in the platform independent file locking system because file lock headers operate independently of the I/O subsystem and are manipulated by the database engine in the same manner as database blocks.

struct gxFileLockHeader
{
  // The lock protect is used to serialize access to the file lock
  // values. The locking scheme will allow a total of 2^32 or
  // 4,294,967,295 threads to read lock a file and single thread to
  // write lock a file.
  gxUINT32 file_lock_protect; // Serialize access to the file lock members
  gxUINT32 file_read_lock;    // Shared (or read-only) file lock 
  gxUINT32 file_write_lock;   // Exclusive (or write-only) file lock
};

Record lock headers are used by an application in revision 'C' to lock a specific block during a multi-threaded/multi-machine read or write operation. The advisory record locking scheme includes a lock protect member required to protect the lock values during multiple file access. NOTE: The lock protect member is required in the platform independent node locking system because record lock headers operate independently of the I/O subsystem and are manipulated by the database engine in the same manner as database blocks.

struct gxRecordLockHeader // Lock a contiguous set of bytes
{
  // The lock protect is used to serialize access to each record 
  // lock. The locking scheme will allow a total of 2^32 or
  // 4,294,967,295 threads to read lock a record and single thread to
  // write lock a record.
  gxUINT32 record_lock_protect; // Serialize access to the record lock
  gxUINT32 record_read_lock;    // Shared (or read-only) file lock 
  gxUINT32 record_write_lock;   // Exclusive (or write-only) file lock
};