Derived Schemas

Introduction

XDR Derived Schemas provide a structured way to define and version data schemas for various log sources. These schemas are essential for processing and analyzing log data in a consistent manner across the XDR Data Engine.

Directory Structure

The derived schemas follow a standardized directory structure:

xxx_base_xde_config_xxx/xdr_derived_schemas/
├── metadata.txt                 # Contains DERIVED_SCHEMA_RELEASE_VERSION
└── logs_beats_filebeat/
    └── logs_beats_filebeat_{source}/
        └── v{major}_{minor}_{patch}/
            └── logs_beats_filebeat_{source}_sub.csv

Example:

xxx_base_xde_config_xxx/xdr_derived_schemas/
├── metadata.txt
└── logs_beats_filebeat/
    └── logs_beats_filebeat_activemq/
        └── v001_000_000/
            └── logs_beats_filebeat_activemq_sub.csv

Integration with Schema Management

XDR Package Configuration

Configure derived schemas in your xdr_package.yaml:

global_settings:
  # Point to your derived schemas directory
  derived_schema_paths: xxx_base_xde_config_xxx/xdr_derived_schemas

  # Optional: Configure schema defaults
  schema_common_version: v001_001_005
  schema_output_path: ../.xdr_schema_output/

schemas:
  # Example schema configuration
  logs_beats_filebeat_activemq:
    name: logs_beats_filebeat_activemq
    meta_schema: logs_beats_filebeat.csv
    meta_schema_version: v001_000_003
    derived_schema_file_path: logs_beats_filebeat/logs_beats_filebeat_activemq_sub.csv
    derived_schema_ttl: 90

Key Configuration Points:

derived_schema_paths: Points to the root directory containing all derived schemas
schema_common_version: Specifies the common schema version to use
derived_schema_file_path: Relative path to the schema file from derived_schema_paths
derived_schema_ttl: Time-to-live in days for the schema data

Schema Builder Integration

The Schema Builder uses derived schemas to:

Define field mappings
Set data types
Configure indexes
Establish relationships with meta schemas

Working with Derived Schemas

Creating New Schemas

Create Directory Structure:

bash mkdir -p logs_beats_filebeat/logs_beats_filebeat_<source>/v001_000_000 2. Create Schema File:

csv column,index_order,os_order timestamp,, message,, host*,, 3. Update Configuration: * Add schema to xdr_package.yaml * Set appropriate meta schema * Configure TTL and other parameters

Version Control

Version format: v{major}{minor}
Version tracked in metadata.txt using DERIVED_SCHEMA_RELEASE_VERSION
Each schema has its own version directory
Changes documented in release notes

CLI Commands

# Build schemas
xdrcli build-schemas --schema_filter_list=logs_beats_filebeat_<source>

# Apply schemas
xdrcli apply-schema --schema_filter_list=logs_beats_filebeat_<source>