Derived Schemas

Introduction

XDR Derived Schemas provide a structured way to define and version data schemas for various log sources. These schemas are essential for processing and analyzing log data in a consistent manner across the XDR Data Engine.

Directory Structure

The derived schemas follow a standardized directory structure:

xxx_base_xde_config_xxx/xdr_derived_schemas/
├── metadata.txt                 # Contains DERIVED_SCHEMA_RELEASE_VERSION
└── logs_beats_filebeat/
    └── logs_beats_filebeat_{source}/
        └── v{major}_{minor}_{patch}/
            └── logs_beats_filebeat_{source}_sub.csv

Example:

xxx_base_xde_config_xxx/xdr_derived_schemas/
├── metadata.txt
└── logs_beats_filebeat/
    └── logs_beats_filebeat_activemq/
        └── v001_000_000/
            └── logs_beats_filebeat_activemq_sub.csv

Integration with Schema Management

XDR Package Configuration

Configure derived schemas in your xdr_package.yaml:

global_settings:
  # Point to your derived schemas directory
  derived_schema_paths: xxx_base_xde_config_xxx/xdr_derived_schemas

  # Optional: Configure schema defaults
  schema_common_version: v001_001_005
  schema_output_path: ../.xdr_schema_output/

schemas:
  # Example schema configuration
  logs_beats_filebeat_activemq:
    name: logs_beats_filebeat_activemq
    meta_schema: logs_beats_filebeat.csv
    meta_schema_version: v001_000_003
    derived_schema_file_path: logs_beats_filebeat/logs_beats_filebeat_activemq_sub.csv
    derived_schema_ttl: 90

Key Configuration Points:

  • derived_schema_paths: Points to the root directory containing all derived schemas

  • schema_common_version: Specifies the common schema version to use

  • derived_schema_file_path: Relative path to the schema file from derived_schema_paths

  • derived_schema_ttl: Time-to-live in days for the schema data

Schema Builder Integration

The Schema Builder uses derived schemas to:

  1. Define field mappings

  2. Set data types

  3. Configure indexes

  4. Establish relationships with meta schemas

Working with Derived Schemas

Creating New Schemas

  1. Create Directory Structure:

    mkdir -p logs_beats_filebeat/logs_beats_filebeat_<source>/v001_000_000

  2. Create Schema File:

    column,index_order,os_order
timestamp,,
message,,
host*,,

  3. Update Configuration:

    • Add schema to xdr_package.yaml

    • Set appropriate meta schema

    • Configure TTL and other parameters

Version Control

  • Version format: v{major}{minor}{patch}

  • Version tracked in metadata.txt using DERIVED_SCHEMA_RELEASE_VERSION

  • Each schema has its own version directory

  • Changes documented in release notes

CLI Commands

# Build schemas
xdrcli build-schemas --schema_filter_list=logs_beats_filebeat_<source>

# Apply schemas
xdrcli apply-schema --schema_filter_list=logs_beats_filebeat_<source>

Related Documentation

PreviousXDE Build System Release NotesNextRelease Notes

Last updated