Derived Schemas

Introduction

XDR Derived Schemas provide a structured way to define and version data schemas for various log sources. These schemas are essential for processing and analyzing log data in a consistent manner across the XDR Data Engine.

Directory Structure

The derived schemas follow a standardized directory structure:

xxx_base_xde_config_xxx/xdr_derived_schemas/
├── metadata.txt                 # Contains DERIVED_SCHEMA_RELEASE_VERSION
└── logs_beats_filebeat/
    └── logs_beats_filebeat_{source}/
        └── v{major}_{minor}_{patch}/
            └── logs_beats_filebeat_{source}_sub.csv

Example:

xxx_base_xde_config_xxx/xdr_derived_schemas/
├── metadata.txt
└── logs_beats_filebeat/
    └── logs_beats_filebeat_activemq/
        └── v001_000_000/
            └── logs_beats_filebeat_activemq_sub.csv

Integration with Schema Management

XDR Package Configuration

Configure derived schemas in your xdr_package.yaml:

Key Configuration Points:

  • derived_schema_paths: Points to the root directory containing all derived schemas

  • schema_common_version: Specifies the common schema version to use

  • derived_schema_file_path: Relative path to the schema file from derived_schema_paths

  • derived_schema_ttl: Time-to-live in days for the schema data

Schema Builder Integration

The Schema Builder uses derived schemas to:

  1. Define field mappings

  2. Set data types

  3. Configure indexes

  4. Establish relationships with meta schemas

Working with Derived Schemas

Creating New Schemas

  1. Create Directory Structure:

  2. Create Schema File:

  3. Update Configuration:

    • Add schema to xdr_package.yaml

    • Set appropriate meta schema

    • Configure TTL and other parameters

Version Control

  • Version format: v{major}{minor}{patch}

  • Version tracked in metadata.txt using DERIVED_SCHEMA_RELEASE_VERSION

  • Each schema has its own version directory

  • Changes documented in release notes

CLI Commands

Related Documentation

PreviousXDE Build System Release NotesNextRelease Notes

Last updated