Derived Schemas

Introduction

XDR Derived Schemas provide a structured way to define and version data schemas for various log sources. These schemas are essential for processing and analyzing log data in a consistent manner across the XDR Data Engine.

Directory Structure

The derived schemas follow a standardized directory structure:

xxx_base_xde_config_xxx/xdr_derived_schemas/
├── metadata.txt                 # Contains DERIVED_SCHEMA_RELEASE_VERSION
└── logs_beats_filebeat/
    └── logs_beats_filebeat_{source}/
        └── v{major}_{minor}_{patch}/
            └── logs_beats_filebeat_{source}_sub.csv

Example:

xxx_base_xde_config_xxx/xdr_derived_schemas/
├── metadata.txt
└── logs_beats_filebeat/
    └── logs_beats_filebeat_activemq/
        └── v001_000_000/
            └── logs_beats_filebeat_activemq_sub.csv

Integration with Schema Management

XDR Package Configuration

Configure derived schemas in your xdr_package.yaml:

global_settings:
  # Point to your derived schemas directory
  derived_schema_paths: xxx_base_xde_config_xxx/xdr_derived_schemas

  # Optional: Configure schema defaults
  schema_common_version: v001_001_005
  schema_output_path: ../.xdr_schema_output/

schemas:
  # Example schema configuration
  logs_beats_filebeat_activemq:
    name: logs_beats_filebeat_activemq
    meta_schema: logs_beats_filebeat.csv
    meta_schema_version: v001_000_003
    derived_schema_file_path: logs_beats_filebeat/logs_beats_filebeat_activemq_sub.csv
    derived_schema_ttl: 90

Key Configuration Points:

• derived_schema_paths: Points to the root directory containing all derived schemas

• schema_common_version: Specifies the common schema version to use

• derived_schema_file_path: Relative path to the schema file from derived_schema_paths

• derived_schema_ttl: Time-to-live in days for the schema data

Schema Builder Integration

The Schema Builder uses derived schemas to:

1. Define field mappings

2. Set data types

3. Configure indexes

4. Establish relationships with meta schemas

Working with Derived Schemas

Creating New Schemas

1. Create Directory Structure:

   mkdir -p logs_beats_filebeat/logs_beats_filebeat_<source>/v001_000_000

2. Create Schema File:

   column,index_order,os_order
   timestamp,,
   message,,
   host*,,

3. Update Configuration:

   • Add schema to xdr_package.yaml

   • Set appropriate meta schema

   • Configure TTL and other parameters

Version Control

• Version format: v{major}{minor}{patch}

• Version tracked in metadata.txt using DERIVED_SCHEMA_RELEASE_VERSION

• Each schema has its own version directory

• Changes documented in release notes

CLI Commands

# Build schemas
xdrcli build-schemas --schema_filter_list=logs_beats_filebeat_<source>

# Apply schemas
xdrcli apply-schema --schema_filter_list=logs_beats_filebeat_<source>

Related Documentation

• Schema Builder Guide

• Schema Parameters

• Schema Updates

• Release Notes