Meta Schemas

Introduction

XDR Meta Schemas define the core structure and validation rules for various data types in the XDR Data Engine. These schemas serve as the foundation for derived schemas and ensure data consistency across the platform.

Directory Structure

The meta schemas follow a standardized directory structure:

post_build_artefacts/xdr_meta_schemas/
└── schemas/
    ├── logs_alerts
    |   └── v{major}_{minor}_{patch}/
    |       └── logs_alerts.csv
    ├── logs_beats_filebeat
    |   └── v{major}_{minor}_{patch}/
    |       └── logs_beats_filebeat.csv
    ├── logs_beats_winlogbeat
    |   └── v{major}_{minor}_{patch}/
    |       └── logs_beats_winlogbeat.csv
    ├── logs_base
    |   └── v{major}_{minor}_{patch}/
    |       └── logs_base.csv
    ├── logs_hypercol_internal
    |   └── v{major}_{minor}_{patch}/
    |       └── logs_hypercol_internal.csv
    ├── logs_hypercol_metric
    |   └── v{major}_{minor}_{patch}/
    |       └── logs_hypercol_metric.csv
    ├── logs_nxlog_windows
    |   └── v{major}_{minor}_{patch}/
    |       └── logs_nxlog_windows.csv
    ├── logs_syslog
    |   └── v{major}_{minor}_{patch}/
    |      └── logs_syslog.csv
    ├── logs_syslog_linux
    |   └── v{major}_{minor}_{patch}/
    |      └── logs_syslog_linux.csv
    └── logs_syslog_linux_audited
        └── v{major}_{minor}_{patch}/
            └── logs_syslog_linux_audited.csv

Schema Types

Core Meta Schemas

logs_base.csv * Base schema for all log types * Defines common fields like timestamp, message, host

Specialized Meta Schemas

logs_beats_filebeat.csv (v001.000.003) * Schema for Filebeat log data * Used by multiple derived schemas (activemq, aws, zookeeper)
logs_beats_winlogbeat.csv (v001.000.000) * Windows event log schema * Specialized fields for Windows events
logs_nxlog_windows.csv (v001.000.003) * NXLog Windows data schema * Supports Windows event collection
logs_syslog.csv (v001.000.000) * Syslog format schema * Standard syslog fields and parsing
logs_hypercol_metric.csv (v001.000.001) * Metric data schema * Time-series and measurement fields

Integration with Schema Management

Configuration in xdr_package.yaml

meta_schema_paths: ../post_build_artefacts/xdr_meta_schemas_package

schemas:
  logs_alerts:
    meta_schema: logs_alerts.csv
    meta_schema_version: v001.000.000
  logs_beats_filebeat_activemq:
    meta_schema: logs_beats_filebeat.csv
    meta_schema_version: v001.000.003

Key Configuration Points:

meta_schema_paths: Points to the meta schemas directory
meta_schema: Specifies which meta schema to use
meta_schema_version: Controls schema versioning

Version Control

Version format: v{major}{minor}
Each schema versioned independently
Changes documented in release notes
Backward compatibility maintained