Schema Quick Start

Introduction

The XDR Data Engine provides tools for managing database schemas, ingestion pipelines, and data processing for HyperSec XDR. This guide will help you get started quickly with schema management.

Prerequisites

1. Python 3.11 or higher

2. Poetry for dependency management

3. Access to ClickHouse database

4. Required configuration files

Basic Setup

1. Configure your environment:

   # xdr_package.yaml
   global_settings:
     schema_common_version: v001.001.005
     schema_directory: ../.xdr_schema_config/stable_schemas
     schema_output_path: ../.xdr_schema_output/
     default_target: localhost
     target_path: ~/.xdr/xdr_targets.yaml
   
   build_schemas:
     no_cluster_declarations_needed: true
     use_replicated_merge_tree: false
   
   apply_schemas:
     do_add_roles: false
     do_add_columns: true
   
   organisations:
     - org_id: org321
       cluster_name: cluster1
     - org_id: org111111
       cluster_name: cluster1
     - org_id: detectionlab
       cluster_name: cluster1
   
   schemas:
     logs_alerts:
       name: logs_alerts
       meta_schema: logs_alerts.csv
       meta_schema_version: v100.000.000
     logs_hypercol_metric:
       name: logs_hypercol_metric
       meta_schema: logs_hypercol_metric.csv
       meta_schema_version: v100.000.001
       derived_schema_file_path: logs_hypercol_metric/logs_hypercol_metric_sub.csv
       additional_fields_config: logs_hypercol_metric/logs_hypercol_metric_add.csv
       derived_schema_ttl: 90

2. Set up your target environment:

   # xdr_targets.yaml
   default_target: dev
   targets:
     dev:
       ch_host: localhost
       ch_port: 8123
       hunt_config_path: ../.xdr_hunt_config/first_ten_windows_audit_hunt/hunt
       hunt_rules_path: ../.xdr_hunt_config/first_ten_windows_audit_hunt/rules
       ip_config_bucket_name: xdr_config_bucket
       ip_config_bucket_region: ap-southeast-2
       ip_config_standard_enrichment_path: /etc/vector/vector_templates/standard_enrichment_files
       ip_config_geo_ip_path: vector_templates/geoip
       ip_config_receiver_path: vector_templates/vector_receiver
       ip_templates_path: /etc/vector/vector_templates

Common Tasks

Building Schemas

# Build all schemas
xdrcli build-schemas

# Build specific schema
xdrcli build-schemas --schema_filter_list logs_alerts

Applying Schemas

# Apply all schemas
xdrcli apply-schemas

# Apply with column additions
xdrcli apply-schemas --do_add_columns

Planning Schema Changes

# Preview schema changes
xdrcli plan-schemas --schema_filter_list logs_alerts --organisation_lookup_key org321

Updating Schemas

# Apply schema updates
xdrcli update-schemas --schema_filter_list logs_alerts --organisation_lookup_key org321 --schema_update_flag YES

Next Steps

• Read Schema Overview for detailed schema concepts

• Check CLI Commands for complete command reference

• See Configuration for detailed configuration options

• Review Schema Builder Guide for schema development