HyperSec XDR Data Engine

Schema Quick Start

Introduction

The XDR Data Engine provides tools for managing database schemas, ingestion pipelines, and data processing for HyperSec XDR. This guide will help you get started quickly with schema management.

Prerequisites

  1. Python 3.11 or higher
  2. Poetry for dependency management
  3. Access to ClickHouse database
  4. Required configuration files

Basic Setup

  1. Configure your environment:

    ```yaml

    xdr_package.yaml

    global_settings: schema_common_version: v001.001.005 schema_directory: ../.xdr_schema_config/stable_schemas schema_output_path: ../.xdr_schema_output/ default_target: localhost target_path: ~/.xdr/xdr_targets.yaml

    build_schemas: no_cluster_declarations_needed: true use_replicated_merge_tree: false

    apply_schemas: do_add_roles: false do_add_columns: true

    organisations: - org_id: org321 cluster_name: cluster1 - org_id: org111111 cluster_name: cluster1 - org_id: detectionlab cluster_name: cluster1

    schemas: logs_alerts: name: logs_alerts meta_schema: logs_alerts.csv meta_schema_version: v100.000.000 logs_hypercol_metric: name: logs_hypercol_metric meta_schema: logs_hypercol_metric.csv meta_schema_version: v100.000.001 derived_schema_file_path: logs_hypercol_metric/logs_hypercol_metric_sub.csv additional_fields_config: logs_hypercol_metric/logs_hypercol_metric_add.csv derived_schema_ttl: 90 ``` 2. Set up your target environment:

    ```yaml

    xdr_targets.yaml

    default_target: dev targets: dev: ch_host: localhost ch_port: 8123 hunt_config_path: ../.xdr_hunt_config/first_ten_windows_audit_hunt/hunt hunt_rules_path: ../.xdr_hunt_config/first_ten_windows_audit_hunt/rules ip_config_bucket_name: xdr_config_bucket ip_config_bucket_region: ap-southeast-2 ip_config_standard_enrichment_path: /etc/vector/vector_templates/standard_enrichment_files ip_config_geo_ip_path: vector_templates/geoip ip_config_receiver_path: vector_templates/vector_receiver ip_templates_path: /etc/vector/vector_templates ```

Common Tasks

Building Schemas

# Build all schemas
xdrcli build-schemas

# Build specific schema
xdrcli build-schemas --schema_filter_list logs_alerts

Applying Schemas

# Apply all schemas
xdrcli apply-schemas

# Apply with column additions
xdrcli apply-schemas --do_add_columns

Planning Schema Changes

# Preview schema changes
xdrcli plan-schemas --schema_filter_list logs_alerts --organisation_lookup_key org321

Updating Schemas

# Apply schema updates
xdrcli update-schemas --schema_filter_list logs_alerts --organisation_lookup_key org321 --schema_update_flag YES

Next Steps