Package Configuration

The xdr_package.yaml file is the central configuration file for the XDR Data Engine. It defines the system's behavior, data processing pipelines, and schema configurations.

Global Settings

global_settings:
  ingestion_template_s3_bucket: "development-config-bucket-afterburner"
  schema_common_version: "v001.001.005"
  schema_output_path: "../.xdr_schema_output/"
  ingestion_pipeline_output_path: "../.xdr_ingestion_pipelines_output"
  upload_ingestion_template_output_path: "../.xdr_ingestion_template_output"
  default_profile: "ghostburner"
  target_path: "~/.xdr/xdr_targets.yaml"

Organizations Configuration

Define organizations and their cluster settings:

organisations:
  - org_id: "org123"
    cluster_name: ""
  - org_id: "org456"
    cluster_name: ""

Schema Configuration

Build Settings

build_schemas:
  no_cluster_declarations_needed: true
  use_replicated_merge_tree: false

apply_schemas:
  do_add_roles: false
  do_add_columns: true

Schema Definitions

schemas:
  logs_alerts:
    name: "logs_alerts"
    meta_schema: "logs_alerts.csv"
    meta_schema_version: "v001.000.000"
    derived_schema_file_path: "logs_alerts/logs_alerts_sub.csv"
    additional_fields_config: "logs_alerts/logs_alerts_add.csv"

Ingestion Pipeline Configuration

Pipeline Globals

ingestion_pipeline_globals:
  ingestion_data_dir: "/vector-data-dir"
  ingestion_mtls_path: "/etc/vector_tls"
  kafka_mtls_path: "/etc/vector_mtls"

Pipeline Definition

Each pipeline can have multiple stages:

ingestion_pipelines:
  - name: "logs_alerts"
    stages:
      - name: "finalise"
        description: "Finalization stage"
        config:
          kafka_source_topic: "logs_alerts"
          kafka_sink_topic: "logs_alerts_load"
          # ... additional configuration

Vector Templates

Define vector transformation templates:

vector_templates:
  core:
    - name: "000-source-file.yml"
      version: "v001.000.000"
    - name: "001-source-kafka-aws-saas.yml"
      version: "v001.000.000"

Best Practices

1. Version Management * Use semantic versioning for schemas and templates * Document version changes * Maintain backward compatibility
2. Pipeline Configuration * Group related transformations into stages * Use descriptive names for pipelines and stages * Document pipeline dependencies
3. Schema Management * Keep schema definitions organized * Use consistent naming conventions * Document schema relationships
4. Resource Management * Configure appropriate resource limits * Monitor pipeline performance * Adjust configurations based on usage patterns

Common Tasks

Adding a New Schema

1. Define schema in the schemas section
2. Specify meta schema and version
3. Add any derived schema configurations
4. Configure additional fields if needed

Creating a New Pipeline

1. Add pipeline definition under ingestion_pipelines
2. Configure required stages
3. Define transformation steps
4. Set appropriate resource limits

Updating Vector Templates

1. Add new template under vector_templates
2. Specify version number
3. Reference in pipeline configurations
4. Test transformations before deployment