HyperSec XDR Data Engine

CLI Beta Features

Introduction

Welcome to the Vector Ingestion Pipeline Builder! This tool facilitates the configuration and deployment of ingestion pipelines using Vector.

Description

The Vector Ingestion Pipeline Builder simplifies the process of configuring and deploying ingestion pipelines by providing a clean and intuitive command-line interface (CLI).

Configuration Files

To get started with the Vector Ingestion Pipeline Builder, follow these steps:

  1. Configuration: Create a xdr_package.yaml file with the necessary settings and define target configuration file xdr_targets.yaml for each environment.
  2. Usage: Use the provided CLI commands to build ingestion pipelines, upload configurations, templates, and MaxMind GeoIP databases to your desired storage location.

xdr_package.yaml

The xdr_package.yaml file serves as the central configuration file for the Vector Ingestion Pipeline Builder. It contains global settings, such as file paths and bucket names, as well as definitions for ingestion pipelines and templates.

Example xdr_package.yaml format :

global_settings:
  ingestion_template_s3_bucket: ghostburner-config-bucket
  ingestion_pipeline_output_path: ../.xdr_ingestion_pipelines_output 
  default_target: localhost
  target_path: ~/.xdr/xdr_targets.yaml


ip_config_standard_enrichment:
  # Paths to standard enrichment files
  - name: hypersec-enrichment-ch-array-fields.csv
    version: 100.000.000 
  # Other enrichment configurations...

ip_config_receiver:
  # Receiver configuration paths
  - name: hypersec-receiver-event-category-map.csv
    version: 100.000.000 
  # Other receiver configurations...

vector_enrichment_maxmind_geoip:
- name: GeoLite2-Country.mmdb
  version: 100.000.000
- name: GeoLite2-City.mmdb
  version: 100.000.000
- name: GeoLite2-ASN.mmdb
  version: 100.000.000

vector_templates:
  core:
  - name: 000-source-file.yml
    version: 100.000.000
  - name: 000-source-test-file-with-receiver-emulation.yml
    version: 100.000.000
  - name: 001-source-kafka-aws-saas.yml
    version: 100.000.000
  - name: 101-transform-flatten-message.yml
    version: 100.000.000

ingestion_pipeline_globals: 
  ingestion_data_dir: "/vector-data-dir"    # This points vector pods where to save the checkpointing and state.
  ingestion_mtls_path: "/etc/vector_tls"
  kafka_mtls_path: "/etc/vector_mtls"
  kafka_brokers: "b-2.ghostburner.ra5n0s.c3.kafka.ap-southeast-2.amazonaws.com:9094,b-3.ghostburner.ra5n0s.c3.kafka.ap-southeast-2.amazonaws.com:9094,b-1.ghostburner.ra5n0s.c3.kafka.ap-southeast-2.amazonaws.com:9094"

ingestion_pipelines:
  - name: logs_nxlog_windows
    stages:
      - name: finalise
        description: "Finalization stage for processing Windows NXLog data"
        config:
          kafka_source_topic: "logs_nxlog_windows"
          kafka_source_topic_suffix: "_land"
          kafka_sink_topic: "logs_nxlog_windows_load"
          kafka_consumer_group: "vector-logs-nxlog-windows-finalize"
          container_config:
            min_replicas: 1
            max_replicas: 9
            target_memory_util_percentage: 125
            target_CPU_utilization_percentage:  125
            persistence_size: 1Gi
          config_files:
            ip_fields: '/standard_enrichment_files/hypersec-enrichment-ip-fields.csv'                  # 
            extract_fields_csv: '/standard_enrichment_files/hypersec-enrichment-extract-fields.csv'    # 
            field_setter_iteration_group: 'finalise'                                                   # 
          steps:
            - id: hs-xdr-vector-ct-all-main
              type: base
            - id: 001-source-kafka-aws-saas
              type: source
            - id: 101-transform-flatten-message
              type: transform
            - id: 104-transform-extract-fields
              type: transform
            - id: 105-transform-junk-filter
              type: filter
            - id: 115-transform-timestamp-load-field
              type: transform
            - id: 119-transform-split-ip-field
              type: transform
            - id: 129-transform-event-hash-field
              type: transform
            - id: 202-sink-kafka-aws-saas
              type: sink
      - name: load_ch
        description: "ClickHouse loading stage"
        config:
          kafka_source_topic: "logs_nxlog_windows"
          kafka_source_topic_suffix: "_load"
          kafka_consumer_group: "vector_logs_nxlog_windows-load-ch-1"
          container_config:
            min_replicas: 1
            max_replicas: 1
            target_memory_util_percentage: 125
            targetCPUUtilizationPercentage:  125
            persistence_size: 1Gi
          config_files:
            json_fields: '/standard_enrichment_files/hypersec-enrichment-json-fields.csv'
            supported_timestamp_format: '["%FT%X%.3fZ", "%FT%X%.6fZ", "%FT%X%.9fZ", "%F %X%.3f", "%F %X%.6f", "%F %X%.9f","%FT%X%.3f", "%FT%X%.6f", "%FT%X%.9f","%FT%XZ","%FT%X","%F %X", "%FT%X"]'
            remap_fields: '/standard_enrichment_files/hypersec-enrichment-ch-remap-fields.csv'
          steps:
            - id: hs-xdr-vector-ct-all-main
              type: base
            - id: 001-source-kafka-aws-saas
              type: source
            - id: 101-transform-flatten-message
              type: transform
            - id: 108-transform-ch-json-remap
              type: transform
            - id: 102-transform-remap-fields
              type: transform
            - id: 116-transform-fields-to-camel-case
              type: transform
            - id: 109-transform-ch-custom-transformations
              type: transform
            - id: 201-sink-clickhouse-saas
              type: sink
      - name: load_os
        description: "OpenSearch loading stage"
        config:
          kafka_source_topic: "logs_nxlog_windows"
          kafka_source_topic_suffix: "_load"
          kafka_consumer_group: "vector-logs-nxlog-windows-load-os-1"
          container_config:
            min_replicas: 1
            max_replicas: 1
            target_memory_util_percentage: 125
            targetCPUUtilizationPercentage:  125
            persistence_size: 1Gi
          config_files:
            remap_fields: '/standard_enrichment_files/hypersec-enrichment-ch-remap-fields.csv'
            opensearch_endpoint: 'https://vpc-ghostburner-vc3t7e5zqeeu4nsa6vrcz6hwfy.ap-southeast-2.es.amazonaws.com'
          steps:
            - id: hs-xdr-vector-ct-all-main
              type: base
            - id: 001-source-kafka-aws-saas
              type: source
            - id: 101-transform-flatten-message
              type: transform
            - id: 111-transform-os-set-data-stream-fields
              type: transform
            - id: 102-transform-remap-fields
              type: transform
            - id: 116-transform-fields-to-camel-case
              type: transform
            - id: 204-sink-opensearch-aws-saas-stream
              type: sink
      - name: load_s3
        description: "S3 loading stage"
        config:
          kafka_source_topic: "logs_nxlog_windows_load"
          kafka_source_topic_suffix: ""
          kafka_consumer_group: "vector-logs-nxlog-windows-load-s3"
          container_config:
            min_replicas: 1
            max_replicas: 1
            target_memory_util_percentage: 125
            targetCPUUtilizationPercentage:  125
            persistence_size: 1Gi
          sink_config:
            sink_aws_s3_bucket_name.: 'ghostburner-archive'
            sink_aws_s3_region: 'ap-southeast-2'
            sink_aws_s3_batch_max_bytes: "21474836480"
            sink_aws_s3_batch_timeout_secs: "3600"
          steps:
            - id: hs-xdr-vector-ct-all-main
              type: base
            - id: 001-source-kafka-aws-saas
              type: source
            - id: 101-transform-flatten-message
              type: transform
            - id: 207-sink-aws-s3
              type: sink

Target Configuration Files

Target configuration files provide specific settings for different environments, such as development, staging, and production. These files override settings defined in the xdr_package.yaml file to tailor the pipeline setup for each environment.

Example xdr_targets,yaml format:

xdr_targets.yaml

default_target: dev
targets:
  dev:
    # Development environment configurations
    ch_host: localhost
    ch_port: 8123
    hunt_config_path: ../.xdr_hunt_config/first_ten_windows_audit_hunt/hunt
    hunt_rules_path: ../.xdr_hunt_config/first_ten_windows_audit_hunt/rules
    ip_config_bucket_name: xdr_config_bucket
    ip_config_bucket_region: ap-southeast-2
    ip_config_standard_enrichment_path: /etc/vector/vector_templates/standard_enrichment_files
    ip_config_geo_ip_path: vector_templates/geoip
    ip_config_receiver_path: vector_templates/vector_receiver
    ip_templates_path: /etc/vector/vector_templates
    # Other environment configurations....

Templates Details:

Global Settings

Setting Description
ingestion_template_s3_bucket Specifies the S3 bucket name where ingestion templates and configuration files are stored.

Vector Core Configuration

Setting Description
ch_array_fields Path to the CSV file defining fields that should be treated as arrays within ClickHouse.
ch_event_subschema_map CSV file specifying the subschema mapping for events.
ch_fields_with_invalid_char Lists fields with characters that are considered invalid for ClickHouse and need special handling.
ch_remap_fields Defines how certain fields should be remapped in the ClickHouse schema.
domain_fields Specifies the fields related to domain names for enrichment or processing.
extract_fields Lists fields where specific values need to be extracted from composite fields.
filter_whitelist Provides a whitelist of fields and their acceptable values for filtering purposes.
ip_fields Defines fields to be treated or processed as IP addresses.
json_fields Specifies which fields contain JSON data and require special processing.
logs_syslog_remap_fields Contains syslog fields mapping specific to logs ingestion and processing.
maxmind_list Path to the CSV file listing fields enriched using MaxMind databases.
public_suffix_list Provides the CSV file mapping domain names to their respective public suffixes.
time_fields Lists fields related to timestamps and their formats.
ts_match_tz Path to the file defining mappings between timestamps and timezone data.
tz_offset_mapping Specifies mappings between geographic locations and their timezone offsets.
update_datatype Defines fields that require datatype updates or transformations.
hypersec_receiver_event_category_map Maps event categories to predefined classifications within the ClickHouse schema.

Vector Enrichment MaxMind GeoIP

Setting Description
name The name of the MaxMind GeoIP database file.
version The version of the GeoIP database being used.

Vector Templates

Core Templates

Setting Description
name Unique identifier for the vector template file.
version Specifies the version of the template to help manage updates and compatibility.

Custom Templates

Setting Description
name Defines the name of a custom vector template not included in the core list.
version Indicates the version of the custom template for management and reference.

Ingstion Pipeline CLI Commands

Example:

xdrcli build-ingestion-pipelines --xdr_package_file_path /path/to/xdr_package.yaml --ingestion_pipeline_output_path /path/to/output --xdr_root_log_path /path/to/logs --target production --target_file_path /pathto/.xdr/xdr_targets.yaml

Options:

Example:

xdrcli upload-vector-core-config --aws-profile myprofile --log_path /path/to/logs

Options:

xdrcli upload-vector-templates --aws-profile myprofile --log_path /path/to/logs

Options:

xdrcli upload-vector-maxmind-mmdb --aws-profile myprofile --log_path /path/to/logs

Options: