CLI Beta Features

Introduction

Welcome to the Vector Ingestion Pipeline Builder! This tool facilitates the configuration and deployment of ingestion pipelines using Vector.

Description

The Vector Ingestion Pipeline Builder simplifies the process of configuring and deploying ingestion pipelines by providing a clean and intuitive command-line interface (CLI).

Configuration Files

To get started with the Vector Ingestion Pipeline Builder, follow these steps:

1. Configuration: Create a xdr_package.yaml file with the necessary settings and define target configuration file xdr_targets.yaml for each environment.

2. Usage: Use the provided CLI commands to build ingestion pipelines, upload configurations, templates, and MaxMind GeoIP databases to your desired storage location.

xdr_package.yaml

The xdr_package.yaml file serves as the central configuration file for the Vector Ingestion Pipeline Builder. It contains global settings, such as file paths and bucket names, as well as definitions for ingestion pipelines and templates.

Example xdr_package.yaml format :

global_settings:
  ingestion_template_s3_bucket: ghostburner-config-bucket
  ingestion_pipeline_output_path: ../.xdr_ingestion_pipelines_output 
  default_target: localhost
  target_path: ~/.xdr/xdr_targets.yaml


ip_config_standard_enrichment:
  # Paths to standard enrichment files
  - name: hypersec-enrichment-ch-array-fields.csv
    version: 100.000.000 
  # Other enrichment configurations...

ip_config_receiver:
  # Receiver configuration paths
  - name: hypersec-receiver-event-category-map.csv
    version: 100.000.000 
  # Other receiver configurations...

vector_enrichment_maxmind_geoip:
- name: GeoLite2-Country.mmdb
  version: 100.000.000
- name: GeoLite2-City.mmdb
  version: 100.000.000
- name: GeoLite2-ASN.mmdb
  version: 100.000.000

vector_templates:
  core:
  - name: 000-source-file.yml
    version: 100.000.000
  - name: 000-source-test-file-with-receiver-emulation.yml
    version: 100.000.000
  - name: 001-source-kafka-aws-saas.yml
    version: 100.000.000
  - name: 101-transform-flatten-message.yml
    version: 100.000.000

ingestion_pipeline_globals: 
  ingestion_data_dir: "/vector-data-dir"    # This points vector pods where to save the checkpointing and state.
  ingestion_mtls_path: "/etc/vector_tls"
  kafka_mtls_path: "/etc/vector_mtls"
  kafka_brokers: "b-2.ghostburner.ra5n0s.c3.kafka.ap-southeast-2.amazonaws.com:9094,b-3.ghostburner.ra5n0s.c3.kafka.ap-southeast-2.amazonaws.com:9094,b-1.ghostburner.ra5n0s.c3.kafka.ap-southeast-2.amazonaws.com:9094"

ingestion_pipelines:
  - name: logs_nxlog_windows
    stages:
      - name: finalise
        description: "Finalization stage for processing Windows NXLog data"
        config:
          kafka_source_topic: "logs_nxlog_windows"
          kafka_source_topic_suffix: "_land"
          kafka_sink_topic: "logs_nxlog_windows_load"
          kafka_consumer_group: "vector-logs-nxlog-windows-finalize"
          container_config:
            min_replicas: 1
            max_replicas: 9
            target_memory_util_percentage: 125
            target_CPU_utilization_percentage:  125
            persistence_size: 1Gi
          config_files:
            ip_fields: '/standard_enrichment_files/hypersec-enrichment-ip-fields.csv'                  # 
            extract_fields_csv: '/standard_enrichment_files/hypersec-enrichment-extract-fields.csv'    # 
            field_setter_iteration_group: 'finalise'                                                   # 
          steps:
            - id: hs-xdr-vector-ct-all-main
              type: base
            - id: 001-source-kafka-aws-saas
              type: source
            - id: 101-transform-flatten-message
              type: transform
            - id: 104-transform-extract-fields
              type: transform
            - id: 105-transform-junk-filter
              type: filter
            - id: 115-transform-timestamp-load-field
              type: transform
            - id: 119-transform-split-ip-field
              type: transform
            - id: 129-transform-event-hash-field
              type: transform
            - id: 202-sink-kafka-aws-saas
              type: sink
      - name: load_ch
        description: "ClickHouse loading stage"
        config:
          kafka_source_topic: "logs_nxlog_windows"
          kafka_source_topic_suffix: "_load"
          kafka_consumer_group: "vector_logs_nxlog_windows-load-ch-1"
          container_config:
            min_replicas: 1
            max_replicas: 1
            target_memory_util_percentage: 125
            targetCPUUtilizationPercentage:  125
            persistence_size: 1Gi
          config_files:
            json_fields: '/standard_enrichment_files/hypersec-enrichment-json-fields.csv'
            supported_timestamp_format: '["%FT%X%.3fZ", "%FT%X%.6fZ", "%FT%X%.9fZ", "%F %X%.3f", "%F %X%.6f", "%F %X%.9f","%FT%X%.3f", "%FT%X%.6f", "%FT%X%.9f","%FT%XZ","%FT%X","%F %X", "%FT%X"]'
            remap_fields: '/standard_enrichment_files/hypersec-enrichment-ch-remap-fields.csv'
          steps:
            - id: hs-xdr-vector-ct-all-main
              type: base
            - id: 001-source-kafka-aws-saas
              type: source
            - id: 101-transform-flatten-message
              type: transform
            - id: 108-transform-ch-json-remap
              type: transform
            - id: 102-transform-remap-fields
              type: transform
            - id: 116-transform-fields-to-camel-case
              type: transform
            - id: 109-transform-ch-custom-transformations
              type: transform
            - id: 201-sink-clickhouse-saas
              type: sink
      - name: load_os
        description: "OpenSearch loading stage"
        config:
          kafka_source_topic: "logs_nxlog_windows"
          kafka_source_topic_suffix: "_load"
          kafka_consumer_group: "vector-logs-nxlog-windows-load-os-1"
          container_config:
            min_replicas: 1
            max_replicas: 1
            target_memory_util_percentage: 125
            targetCPUUtilizationPercentage:  125
            persistence_size: 1Gi
          config_files:
            remap_fields: '/standard_enrichment_files/hypersec-enrichment-ch-remap-fields.csv'
            opensearch_endpoint: 'https://vpc-ghostburner-vc3t7e5zqeeu4nsa6vrcz6hwfy.ap-southeast-2.es.amazonaws.com'
          steps:
            - id: hs-xdr-vector-ct-all-main
              type: base
            - id: 001-source-kafka-aws-saas
              type: source
            - id: 101-transform-flatten-message
              type: transform
            - id: 111-transform-os-set-data-stream-fields
              type: transform
            - id: 102-transform-remap-fields
              type: transform
            - id: 116-transform-fields-to-camel-case
              type: transform
            - id: 204-sink-opensearch-aws-saas-stream
              type: sink
      - name: load_s3
        description: "S3 loading stage"
        config:
          kafka_source_topic: "logs_nxlog_windows_load"
          kafka_source_topic_suffix: ""
          kafka_consumer_group: "vector-logs-nxlog-windows-load-s3"
          container_config:
            min_replicas: 1
            max_replicas: 1
            target_memory_util_percentage: 125
            targetCPUUtilizationPercentage:  125
            persistence_size: 1Gi
          sink_config:
            sink_aws_s3_bucket_name.: 'ghostburner-archive'
            sink_aws_s3_region: 'ap-southeast-2'
            sink_aws_s3_batch_max_bytes: "21474836480"
            sink_aws_s3_batch_timeout_secs: "3600"
          steps:
            - id: hs-xdr-vector-ct-all-main
              type: base
            - id: 001-source-kafka-aws-saas
              type: source
            - id: 101-transform-flatten-message
              type: transform
            - id: 207-sink-aws-s3
              type: sink

Target Configuration Files

Target configuration files provide specific settings for different environments, such as development, staging, and production. These files override settings defined in the xdr_package.yaml file to tailor the pipeline setup for each environment.

Example xdr_targets,yaml format:

xdr_targets.yaml

default_target: dev
targets:
  dev:
    # Development environment configurations
    ch_host: localhost
    ch_port: 8123
    hunt_config_path: ../.xdr_hunt_config/first_ten_windows_audit_hunt/hunt
    hunt_rules_path: ../.xdr_hunt_config/first_ten_windows_audit_hunt/rules
    ip_config_bucket_name: xdr_config_bucket
    ip_config_bucket_region: ap-southeast-2
    ip_config_standard_enrichment_path: /etc/vector/vector_templates/standard_enrichment_files
    ip_config_geo_ip_path: vector_templates/geoip
    ip_config_receiver_path: vector_templates/vector_receiver
    ip_templates_path: /etc/vector/vector_templates
    # Other environment configurations....

Templates Details:

Global Settings

Setting	Description
ingestion_template_s3_bucket	Specifies the S3 bucket name where ingestion templates and configuration files are stored.

Vector Core Configuration

Setting	Description
ch_array_fields	Path to the CSV file defining fields that should be treated as arrays within ClickHouse.
ch_event_subschema_map	CSV file specifying the subschema mapping for events.
ch_fields_with_invalid_char	Lists fields with characters that are considered invalid for ClickHouse and need special handling.
ch_remap_fields	Defines how certain fields should be remapped in the ClickHouse schema.
domain_fields	Specifies the fields related to domain names for enrichment or processing.
extract_fields	Lists fields where specific values need to be extracted from composite fields.
filter_whitelist	Provides a whitelist of fields and their acceptable values for filtering purposes.
ip_fields	Defines fields to be treated or processed as IP addresses.
json_fields	Specifies which fields contain JSON data and require special processing.
logs_syslog_remap_fields	Contains syslog fields mapping specific to logs ingestion and processing.
maxmind_list	Path to the CSV file listing fields enriched using MaxMind databases.
public_suffix_list	Provides the CSV file mapping domain names to their respective public suffixes.
time_fields	Lists fields related to timestamps and their formats.
ts_match_tz	Path to the file defining mappings between timestamps and timezone data.
tz_offset_mapping	Specifies mappings between geographic locations and their timezone offsets.
update_datatype	Defines fields that require datatype updates or transformations.
hypersec_receiver_event_category_map	Maps event categories to predefined classifications within the ClickHouse schema.

Vector Enrichment MaxMind GeoIP

Setting	Description
name	The name of the MaxMind GeoIP database file.
version	The version of the GeoIP database being used.

Vector Templates

Core Templates

Setting	Description
name	Unique identifier for the vector template file.
version	Specifies the version of the template to help manage updates and compatibility.

Custom Templates

Setting	Description
name	Defines the name of a custom vector template not included in the core list.
version	Indicates the version of the custom template for management and reference.

Ingstion Pipeline CLI Commands

• build-ingestion-pipelines: Builds ingestion pipelines based on the configuration.

Example:

xdrcli build-ingestion-pipelines --xdr_package_file_path /path/to/xdr_package.yaml --ingestion_pipeline_output_path /path/to/output --xdr_root_log_path /path/to/logs --target production --target_file_path /pathto/.xdr/xdr_targets.yaml

Options:

• --xdr_package_file_path: Path to the xdr_package.yaml configuration file. (Required)

• --ingestion_pipeline_output_path: Path to the directory where ingestion output will be stored. (Required)

• --xdr_root_log_path: Path to the common directory for logs. (Optional, default: current directory)

• --target: Target name for the specific environment. (Optional, default: None)

• --target_file_path: Path to the target's configuration file. (Optional, default: None)

• upload-vector-core-config: Uploads core configuration files to an S3 bucket.

Example:

xdrcli upload-vector-core-config --aws-profile myprofile --log_path /path/to/logs

Options:

• --aws-profile: AWS profile to use for authentication. (Optional, default: None)

• --log_path: Path to the directory where logs will be stored. (Optional, default: current directory)

• upload-vector-templates: Uploads vector templates to a specified S3 bucket.

  Example:

xdrcli upload-vector-templates --aws-profile myprofile --log_path /path/to/logs

Options:

• --aws-profile: AWS profile to use for authentication. (Optional, default: None)

• --log_path: Path to the directory where logs will be stored. (Optional, default: current directory)

• upload-vector-maxmind-mmdb: Uploads MaxMind GeoIP databases to an S3 bucket.

  Example:

xdrcli upload-vector-maxmind-mmdb --aws-profile myprofile --log_path /path/to/logs

Options:

• --aws-profile: AWS profile to use for authentication. (Optional, default: None)

• --log_path: Path to the directory where logs will be stored. (Optional, default: current directory)