Commands Reference

apply-schemas

• Description: Apply and update ClickHouse schemas based on the given configuration.

• Options:

  • --xdr_package_file_path: Path to the configuration file.

  • --schema_directory: Name of the schema config directory in the .xdr_schema/ folder.

  • --do_add_roles: If True, apply HyperSec core roles for RBAC.

  • --do_add_columns: If True, add columns to the schemas.

  • --xdr_root_log_path: Path to common directory.

  • --target: Target name for specific environment we are building with.

  • --target_file_path: Path to the targets configuration file.

  • --schema_update_flag: Run additional schema updates if set to YES. This will force tables to be re-built, consuming substantial compute and memory. It is recommended to plan this carefully and execute one org or schema at a time to avoid system overload.

  • --drop_replacement_table_flag: Flag to determine if replacement table can be deleted or not.

  • --use_replicated_merge_tree: Use Replicated Merge Trees. Turn this off when using single node cluster.

  • --schema_filter_list: Comma separated list to filter schemas to apply.

  • --derived_schema_filter_list: Comma separated list to filter sub-schemas to apply.

  • --schema_filter_wildchar: Filter schemas using wildcards.

  • --derived_schema_filter_wildchar: Filter sub-schemas using wildcards.

  • --org_filter_list: Comma separated list to filter schemas to plan.

  • --max_insert_threads: Max threads for INSERT SELECT query.

  • --min_insert_block_size_rows: Minimum block size for insertion.

  • --verbose: Enable verbose logging.

• Example:


    $ xdrcli apply-schemas --xdr_package_file_path /path/to/xdr_package.yaml --schema_filter_list logs_alerts --sub_schema_filter_list logs_nxlog_windows,logs_nxlog_windows_dns --schema_filter_wildchar logs_alerts* --sub_schema_filter_wildchar logs_alerts* --schema_update_flag YES --drop_replacement_table_flag YES --verbose

build-ingestion-pipelines

• Description: Build ingestion pipelines based on the configuration.

• Options:

  • --xdr_package_file_path: Path to the configuration file.

  • --ingestion_pipeline_output_path: Path to ingestion output.

  • --xdr_root_log_path: Path to common directory.

  • --target: Target name for specific environment we are building with.

  • --target_file_path: Path to the targets configuration file.

• Example:


    $ xdrcli build-ingestion-pipelines --xdr_package_file_path /path/to/xdr_package.yaml --ingestion_pipeline_output_path /path/to/output --xdr_root_log_path /path/to/logs --target production --target_file_path /pathto/.xdr/xdr_targets.yaml

build-schemas

• Description: Builds ClickHouse schemas based on given configuration.

• Options:

  • --xdr_package_file_path: Path to the configuration file.

  • --schema_directory: Path to custom schema directory.

  • --schema_filter_list: Comma seperated list to filter the schemas that get build.

  • --derived_schema_filter_list: Comma seperated list to filter the sub-schemas that get build.

  • --schema_filter_wildchar: Filter the schemas that get build using wildchar.

  • --derived_schema_filter_wildchar: Filter the sub-schemas that get build using wildchar.

  • --no_cluster_declarations_needed: Remove references to the cluster as the SaaS offering abstracts the concept of clusters.

  • --use_replicated_merge_tree: Use Replicated Merge Trees. Turn this off when using single node cluster.

  • --xdr_root_log_path: Path to common directory.

  • --target: Target name for specific environment we are building with.

  • --target_file_path: Path to the targets configuration file.

  • --all: Process all schemas.

  • --only-beats: Scan for all beats schemas.

  • --verbose: Enable verbose logging.

  • --max_workers: Maximum Number of Workers.

  • --opensearch_flag: For OpenSeach Templates

• Example:


    $ xdrcli build-schemas --xdr_package_file_path /path/to/xdr_package.yaml --schema_directory /path/to/custom_schema --schema_filter_list logs_beats_filebeat,logs_alerts --derived_schema_filter_list logs_nxlog_windows,logs_nxlog_windows_dns --schema_filter_wildchar logs_alerts* --derived_schema_filter_wildchar logs_alerts* --no_cluster_declarations_needed --use_replicated_merge_tree --xdr_root_log_path /path/to/logs --target production --target_file_path /pathto/xdr_targets.yaml --verbose --opensearch_flag True

build-sigma

• Description: Generate Sigma rules from YAML files in the specified input directory and save the converted rules to the output directory.

• Options:

  • --xdr_package_file_path: Path to the XDR package configuration file.

  • -i: Input directory containing YAML rules.

  • -o: Output directory for converted rules.

  • --xdr_root_log_path: Path to common directory.

• Example:


    $ xdrcli build-sigma --input_directory /path/to/yaml --output_directory /path/to/output --xdr_root_log_path /path/to/logs

describe-xdr-clickhouse-schema-types

• Description: Lists available meta schemas .

• Options:

  • --xdr_root_log_path: Path to common directory.

  • --xdr_package_file_path: Path to the configuration file.

• Example:


    $ xdrcli describe_xdr_clickhouse_schema_types --xdr_root_log_path /path/to/logs -- xdr_package_file_path /path/to/logs

download-ingestion-pipeline-templates

• Description: Downloads only vector templates that are referenced in xdr_package.yaml and saves them as a ZIP file.

• Options:

  • --xdr_package_file_path: Path to the configuration file.

  • --xdr_root_log_path: Path to common directory.

  • --output_zip: Output ZIP file name.

• Example:


    $ xdrcli download-ingestion-pipeline-templates --xdr_root_log_path /path/to/logs --output_zip vector_templates.zip

download-ingestion-pipeline-templates-all

• Description: Downloads ALL vector templates and saves them as a ZIP file.

• Options:

  • --xdr_package_file_path: Path to the configuration file.

  • --xdr_root_log_path: Path to common directory.

  • --output_zip: Output ZIP file name.

• Example:


    $ xdrcli download-ingestion-pipeline-templates-all --xdr_root_log_path /path/to/logs --output_zip vector_templates.zip

download-meta-schemas

• Description: Downloads meta schemas and saves them as a ZIP file.

• Options:

  • --xdr_package_file_path: Path to the configuration file.

  • --xdr_root_log_path: Path to common directory.

  • --output_zip: Output ZIP file name.

• Example:


    $ xdrcli download-meta-schemas --xdr_root_log_path /path/to/logs --output_zip meta_schemas.zip

init-target

• Description: Initialize or update configuration.

• Options:

  • --target: Target name for specific environment we are building with.

  • --target_file_path: Path to the targets configuration file.

• Example:


    $ xdrcli init-target --target production --target_file_path /pathto/.xdr/xdr_targets.yaml

is-hunt-scheduler-healthy

• Description: Health check command to verify if the application is running.

• Example:

xdrcli is-hunt-scheduler-healthy

kill-all-hunts

• Description: Terminate all activities based on PIDs in the thread_tracking.log and delete the file.

• Options:

  • --xdr_root_log_path: Path to common directory.

  • --hunt_log_path: Path to hunt logs directory.

• Example:


    $ xdrcli kill-all-hunts --xdr_root_log_path /path/to/logs --hunt_log_path /path/to/hunt_logs

kill-hunt

• Description: Kill Hunt

• Options:

  • --xdr_package_file_path: Path to the XDR package configuration file.

  • --kill_pid: PID of the process to stop.

  • --xdr_root_log_path: Path to common directory.

• Example:


    $ xdrcli kill-hunt --xdr_package_file_path /path/to/xdr_package.yaml --kill_pid 12345 --xdr_root_log_path /path/to/logs

list-hunts

• Description: List Hunts

• Options:

  • --look_back_hours: Number of hours to look back for hunts.

  • --xdr_root_log_path: Path to log path directory.

  • --hunt_log_path: Path to hunt logs directory.

• Example:


    $ xdrcli list-hunts --look_back_hours 24 --xdr_root_log_path /path/to/logs --hunt_log_path /path/to/hunt_logs

list-ingestion-templates

• Description: List available ingestion templates.

• Options:

  • --xdr_package_file_path: Path to the configuration file.

  • --xdr_root_log_path: Path to common directory.

• Example:


    $ xdrcli list-ingestion-templates --xdr_root_log_path /path/to/logs

list-meta-schemas

• Description: Lists available meta schemas .

• Options:

  • --xdr_package_file_path: Path to the configuration file.

  • --xdr_root_log_path: Path to common directory.

• Example:


    $ xdrcli list-meta-schemas --xdr_root_log_path /path/to/logs

list-schema-fields

• Description: Lists fields for a given schema and template version.

• Options:

  • --xdr_package_file_path: Path to the configuration file.

  • --schema_name: Schema name.

  • --template_version: Template version.

  • --xdr_root_log_path: Path to common directory.

• Example:


    $ xdrcli list-schema-fields --schema_name logs_alerts --template_version v100_000_000 --xdr_root_log_path /path/to/logs

list-targets

• Description: List all profiles in the configuration.

• Options:

  • --xdr_root_log_path: Path to common directory.

  • --target_file_path: Path to the targets configuration file.

• Example:


    $ xdrcli list-targets --xdr_root_log_path /path/to/logs --target_file_path /pathto/.xdr/xdr_targets.yaml

plan-schemas

• Description: Run plan for schema changes

• Options:

  • --xdr_package_file_path: Path to the configuration file.

  • --xdr_root_log_path: Path to common directory.

  • --target: Target name for specific environment we are building with.

  • --target_file_path: Path to the targets configuration file.

  • --schema_filter_list: Comma seperated list to filter the schemas that get planned.

  • --derived_schema_filter_list: Comma seperated list to filter the sub-schemas that get planned.

  • --schema_filter_wildchar: Filter the schemas that get planned using wildchar.

  • --derived_schema_filter_wildchar: Filter the sub-schemas that get planned using wildchar.

  • --org_filter_list: Comma seperated list to filter the schemas that get planned.

  • --verbose: Enable verbose logging.

• Example:


    $ xdrcli plan-schemas --xdr_package_file_path /path/to/xdr_package.yaml --xdr_root_log_path /path/to/logs --target production --target_file_path /pathto/xdr_targets.yaml --schema_filter_list logs_alerts --derived_schema_filter_list logs_nxlog_windows* --schema_filter_wildchar logs_alerts* --derived_schema_filter_wildchar logs_nxlog_windows* --org_filter_list org321 --verbose

print-default-target

• Description: List all profiles in the configuration.

• Options:

  • --xdr_root_log_path: Path to common directory.

  • --target_file_path: Path to the targets configuration file.

• Example:


    $ xdrcli print-default-target --xdr_root_log_path /path/to/logs --target_file_path /pathto/.xdr/xdr_targets.yaml

print-hunt-parameters

• Description: Print XDR hunt parameters.

• Options:

  • --xdr_package_file_path: Path to configuration file

  • --xdr_root_log_path: Path to log path directory

  • --hunt_log_path: Path to hunt logs directory

  • --target: Target name for specific environment we are building with.

  • --target_file_path: Path to the targets configuration file.

• Example:


    $ xdrcli print-hunt-parameters --xdr_package_file_path /path/to/xdr_package.yaml --xdr_root_log_path /path/to/logs --hunt_log_path /path/to/hunt_logs --target production --target_file_path /pathto/xdr_targets.yaml

run-hunt

• Description: Run XDR hunt.

• Options:

  • --xdr_package_file_path: Path to the configuration file.

  • --hunt_dir: Directory containing hunt configuration describing the hunt actions against specific customers.

  • --hunt_rule_repo_dir: Directory containing target YAML files.

  • --checkpoint_destination: Directory containing hunt configuration describing the hunt actions against specific customers.

  • --checkpoint_timestamp_field: Timestamp field used by each hunt config to checkpoint its position in the logs.

  • --hunt_timeout: Timeout setting for individual Cron tasks.

  • --hunt_num_threads: Number of threads.

  • --xdr_root_log_path: Path to the common directory.

  • --hunt_log_path: Path to hunt logs directory.

  • --target: Target name for specific environment we are building with.

  • --target_file_path: Path to the targets configuration file.

• Example:


    $ xdrcli run-hunt --xdr_package_file_path /path/to/xdr_package.yaml --hunt_dir /path/to/hunt_dir --hunt_rule_repo_dir /path/to/hunt_rules --hunt_timeout 60 --hunt_num_threads 4 --xdr_root_log_path /path/to/logs --hunt_log_path /path/to/hunt_logs --target production --target_file_path /path/to/xdr_targets.yaml

upload-ingestion-pipeline-core-config

• Description: Uploads vector core enrichment configuration files to S3 based on given YAML configuration.

• Options:

  • --xdr_package_file_path: Path to the configuration file.

  • --aws_profile: AWS profile to use for credentials.

  • --xdr_root_log_path: Path to common directory.

  • --target: Target name for specific environment we are building with.

  • --target_file_path: Path to the targets configuration file.

  • --upload_file_target: File target for the upload. Choose either "local" or "S3".

• Example:


    $ xdrcli upload-ingestion-pipeline-core-config --xdr_package_file_path /path/to/config.yaml --aws_profile my-aws-profile --xdr_root_log_path /path/to/logs --target production --target_file_path /pathto/.xdr/xdr_targets.yaml

upload-ingestion-pipeline-maxmind-mmdb

• Description: Uploads MaxMind MMDB files to S3 based on configuration.

• Options:

  • --xdr_package_file_path: Path to the configuration file.

  • --aws_profile: AWS profile to use for credentials.

  • --xdr_root_log_path: Path to common directory.

  • --target: Target name for specific environment we are building with.

  • --target_file_path: Path to the targets configuration file.

  • --upload_file_target: File target for the upload. Choose either "local" or "S3".

• Example:


    $ xdrcli upload-ingestion-pipeline-maxmind-mmdb --aws_profile my-aws-profile --xdr_root_log_path /path/to/logs --target production --target_file_path /pathto/.xdr/xdr_targets.yaml ----upload_file_target local

upload-ingestion-pipeline-templates

• Description: Uploads vector templates to S3 based on configuration.

• Options:

  • --xdr_package_file_path: Path to the configuration file.

  • --aws_profile: AWS profile to use for credentials.

  • --xdr_root_log_path: Path to common directory.

  • --target: Target name for specific environment we are building with.

  • --target_file_path: Path to the targets configuration file.

  • --upload_file_target: File target for the upload. Choose either "local" or "S3".

• Example:


    $ xdrcli upload-ingestion-pipeline-templates --aws_profile my-aws-profile --xdr_root_log_path /path/to/logs --target production --target_file_path /pathto/.xdr/xdr_targets.yaml

view-hunts

• Description: View Hunts

• Options:

  • --view_pid: Process ID (PID) or list of PIDs to fetch hunts for.

  • --xdr_root_log_path: Path to common directory.

  • --hunt_log_path: Path to hunt logs directory.

• Example:


    $ xdrcli view-hunts --view_pid 12345 --xdr_root_log_path /path/to/logs --hunt_log_path /path/to/hunt_logs