HyperSec XDR Data Engine

XDE Build System Release Notes

XDE Release Notes

v0.3.10 (December 23, 2024)

Schema Data Type Mismatch:

v0.3.9 (December 5, 2024)

Overview

v0.3.8 (November 19, 2024)

Overview

Fixed skip index logic into the schema-ch class, addressing issues with columns names having "." DOT in the name.

v0.3.7 (November 19, 2024)

Overview

Reintroduced skip index logic into the schema-ch class, addressing issues caused by major schema refactoring.

Key Changes

  1. Index Type applycation for clickhouse schemas * enabling derived schemas index_type override the meta schema index_typ schema-ch Updates*

v0.3.6 (November 18, 2024)

Overview

Reintroduced skip index logic into the schema-ch class, addressing issues caused by major schema refactoring.

Key Changes

  1. schema-ch Updates * Restored skip index logic in the schema-ch class to handle skipped indexes effectively.
  2. Unit Tests * Enhanced unit tests to verify the creation and application of skip indexes.
  3. GitLab CI Pipeline * Temporarily disabled documentation generation logic to focus on building the release and package.
  4. Common Schema V001_001_005 * Added a skip index for the timestamp column to improve query performance.

v0.3.5 (November 13, 2024)

Overview: This release improves asynchronous task management, error handling, and default settings for CLI commands, with a focus on CronJob and HuntScheduler functionalities. Checkpoint structure has also been updated for better JSON serialization.

Key Changes:

  1. Asynchronous Task Management Fixes: * Addressed unawaited coroutine warnings by ensuring tasks are properly awaited. * Updated task scheduling to prevent blocking and ensure proper handling of async functions.
  2. Checkpoint Serialization: * Removed non-serializable xdr_logger from successful_checkpoints to enhance JSON compatibility.
  3. Packaged pipeline_template.yaml: * Bundled the pipeline_template.yaml file for access via the package's resource files, making it available in the package distribution.

v0.3.4 (November 15, 2024)

Changes:

v0.3.3 (November 5, 2024)

Changes:

Deployment Steps

  1. Prepare and Test Pipeline Updates: * Replace update-schema with apply-schema and use --schema_update_flag=YES in all relevant pipelines. * Validate these changes in integration and ghostburner environments to ensure they trigger schema updates correctly.
  2. Verify OpenSearch Flag Functionality: * Confirm that setting opensearch_flag=False prevents the creation of OpenSearch templates as expected.
  3. Run Schema and Compatibility Tests: * Confirm that schema duplication issues (e.g., event_hash) are resolved. * Run all tests to verify alignment with the updated schema logic and OpenSearch settings.
  4. Upgrade Common Schema: * Update the xdr_package.yaml configuration files to use the new common schema version common/v001.001.005. * Deploy the upgraded OpenSearch framework (v3.2.6) to dev-test for initial validation.
  5. Documentation Generator and Fetching Sigma Rules: * Run the document_generator.py script to automatically generate CLI documentation. * Build Sigma rules from the Sigma open-source repository, placing them in the post_build_artefacts/sigma_rules folder for reference.
  6. Dependency and Environment Preparation: * Upgrade poetry to the latest version for improved dependency management.

v0.3.2 (October 31, 2024)

Changes

v0.3.1 (October 31, 2024)

Changes:

v0.3.0 (October 30, 2024)

Changes:

🚀 Beats Optimisations:

Deployment Steps:

  1. Update all xdr_package.yaml global settings:
  2. Update the xdr_package.yaml from xdr_schema_template to meta_schema_paths
  3. Update the xdr_package.yaml from xdr_schema_template_version to meta_schema_version
  4. Update the xdr_package.yaml from sub_schema_file_path to derived_schema_paths
  5. Update the xdr_package.yaml to reference the ingestion pipeline transforms mappings and templates to below format

  6. Update the xdr_package.yaml schemas to use the terms meta and derived schemas.

    ```yaml global_settings: ingestion_template_s3_bucket: development-config-bucket-afterburner schema_common_version: v001.001.004 schema_output_path: ../.xdr_schema_output/ ingestion_pipeline_output_path: ../.xdr_ingestion_pipelines_output upload_ingestion_template_output_path: ../.xdr_ingestion_template_output schema_versioning_output_path: .migrations/xdr_schema_versioning default_profile: ghostburner target_path: ~/.xdr/xdr_targets.yaml ingestion_pipeline_transform_mappings_paths: standard: - ../post_build_artefacts/ingestion_pipeline_enrichment_standard - ../post_build_artefacts/ingestion_pipeline_enrichment_standard_custom geoip: - ../post_build_artefacts/ingestion_pipeline_enrichment_geoip ingestion_pipeline_templates_paths: vector: - ../post_build_artefacts/ingestion_pipeline_templates meta_schema_paths: ../post_build_artefacts/xdr_meta_schemas_package derived_schema_paths: ../_sample_xdr_config/sample_schemas/openmss_schemas/stable_schemas

    schemas: logs_beats_filebeat_activemq: name: logs_beats_filebeat_activemq meta_schema: logs_beats_filebeat.csv meta_schema_version: v001.000.003 derived_schema_file_path: logs_beats_filebeat/logs_beats_filebeat_activemq.csv derived_schema_ttl: 90 ``` 7. All meta schemas, pipeline config files and templates are delivered as external artefacts: 8. Download the artefacts from HyperSec gitlab account and reference them in a single directory with the right sub directory structures.

  7. Update all xdr_package.yaml files to reference .csv as there core templates:

  8. Update the xdr_package.yaml files from logs_beats_filebeat.json to logs_beats_filebeat.csv

    yaml logs_beats_filebeat: name: logs_beats_filebeat meta_schema: logs_beats_filebeat.csv # Change here, swapped from logs_beats_filebeat.json meta_schema_version: v001.000.003 derived_schema_ttl: 90

Download Artefacts

Here are the downloadable links for the package assets:

Name Size Created Link
xdr_data_engine-0.2.59.tar.gz 40.59 MiB 1 week ago Download
xdr_data_engine-0.2.59-py3-none-any.whl 41.18 MiB 1 week ago Download

v0.2.59 (October 17, 2024)

Changes:

🚀 Build Schemas:

v0.2.58 (October 16, 2024)

Changes:

🚀 Build Schemas:

v0.2.57 (October 16, 2024)

Changes:

🚀 Build Schemas:

v0.2.56 (October 15, 2024)

Changes:

🚀 Fixed Failing Schema Unit tests:

v0.2.55 (October 14, 2024)

Changes:

🚀 Enhancements to Common Schema and CLI Commands:

🚀 New CLI Commands Introduced:

v0.2.55 (14/10/2024)

Changes:

🚀 Update Schemas:

v0.2.54 (12/10/2024)

Changes:

🚀 Update Schemas:

v0.2.53 (30/09/2024)

Changes:

🚀 New Features:

v0.2.52 (20/09/2024)

Changes:

🚀 Vectors:**:

🛠 Enhancements:

v0.51 (09/18/2024)

Changes:

  • API Development:
  • Introduced a new FastAPI application to manage schema and hunt-related operations. This includes a unified API interface for schema creation and hunt execution, previously handled via CLI commands.
  • New Features:
  • Schema Management using FastAPI:
    • Added all schema CLI commands into the FastAPI application with dedicated endpoints.
  • Hunt Framework:
    • Integrated hunt management into the FastAPI application with dedicated endpoints for scheduling and executing hunts.
  • UI Support:
    • Built an API class to support a user interface, enabling interaction with schema and hunt operations through RESTful endpoints.
  • Prometheus Metrics and Logging:
    • Exposed metrics and logs for Prometheus, allowing for better monitoring and performance tracking of the FastAPI applications.
  • Health Checks:
    • Implemented health checks to ensure the applications can access essential resources like ClickHouse and required configuration files.
  • Containerization:
    • Containerized both the schema management and hunt management FastAPI applications, facilitating local testing and deployment.

gitbook/RELEASE_NOTES.md

v0.2.50 (18/09/2024)

Changes:

🚀 New Features:

🛠 Enhancements:

v0.2.49 (09/09/2024)

Changes:

  • CLI Commands: Changed the names of CLI commands from build-schemas to plan-schemas and update-schemas for improved clarity.
  • New Features: - Incorporated --all and --only-beats flags in build-schemas command: - --all: Processes all schemas without filtering. - --only-beats: Scans and processes only beats schemas.
  • Enhancements: - Updated plan-schemas to handle logging more effectively when databases or schemas are missing. - Added new unit tests and updated existing ones to cover the latest functionality and improvements. - Incorporated multi-threading using ThreadPoolExecutor from concurrent.futures in build and apply schemas to improve performance and scalability.

v0.2.48 (09/09/2024)

Enhancements:

Additional Improvements:

v0.2.47 (08/09/2024)

Changes:

  • GITLAB pipline: Gitlab pipeline container registry version fix. The current pipeline takes poetry version from the feature branch , we want to pull latest version from main branch for coker image pushing to container registry .

v0.2.46 (08/09/2024)

Changes:

  • GITLAB pipline: Gitlab pipeline release version fix. The current pipeline takes poetry version from the feature branch , we want to pull latest version from main branch.

v0.2.45 (07/09/2024)

Changes:

  • GITLAB pipline: Gitlab pipeline package distribution fix.

v0.2.44 (31/08/2024)

Changes:

  • CLI Commands:
  • plan-schemas: Introduced a new command to generate a detailed plan for schema changes before applying them. This allows users to review and verify schema modifications in advance.
  • update-schemas: Added a command to apply all schema updates across the ClickHouse environment, streamlining the process of implementing changes based on the schema changes.

v0.2.43 (25/08/2024)

Changes:

  • Vector Templates: Refreshed Vector Templates from XDR config to xdr-data-engine in src/resources/vector_templates/** using package_vector.py python job.

origin/main

v0.2.42 (14/08/2024)

Changes:

  • schema builder: Updated a logic in schema builder to create all schemas for all customers ie removed the logic of customers subscribing to schemas.

v0.2.41 (14/08/2024)

Changes:

  • cronjob: Added a logic in hunt config to handle the orchestration of multiiple CRON's for multiple Customers. Updated unit tests to support this change.

v0.2.37 (13/08/2024)

Changes:

  • modify-orderby-table and modify-index-table: Added 2 new CLI commands : modify-orderby-table is to add or remove the columns from order by and primary statements in clickhouse table. modify-index-table is to add or remove the indexes from the clickhouse table.

v0.2.36 (11/08/2024)

Changes:

  • Schema Upgrade-Skip Indexes: Updated logic for making primary key columns same as order by columns.
  • Common Header v1.1.3: Removed indexing from event_hash column

v0.2.35 (10/08/2024)

Changes:

  • Schema Upgrade-Skip Indexes: A new index_type was added to the schema definition language that will allow users to define skip indexs.
  • Common Header v1.1.3: A series of new fields added to the common header.

v0.2.34 (04/08/2024)

v0.2.33 (04/08/2024)

v0.2.32 (04/08/2024)

Changes:

  • Hunt Checkpointing: Upgraded the HuntCheckpointManager class to create unique xdr_audit databases and detection_checkpoint tables for each checkpoint test run.
  • Hunt Checkpointing: Upgraded the HuntCheckpointManager class to write the previous_successful_checkpoint to make it easy to validate checkpoint windows
  • Cron job: Simplified and reduced the function scoping for executing cron jobs on threads and within asyncio loops. Fixed hunt threading to ensure it displays correctly for each hunt per customer. -diagrams Added Hunt Framework Workflow diagrams, including both drawio and PNG files.

Deployment Steps:

  1. Database Schema Update:

  2. Execute the following SQL command to add the new column to the detection_checkpoint table:

    sql ALTER TABLE xdr_audit.detection_checkpoint ADD COLUMN previous_successful_checkpoint DateTime CODEC(DoubleDelta, LZ4);

v0.2.31 (02/08/2024)

Changes:

  • gitlab-ci: Decouple the CI and CD. The unit tests and stress tests will always run but the package, docker image build and tags/release will only run on manual execution of the pipeline on main branch.

v0.2.31 (2/11/2024)

Changes:

  • gitlab-ci: Decouple the CI and CD. The unit tests and stress tests will always run but the package, docker image build and tags/release will only run on manual execution of the pipeline on main branch.

v0.2.27 (31/07/2024)

Changes:

  • Hunts: Added hunt framework test to validate that the hunt framework is spinning up the right number of threads and effectively isolating the hunts to run in parallel.

    Example Output:

``` - 2024-07-17 01:07:46,691 INFO | Summary: - 2024-07-17 01:07:46,692 INFO | Total threads used: 8 - 2024-07-17 01:07:46,692 INFO | Thread 0: Execution Time: 74.97776647203136 seconds - 2024-07-17 01:07:46,693 INFO | Thread 1: Execution Time: 74.7663262829883 seconds - 2024-07-17 01:07:46,693 INFO | Thread 2: Execution Time: 74.69299309497 seconds - 2024-07-17 01:07:46,694 INFO | Thread 3: Execution Time: 74.62824630603427 seconds - 2024-07-17 01:07:46,695 INFO | Thread 4: Execution Time: 74.55346441699658 seconds - 2024-07-17 01:07:46,696 INFO | Thread 5: Execution Time: 74.42174733499996 seconds - 2024-07-17 01:07:46,696 INFO | Thread 6: Execution Time: 74.31992934900336 seconds - 2024-07-17 01:07:46,697 INFO | Thread 7: Execution Time: 74.24981495796237 seconds>

  • 2024-07-17 01:08:57,786 INFO | Benchmark analysis completed.
  • 2024-07-17 01:08:57,788 INFO | Total threads used: 8
  • 2024-07-17 01:08:57,789 INFO | Total execution time: 565.3543259570142 seconds

  • 2024-07-17 01:08:57,790 INFO | Average execution time per thread: 70.66929074462678 seconds ```

  • Detection Checkpoint: Added a new column query_checkpoint_time to the detection_checkpoint table.

Deployment Steps:

  1. Database Schema Update:

  2. Execute the following SQL command to add the new column to the detection_checkpoint table:

    sql ALTER TABLE xdr_audit.detection_checkpoint ADD COLUMN query_checkpoint_time DateTime CODEC(DoubleDelta, LZ4);

  3. Initial Data Scan:

  4. The first hunt run ( ie no existing checkpoint ) will scan all existing data and populate the new query_checkpoint_time column. This process may take time depending on the volume of data.

  5. Watermark Configuration:

  6. To optimize the checkpointing process and avoid a full data scan every time, we have configured a watermark in the query_checkpoint_time column.

  7. Testing:

  8. Ensured that new checkpointing functionality is correctly implemented and that the watermark setting is functioning as intended.

  9. Monitor:

  10. After deployment, monitor system performance and logs to ensure that the update does not negatively impact operations.

v0.2.22 (16/07/2024)

  • [hunts]
  • Handled merge conflicts from v0.2.18
  • Pushed common Header
  • Fixed gitlab pipeline to run build , package and tags only when merged to main. But still run tests on Merge Request.

v0.2.21 (14/07/2024)

Framework Enhancements:

  • Enhanced [vector-ingestion-pipelines] by integrating new templates and abstracting additional variables into the xdr_package.yaml configuration file.

v0.2.20 (14/07/2024)

  • [schema-executor] Regression tested Schema DataType Changes

v0.2.18.3 (14/07/2024)

  • [cron_job] Passed the scheduled start time of the cron task from JobScheduler to Hunts Checkpoint.

v0.2.18.2 (14/07/2024)

  • [stress-testing] Expanded Stress tests to run on 20 databases integration_test1-integration_test20.

v0.2.18.1 (14/07/2024)

  • [cronjob] Upgraded the multi threading for the hunts down from hunt to the individual customers on the hunt.
  • [hunts] We also passed in the schedule start of that cron task and use that at the checkpoint date time.
  • [tests] updated unit and stress tests.

v0.2.15 (10/07/2024)

Framework Enhancements:

  • [schema-builder] Organisation List moved to the xdr_package_build.yaml.
  • [schema-builder] Added Timestamp_receiver and timestamp_fianlise to the common headers. Upgraded to v001.001.001 to build the two fields.
  • [hunt-runner] Checkpointing bug - not pulling
  • [schema-builder] Organisation List moved to the xdr_package_build.yaml.

Bug Fixes:

  • [hunt-framework] Fixing bug that uses same checkpoint for a across all customers, added customer name to the checkpointing solution.

Deployment Steps:

  • [schema-builder] Migrate your list of organisations into the xdr-package.yaml and specific the schemas that you want to build.

yaml organisations: - org_id: detectionlab cluster_name: '' schemas: - logs_alerts - nx_log_windows_sub_ee - logs_nxlog_windows - logs_syslog - org_id: org321 cluster_name: '' schemas: - logs_alerts - logs_syslog Ensure that the org_id, and schemas fields are updated to reflect your organization’s specific requirements. Cluster can be ignored for SaaS ClickHouse Deployments.