> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://hiddenlayer.ferndocs.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://hiddenlayer.ferndocs.com/_mcp/server. # HiddenLayer Azure Sentinel Integration Guide ## Overview This guide provides step-by-step instructions for integrating HiddenLayer with Microsoft Sentinel. Runtime detections will be sent from the HiddenLayer Console to an Azure Log Analytics Workspace for ingestion by Sentinel. ## Architecture ```mermaid graph LR; A[HiddenLayer Console] --> B[Azure Data Collection Endpoint] B --> C[Azure Data Collection Rule] C --> D[Azure Log Analytics Workspace] D --> E[Azure Sentinel] ``` --- ## Prerequisites * Azure subscription with the following permissions: * **Azure Entra ID**: Create and manage app registrations, create client secrets * **Azure Monitor**: Create and manage Data Collection Endpoints (DCE) * **Azure Monitor**: Create and manage Data Collection Rules (DCR) * **Log Analytics Workspace**: Create and manage custom tables * **Access Control (IAM)**: Grant role assignments on Data Collection Rules * Log Analytics Workspace * Azure Sentinel * HiddenLayer Console admin access --- ## Setup Steps ### Step 1: Register Application in Azure Entra ID 1. Navigate to **Azure Portal** → **Microsoft Entra ID** → **App registrations** 2. Click **New registration** 3. Enter a name for the application (e.g., `hl-sentinel-log-analytics-stage`) 4. Select **Accounts in this organizational directory only (Single tenant)** 5. Click **Register** 6. **Save the following values:** * **Application (client) ID** - You'll need this later * **Directory (tenant) ID** - You'll need this later ![App Registration](/_fern-img/0e1035665678a7c4859383b5ad5fbd73a05f92aa9036106369669ea31fc114f4.webp) ### Step 2: Create Client Secret 1. In your app registration, navigate to **Certificates & secrets** 2. Click **New client secret** 3. Enter a description (e.g., `IntegrationToken`) and set expiration (recommended: 180 days or 6 months) ![Add Client Secret](/_fern-img/750cc1f4f3c02b8ed9a885d50be5ea6e33030f320a7614f3a144893eecd5134e.webp) 4. Click **Add** 5. **IMMEDIATELY COPY THE SECRET VALUE** - You cannot view it again after leaving this page ![Client Secret Created](/_fern-img/6ccc4b1ae3e3d99939d7c070679091136211de625b39766ec598cf8682ed32f8.webp) 6. **Save this value securely** - You'll need it for HiddenLayer configuration > **Save Secret Value** > > This value will not be available once you leave this page. If you leave the page without copying the secret value, you must create a new secret. ### Step 3: Create Data Collection Endpoint 1. Navigate to **Azure Portal** → **Monitor** → **Data Collection Endpoints** ![List Data Collection Endpoints](/_fern-img/c9218b6fa375823713f2eb4dfd505c5dace9a48a9ddc5fce859c57f50c825a59.webp) 2. Click **Create** 3. Configure the endpoint: * **Name:** `hl-sentinel-aidr-` (e.g., `hl-sentinel-aidr-stage`) * **Subscription:** Select your subscription * **Resource Group:** Select or create a resource group * **Region:** Choose the region closest to your HiddenLayer deployment ![Create a Data Collection Endpoint](/_fern-img/45be248502e2942f48fa0d91255c86df94ebc31d1337300561703851a0810afa.webp) 4. Click **Review + create** → **Create** 5. Once created, select the endpoint and click **JSON View** 6. **Copy the `dataCollectionEndpointId`** - It looks like: ```text /subscriptions//resourceGroups//providers/Microsoft.Insights/dataCollectionEndpoints/ ``` 7. **Copy the `logsIngestion` URL** - It looks like: ```text https://-..ingest.monitor.azure.com ``` ### Step 4: Create Custom Log Analytics Table 1. Navigate to **Azure Portal** → **Log Analytics workspaces** 2. Select your workspace (where Sentinel is enabled) 3. Navigate to **Settings** → **Tables** ![Log Analytics Tables](/_fern-img/d9ddb37e307ad44e39f1a765fe7f18544d674fb43e34e2b4ffba9b7385c85fa9.webp) 4. Click **Create** → **New custom log (DCR-based)** ![Create Custom Log Menu](/_fern-img/b4cd2d1bc3a5cbe04a90a97c85fb52a361cc70cce4e6389383fd54b6726151af.webp) 5. Configure the custom log: * **Table name:** `HiddenLayerAIDRStage` (without `_CL` suffix) * **Data collection endpoint:** Select the endpoint created in [Step 3](#step-3-create-data-collection-endpoint) * **Data collection rule:** Create new or select existing ![Create Custom Log Basics](/_fern-img/b99365d0e736ec49c46dcea2a687f27ad2c2869c60245e8c47f1855ba91fce7a.webp) 6. Click **Next** 7. Upload the following JSON schema file: ```json { "TimeGenerated": "2024-10-21T00:01:03.123456Z", "conviction_id": "9f891a16-34e6-4e9a-aa5c-22369712e64a", "tenant_id": "80ad8fa2-c1f9-430a-a7b9-ad85a9386d45", "sensor_id": "8d009b0d-75dc-4287-b0d7-b653c51a5ae1", "requester_id": "a_requester_id", "source": "aidr", "detection_category": "A detection category", "attributable_event_id": "8a27bd3e-b7a1-421f-ba43-25f18e595050", "concluding_event_id": "2a4c645e-f08e-41a9-882d-8d22cb4b8e41", "conviction_timestamp": "2024-10-21T00:01:02.123456Z", "mitre": "{ \"Tactic\": { \"UID\": \"AML.TA0001\", \"Name\": \"ML Attack Staging\", \"SrcUrl\": \"https://atlas.mitre.org/tactics/AML.TA001\" }, \"Technique\": { \"UID\": \"AML.T0006\", \"Name\": \"Active Scanning\", \"SrcUrl\": \"https://atlas.mitre.org/tactics/AML.T0006\" } }", "severity": "high", "engine_name": "fuzzy_correlation" } ``` ![Upload Schema](/_fern-img/0768e07f7f3864127c01911bc150ef22bcb7db11998f49888b51f5443205e4b9.webp) 8. Click **Next** → **Create** > **Table and Stream Names** > > Azure will create a table named `HiddenLayerAIDRStage_CL` and a stream named `Custom-HiddenLayerAIDRStage_CL` ### Step 5: Get Data Collection Rule Details 1. Navigate to **Azure Portal** → **Monitor** → **Data Collection Rules** 2. Find and select the data collection rule created in [Step 4](#step-4-create-custom-log-analytics-table) 3. Click **JSON View** 4. **Copy the `immutableId`** - It looks like: ```text dcr-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ``` ![DCR JSON View](/_fern-img/9600cc680268579e4fc82cf77a97bfbeab512a1cd46bc45edafa65c7f78d3d7a.webp) ### Step 6: Grant Permissions to App Registration 1. Stay on the **Data Collection Rule** page 2. Navigate to **Access Control (IAM)** ![DCR IAM](/_fern-img/fd8222c19dce6a6e973a722b84e24a4e763d1ddfcb4bfb61831141e0db330114.webp) 3. Click **Add** → **Add role assignment** 4. Select role: **Monitoring Metrics Publisher** 5. Click **Next** 6. For **Assign access to**, select **User, group, or service principal** 7. Click **Select members** 8. Search for your app registration name (from [Step 1](#step-1-register-application-in-azure-entra-id)) ![Add Role Assignment](/_fern-img/179f62be0089faca5900c89f054b434dca31ad43f67c4d46e987d80d123503d1.webp) 9. Select the application 10. Click **Select** → **Review + assign** → **Review + assign** ### Step 7: Configure HiddenLayer Console 1. Log in to **HiddenLayer Console** 2. Navigate to **Admin** → **Integrations** 3. Find **Azure Sentinel** and click **Configure Integration** (three dots menu) ![HiddenLayer Integration Settings](/_fern-img/5216298f55034c230202b62cf3e762802465d0b5cc9f0c330dba84f01466d449.webp) 4. Enter the following values: | Field | Value | Notes | | ---------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------- | | **Name** | `azure-sentinel-integration` | Any descriptive name | | **Azure Tenant ID** | `` | From Step 1 | | **Azure Client ID** | `` | From Step 1 | | **Azure Client Secret** | `` | From Step 2 (the secret value, not the secret ID) | | **Data Collection Endpoint** | `https://..ingest.monitor.azure.com` | From Step 3 (logsIngestion URL) | | **Resource ID** | `/subscriptions//resourceGroups//providers/Microsoft.Insights/dataCollectionEndpoints/` | From Step 3 (dataCollectionEndpointId) | | **Data Collection Rule ID** | `dcr-xxxxxxxx...` | From Step 5 (immutableId) | | **Azure Table Names** | `Custom-HiddenLayerAIDRStage_CL` | The stream name with "Custom-" prefix and "\_CL" suffix | 5. Click **Submit** --- ## Configuration Summary ### Key Configuration Mapping **What you create in Azure Portal:** * Table name: `HiddenLayerAIDRStage` (no suffix) **What Azure automatically creates:** * Log Analytics table: `HiddenLayerAIDRStage_CL` (adds `_CL` suffix) * DCR stream: `Custom-HiddenLayerAIDRStage_CL` (adds `Custom-` prefix and `_CL` suffix) **What you configure in HiddenLayer:** * Table name: `Custom-HiddenLayerAIDRStage_CL` (the full stream name) ### Critical Field Clarifications | HiddenLayer | Correct Value | Common Mistake | | --------------------------- | -------------------------------------- | --------------------------------------------------------- | | **Resource ID** | Data Collection Endpoint ID (from DCE) | Using DCR Resource ID instead | | **Data Collection Rule ID** | DCR immutableId (`dcr-xxx` format) | Using full ARM path | | **Azure Table Names** | `Custom-HiddenLayerAIDRStage_CL` | Using `HiddenLayerAIDRStage_CL` or `HiddenLayerAIDRStage` | --- ## Verification (Wait 5-10 minutes after configuration) ### Step 1: Create an AI Runtime Security detection 1. In HiddenLayer Console, navigate to **LLM Sandbox** 2. Select **OWASP Scenarios** -> **LLM01: Prompt Injection** 3. Turn on **Block Prompt Injection** in the **Policy Settings** 4. Send the prompt 5. You should see a message saying the prompt was blocked ### Step 2: Check Azure for Errors Run this query in Log Analytics: ```kusto DCRLogErrors | where TimeGenerated > ago(30m) | where _ResourceId contains "" | order by TimeGenerated desc ``` **Expected result:** No errors **If errors exist:** See Troubleshooting section below ### Step 3: Verify Data Ingestion Run this query in Log Analytics: ```kusto HiddenLayerAIDRStage_CL | where TimeGenerated > ago(1h) | order by TimeGenerated desc | take 10 ``` **Expected result:** Conviction data appears in the table --- ## Testing the Integration If you want to manually test the Azure configuration before HiddenLayer sends data, use this Python script: ### Test Script Save as `test_integration.py`: ```python #!/usr/bin/env python3 import datetime import json from azure.identity import ClientSecretCredential from azure.monitor.ingestion import LogsIngestionClient # Configuration - UPDATE THESE VALUES tenant_id = "YOUR_TENANT_ID" client_id = "YOUR_CLIENT_ID" client_secret = "YOUR_CLIENT_SECRET" dce_endpoint = "https://YOUR_DCE_ENDPOINT.eastus2-1.ingest.monitor.azure.com" dcr_immutable_id = "dcr-YOUR_IMMUTABLE_ID" stream_name = "Custom-HiddenLayerAIDRStage_CL" # Sample conviction data sample_log = [{ "TimeGenerated": datetime.datetime.utcnow().isoformat() + "Z", "conviction_id": "test-" + datetime.datetime.utcnow().strftime("%Y%m%d%H%M%S"), "tenant_id": "test-tenant", "sensor_id": "test-sensor", "requester_id": "test-requester", "source": "aidr", "detection_category": "Test Detection", "attributable_event_id": "test-attr", "concluding_event_id": "test-concl", "conviction_timestamp": datetime.datetime.utcnow().isoformat() + "Z", "mitre": json.dumps({"Tactic": {"UID": "AML.TA0001", "Name": "Test"}}), "severity": "high", "engine_name": "test_engine" }] print(f"Testing Azure Sentinel integration...") print(f"Stream: {stream_name}") # Authenticate credential = ClientSecretCredential(tenant_id, client_id, client_secret) client = LogsIngestionClient(endpoint=dce_endpoint, credential=credential, logging_enable=True) try: response = client.upload( rule_id=dcr_immutable_id, stream_name=stream_name, logs=sample_log ) print("✅ Test data sent successfully!") print(f"\nWait 5-10 minutes, then run this query in Log Analytics:") print(f"HiddenLayerAIDRStage_CL | where conviction_id == '{sample_log[0]['conviction_id']}'") except Exception as e: print(f"❌ Error: {e}") ``` Install dependencies: ```bash pip install azure-monitor-ingestion azure-identity ``` Run test: ```bash python3 test_integration.py ``` --- ## Troubleshooting ### Issue 1: Stream Not Configured Error **Error in DCRLogErrors:** ``` The stream HiddenLayerAIDRStage_CL was not configured in the data collection rule ``` **Cause:** Incorrect table name in HiddenLayer Console **Solution:** ✅ **Correct configuration:** * Azure Portal table name: `HiddenLayerAIDRStage` * HiddenLayer Console table name: `Custom-HiddenLayerAIDRStage_CL` ❌ **Incorrect configurations:** * `HiddenLayerAIDRStage_CL` (missing "Custom-" prefix) * `HiddenLayerAIDRStage` (missing both prefix and suffix) * `HiddenLayerAIDRStage_CL_CL` (double suffix - table created incorrectly) ### Issue 2: Authentication Failures **Error:** No data in table **Possible causes:** 1. Incorrect Client ID or Client Secret 2. Expired Client Secret 3. Missing role assignment **Solutions:** **Verify credentials:** ```bash az login --service-principal \ -u \ -p \ --tenant ``` **Check role assignment:** 1. Go to Data Collection Rule → Access Control (IAM) 2. Verify app has "Monitoring Metrics Publisher" role 3. If missing, add the role (see Step 6) **Regenerate secret:** 1. Go to App Registration → Certificates & secrets 2. Create new client secret 3. Update HiddenLayer configuration immediately ### Issue 3: No Data Appearing **Symptom:** No errors in DCRLogErrors, but no data in table **Possible causes:** 1. No convictions being generated 2. Integration not properly activated 3. Timing delay **Solutions:** **Check for detections:** * HiddenLayer Console → AI Runtime Security → Detections * Look for detections in the last 24 hours **Trigger test detection:** * Follow the steps in [Testing the Integration](#testing-the-integration) ## Diagnostic Queries ### Check for Integration Errors ```kusto DCRLogErrors | where TimeGenerated > ago(1h) | where _ResourceId contains "" | summarize Count = count() by Message | order by Count desc ``` ### Check Data Ingestion Rate ```kusto HiddenLayerAIDRStage_CL | where TimeGenerated > ago(24h) | summarize ConvictionsPerHour = count() by bin(TimeGenerated, 1h) | render timechart ``` ### Check Conviction Severity Distribution ```kusto HiddenLayerAIDRStage_CL | where TimeGenerated > ago(24h) | summarize count() by severity | render piechart ``` ### Find Recent High Severity Convictions ```kusto HiddenLayerAIDRStage_CL | where TimeGenerated > ago(24h) | where severity == "high" or severity == "critical" | project TimeGenerated, conviction_id, detection_category, sensor_id | order by TimeGenerated desc ``` --- ## Additional Resources ### Microsoft Documentation * [Data Collection Rules](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-rule-overview) * [Logs Ingestion API](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-ingestion-api-overview) * [Custom Logs](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/custom-logs-overview) --- ## Quick Reference Card Use this for easy reference: | Item | Value | Location | | ------------------- | ------------------------------------------------------ | ----------------------------------------- | | **Tenant ID** | `` | Entra ID → Overview | | **Client ID** | `` | App Registration → Overview | | **Client Secret** | `` | App Registration → Certificates & secrets | | **DCE Endpoint** | `https://..ingest.monitor.azure.com` | DCE → JSON View → logsIngestion | | **DCE Resource ID** | `/subscriptions/.../dataCollectionEndpoints/` | DCE → JSON View → id | | **DCR Rule ID** | `dcr-` | DCR → JSON View → immutableId | | **Stream Name** | `Custom-HiddenLayerAIDRStage_CL` | Use in HiddenLayer Console | | **Table Name** | `HiddenLayerAIDRStage_CL` | Log Analytics table | ---