TZWorks LLC
System Analysis and Programming
www.tzworks.com
(Version 0.46)
TZWorks LLC software and related documentation ("Software") is governed by separate licenses issued from TZWorks LLC. The User Agreement, Disclaimer, and/or Software may change from time to time. By continuing to use the Software after those changes become effective, you agree to be bound by all such changes. Permission to use the Software is granted provided that (1) use of such Software is in accordance with the license issued to you and (2) the Software is not resold, transferred or distributed to any other person or entity. Refer to your specific EULA issued to for your specific the terms and conditions. There are 3 types of licenses available: (i) for educational purposes, (ii) for demonstration and testing purposes and (iii) business and/or commercial purposes. Contact TZWorks LLC (info@tzworks.com) for more information regarding licensing and/or to obtain a license. To redistribute the Software, prior approval in writing is required from TZWorks LLC. The terms in your specific EULA do not give the user any rights in intellectual property or technology, but only a limited right to use the Software in accordance with the license issued to you. TZWorks LLC retains all rights to ownership of this Software.
The Software is subject to U.S. export control laws, including the U.S. Export Administration Act and its associated regulations. The Export Control Classification Number (ECCN) for the Software is 5D002, subparagraph C.1. The user shall not, directly or indirectly, export, re-export or release the Software to, or make the Software accessible from, any jurisdiction or country to which export, re-export or release is prohibited by law, rule or regulation. The user shall comply with all applicable U.S. federal laws, regulations and rules, and complete all required undertakings (including obtaining any necessary export license or other governmental approval), prior to exporting, re-exporting, releasing, or otherwise making the Software available outside the U.S.
The user agrees that this Software made available by TZWorks LLC is experimental in nature and use of the Software is at user's sole risk. The Software could include technical inaccuracies or errors. Changes are periodically added to the information herein, and TZWorks LLC may make improvements and/or changes to Software and related documentation at any time. TZWorks LLC makes no representations about the accuracy or usability of the Software for any purpose.
ALL SOFTWARE ARE PROVIDED "AS IS" AND "WHERE IS" WITHOUT WARRANTY OF ANY KIND INCLUDING ALL IMPLIED WARRANTIES AND CONDITIONS OF MERCHANTABILITY, FITNESS FOR ANY PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT SHALL TZWORKS LLC BE LIABLE FOR ANY KIND OF DAMAGE RESULTING FROM ANY CAUSE OR REASON, ARISING OUT OF IT IN CONNECTION WITH THE USE OR PERFORMANCE OF INFORMATION AVAILABLE FROM THIS SOFTWARE, INCLUDING BUT NOT LIMITED TO ANY DAMAGES FROM ANY INACCURACIES, ERRORS, OR VIRUSES, FROM OR DURING THE USE OF THE SOFTWARE.
The Software are the original works of TZWorks LLC. However, to be in compliance with the Digital Millennium Copyright Act of 1998 ("DMCA") we agree to investigate and disable any material for infringement of copyright. Contact TZWorks LLC at email address: info@tzworks.com, regarding any DMCA concerns.
elmo is a prototype command line utility to assist the analyst in pulling message table data from providers with the objective of integrating these messages to events that are logged in the Windows event log.
The Windows event log conserves space using a number of mechanisms. One way is to reference the provider for each event along with unique event information in the log and store the more common information in a resource binary. The term provider, as used here, is the source of the event that was generated and is recorded in the event log. This can be one of the running services, drivers, or applications. Reconstruction of the complete message for an event that is logged therefore requires one to pull the common information from the resource that houses the provider's infomation. elmo is a utility to help the analyst do this.
One can examine the various providers for the event log by looking at the System hive in the Windows registry, and for newer operating systems, the additional information in the Software hive. Each provider, in turn, points to one or more PE (portable executable) files that contain an embedded table of messages (referred to by Microsoft as a MESSAGETABLE). Within the table of messages, each item in the table equates to an event identifier which is referenced when logging events in the event log. In this way, boiler plate phrases or sentences can be offloaded to the MESSAGETABLE resources, and only the unique values that populate the data in the message need to be stored in the log itself.
These dependencies are integrated in a seamless manner when analyzing event logs using Microsoft tools and when on the same target machine that the log file was on. Doing it offline however, or using a different tool to parse the event log, can be problematic, since understanding the dependencies can be error prone and most of the techniques to extract the entry from the appropriate message table tend to be manual. To complicate the process further, PE files with embedded MESSAGETABLE's can change from one operating system to the next. As an example, the MESSAGETABLE from the PE file msaudite.dll, which is the main one for Security event auditing, differs from WinXP to Win7 to Win8. While some of the events IDs match the messages across the operating systems, some do not. This is also the case when considering different language types. Integrating all these MESSAGETABLES across differing operating systems is thus a difficult task if tried to do manually.
When architecting elmo, our objective was to have a self-contained way to pull all the appropriate message tables from a target box and archive this data into a database that allows for easy retrieval. SQLite was chosen as the database engine, since it is lightweight, portable and ubiquitous across various operating systems.
elmo fills a need in the Windows event log parsing and reconstruction of logged events. At this point in time, however, since it is still new, it should be considered prototype in nature.
The screen shot below shows all the options available.
Usage:
Create the SQLite msgtable database
elmo -livesys -db <dbname> = from live system
elmo -partition <letter> -db <dbname> = from system volume
elmo -image <name> -db <dbname> = from image
elmo -sys <hive> -sw <hive> -dir <folder> -db <name> = from these files
Create Options
-lcid "de-DE, fr-FR, ja-JP" = add lang ids
-only_security = only security provider msgs
-only_system = only system provider msgs
-only_application = only application provider msgs
Add Resource to SQLite msgtable database
elmo -resource <pe file> -db <name> [-provider <name> | -guid <guid>]
Query the SQLite database
elmo -db <name> -id <#> -provider <name> = extract msg from db
elmo -db <name> -id <#> -guid <name> = extract msg from db
elmo -db <name> -msgtable <table name> = dump msgtable
elmo -db <name> -metadata = dump metadata table
elmo -db <name> -providers = dump provider table
elmo -db <name> -tables = dump table names
elmo -db <name> -create_stats = dump db creation stats
elmo -db <name> -errors = dump provider errors
Append event msgs to evtwalk csv data
elmo -db <name> -src <file> -dst <file> = extract msg from db
Append Options (evtwalk csv data)
-os [winxp|vista|win7|win8|win8.1|win10] = os for eventlog [suggestion]
-msg [raw | debug] = how to integrate msgs
The options are grouped into categories that equate to three basic functions:
To make elmo handle a number of different use-cases, we designed the tool to pull data from:
For some of the database creation use-cases, elmo needs to run at administrative privileges. Once the database is created, elmo can be used for event log message re-creation and can be run at the typical user privileges.
The simplest way to create an elmo database, is to just run elmo on the target box and select the -livesys option, as shown below:
elmo64 -livesys -db test.db
From the above command, elmo will scan all the providers identified by the System hive (and depending on OS version, the Software hive as well) of the Windows registry. From each provider found, it will then parse each of the PE resources that were identified and extract the message table data. The provider, PE resource stats, and message table data is then archived into the database.
One can use SQLite tools to query the database, or one can use the built-in elmo commands to enumerate any of the tables, their contents, or to search for matching messages given an event ID and its associated provider. Depending on the mixture of PE resources used during the elmo database creation phase, multiple candidate messages could be displayed for a given Event ID and Provider.
Below is an example where we created one database from both Win7 and WinXP images and queried an event identifier of 615 from the Security provider. The PE resource file used for both Win7 and WinXP is msaudite.dll. While most have similar ID messages, some do not, like that in the example below:
elmo64 -db test.db -id 615 -provider security
Event ID: 0x00000267
Event code: 615
Severity: Success
LogType: security
Provider: security
SrcMsgFile: msaudite.dll
TypeMsgFile: Category; Event
LCID: en-US
Min OS: Win7 or Win2008/R2 <---
Message: %1 <---
Event ID: 0x00000267
Event code: 615
Severity: Success
LogType: security
Provider: security
SrcMsgFile: msaudite.dll
TypeMsgFile: Category; Event
LCID: en-US
Min OS: WinXP <---
Message: IPSec Services: %t%1%n <---
In this case, the first message is just the string "%1", and the second is the message "IPSec Services: %t%1%n". The "%1" is the way Microsoft defines a placeholder for a string argument. The data for the string argument is taken from the parsed event log and substituted into the message. The other variables, %t and %n are for a tab character and line feed character, respectively. So, while this example has a very short message, it was shown to enlighten the user that the combination of Provider and Event ID does not necessarily yield unique messages and depends on how the PE resource message table was constructed. As operating systems are enhanced, the names of many of the system PE resources stay the same, but their respective message tables may or may not change.
evtwalk is a TZWorks command line tool for parsing Windows event logs and outputting the results in a CSV fashion. While evtwalk pulls all the data available from an event log, it doesn't try to locate and extract the message table from the PE resource. To fill that need, we therefore added the functionality into elmo. So, if one has the updated version of evtwalk (v0.30 or later), one can take the CSV output from evtwalk and pass it into elmo.
To invoke this option, one would use the following options:
elmo64 -db test.db -src "evtwalk_parsed_file.csv" -dst "results.csv"
For the command above, elmo will analyze the source (-src) CSV data and extract certain fields (like the provider and event ID for each record) and look up an associated message table. After this, elmo will perform argument substitution by taking the parameter data from the source CSV file and generate a message which then gets appended to the CSV data. To ensure there is no corruption with the original CSV data, a new CSV file is created (-dst), which is identical to the original evtwalk CSV file inputted, but adds a couple of extra fields to each record. These fields include: (a) message translation and (b) task category translation.
This option should be considered experimental, since a number of conditions need to be satisfied for the above process to work. First, it assumes that resulting CSV file produced by evtwalk has all the fields needed by elmo populated. Basically this means that if you run evtwalk in a mode that does not filter any fields, then it should work. If filtering is used and the required fields needed by elmo are filtered out, then the results will be unknown. Second, the CSV file needs to have the required field header names and argument syntax for elmo to extract the proper field during its analysis. If one is using at least version 0.31 of evtwalk, then this latter requirement is satisfied
For the SQLite option to work, elmo makes the of the SQLite library. If one is unfamiliar with SQLite, the official SQLite website is http://www.sqlite.org/. It has documentation and details on everything one would ever want to know about SQLite. Starting with v0.24 we have compiled in the static library into elmo, so everything is self-contained.
| Option | Description |
|---|---|
| -livesys | Create a database from the system volume elmo is being run on. The format is: -livesys -db <resulting db> |
| -partition | Create a database from a mounted image from some offline system volume. The format is: -partition <drive letter containing mounted system volume> -db <resulting db> |
| -image | Create a database from an image of a system volume. The image needs to be a monolithic file in 'dd' format. The format is: -image <file with dd image> -db <resulting db> |
| -sys -sw -dir |
Create a database from this specific System registry hive and/or Software hive. Use the PE resources identified in directory specified. The format is: -sys <system hive> -sw <software hive> -dir <system32 directory> -db <resulting db> |
| -id -provider |
Query the message that matches this event ID and provider. The format is: -id <event ID> -provider <name of provider> -db <db to query> |
| -msgtable | Dump the table data for a specific MESSAGETABLE resource. The format is: -msgtable <table name> -db <db to query> |
| -metadata | Dump the metadata table. The format is: -metadata -db <db to query> |
| -providers | Dump the providers table. The format is: -providers -db <db to query> |
| -tables | Dump the tables used in the elmo database. The format is: -tables -db <db to query> |
| -create_stats | Dump the creation table stats. The format is: -create_stats -db <db to query> |
| -errors | Dump the provider errors. The format is: -errors -db <db to query> |
| -lcid | Include the following language IDs in the MESSAGETABLE resource extraction during the database create option. The format is: -lcid “de-DE, fr-FR, ja-JP, en-US, …” [any of the database creation options] |
| -only_security | Only pull security providers during the extraction of MESSAGETABLE resources during the database create option. The format is: -only_security [any of the database creation options] |
| -only_system | Only pull system providers during the extraction of MESSAGETABLE resources during the database create option. The format is: -only_system [any of the database creation options] |
| only_application | Only pull application providers during the extraction of MESSAGETABLE resources during the database create option. The format is: -only_application [any of the database creation options] |
| -src -dst |
The -src option specifies the CSV file containing the output of evtwalk parsed data. The -dst option specifies where you want the resulting data to put. -src <evtwalk CSV file> -dst <results file> -db <db to use for message tables> |
| -os | Target the following Windows operating system event log message tables (if possible). The format is: -os [winxp|vista|win7|win8|win8.1|win10] |
| -msg | Option for src / -dst for parsing evtwalk CSV output, to tell how you want the messages to be integrated into the final output. raw = show raw message with no parameter substitution; debug = provide parameter substitution along with original argument string. Default mode does parameter substitution without the argument string. If desiring default mode, don't use this option. The format is -msg [raw | debug] |
This tool has authentication built into the binary. The primary authentication mechanism is the digital X509 code signing certificate embedded into the binary (Windows and macOS).
The other mechanism is the runtime authentication, which applies to all the versions of the tools (Windows, Linux and macOS). The runtime authentication ensures that the tool has a valid license. The license needs to be in the same directory of the tool for it to authenticate. Furthermore, any modification to the license, either to its name or contents, will invalidate the license.
The tools from TZWorks will output header information about the tool's version and whether it is running in limited, demo or full mode. This is directly related to what version of a license the tool authenticates with. The limited and demo keywords indicates some functionality of the tool is not available, and the full keyword indicates all the functionality is available. The lacking functionality in the limited or demo versions may mean one or all of the following: (a) certain options may not be available, (b) certain data may not be outputted in the parsed results, and (c) the license has a finite lifetime before expiring.