TZWorks LLC
System Analysis and Programming
www.tzworks.com


TZWorks®
EVTX Fragment eXtension Parser - evtfx

(Version 0.17)



Information about our End User's License Agreements (EULAs)
for software on TZWorks, LLC Website www.tzworks.com

User Agreement

TZWorks LLC software and related documentation ("Software") is governed by separate licenses issued from TZWorks LLC. The User Agreement, Disclaimer, and/or Software may change from time to time. By continuing to use the Software after those changes become effective, you agree to be bound by all such changes. Permission to use the Software is granted provided that (1) use of such Software is in accordance with the license issued to you and (2) the Software is not resold, transferred or distributed to any other person or entity. Refer to your specific EULA issued to for your specific the terms and conditions. There are 3 types of licenses available: (i) for educational purposes, (ii) for demonstration and testing purposes and (iii) business and/or commercial purposes. Contact TZWorks LLC (info@tzworks.com) for more information regarding licensing and/or to obtain a license. To redistribute the Software, prior approval in writing is required from TZWorks LLC. The terms in your specific EULA do not give the user any rights in intellectual property or technology, but only a limited right to use the Software in accordance with the license issued to you. TZWorks LLC retains all rights to ownership of this Software.

Export Regulation

The Software is subject to U.S. export control laws, including the U.S. Export Administration Act and its associated regulations. The Export Control Classification Number (ECCN) for the Software is 5D002, subparagraph C.1. The user shall not, directly or indirectly, export, re-export or release the Software to, or make the Software accessible from, any jurisdiction or country to which export, re-export or release is prohibited by law, rule or regulation. The user shall comply with all applicable U.S. federal laws, regulations and rules, and complete all required undertakings (including obtaining any necessary export license or other governmental approval), prior to exporting, re-exporting, releasing, or otherwise making the Software available outside the U.S.

Disclaimer

The user agrees that this Software made available by TZWorks LLC is experimental in nature and use of the Software is at user's sole risk. The Software could include technical inaccuracies or errors. Changes are periodically added to the information herein, and TZWorks LLC may make improvements and/or changes to Software and related documentation at any time. TZWorks LLC makes no representations about the accuracy or usability of the Software for any purpose.

ALL SOFTWARE ARE PROVIDED "AS IS" AND "WHERE IS" WITHOUT WARRANTY OF ANY KIND INCLUDING ALL IMPLIED WARRANTIES AND CONDITIONS OF MERCHANTABILITY, FITNESS FOR ANY PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT SHALL TZWORKS LLC BE LIABLE FOR ANY KIND OF DAMAGE RESULTING FROM ANY CAUSE OR REASON, ARISING OUT OF IT IN CONNECTION WITH THE USE OR PERFORMANCE OF INFORMATION AVAILABLE FROM THIS SOFTWARE, INCLUDING BUT NOT LIMITED TO ANY DAMAGES FROM ANY INACCURACIES, ERRORS, OR VIRUSES, FROM OR DURING THE USE OF THE SOFTWARE.

Removal

The Software are the original works of TZWorks LLC. However, to be in compliance with the Digital Millennium Copyright Act of 1998 ("DMCA") we agree to investigate and disable any material for infringement of copyright. Contact TZWorks LLC at email address: info@tzworks.com, regarding any DMCA concerns.


About the evtfx Tool (top)

The Windows operating system uses binary XML notation to record various events that occur during the normal course of system usage. In forensics, use of the data recorded in the Window's event log is extremely useful in determining the changes occurring in a machine over time periods of interest. There are many utilities that allow one to extract records from these same logs and package them by time or event to highlight activities that occurred. Occasionally, however, when a parser encounters a corrupted log file, parsing out records can be problematic at best. Therefore, the objective for evtfx was to focus on this area; parsing event logs that were either: (a) corrupted internally either intentionally or accidently by the system (during a crash) or (b) were partially recovered usually from a file carving operation, but still contained gaps in the data.

In order to design a tool that can parse EVTX type records from corrupted or partial fragments of a log, one needs to adjust the way a normal EVTX type parser works. As background, EVTX type logs, as part of their internal design, attempt to minimize the space usage by incorporating a position dependent record structure. Specifically, one record can rely on another record's definitions of field names or data values. This allows a reduction of space and acts as a compression technique when storing many similar events; many of the data labels are the same and will reference a main record containing a shared template definition. While this is very useful under normal circumstances, unfortunately, when the records become corrupted or deleted, this position dependence can cause undefined behavior for any parser relying on traversing the position related pointers. Case in point is the evtwalk tool from TZWorks; it does an excellent job at parsing uncorrupted records, however, does a best guess type parsing for those records that are deleted (using the -inc_slack option, which means to include the slack space during the parsing operation). Furthermore, if for whatever reason, the parser cannot find the template definition, it won't parse records that rely on the missing definition. By extension, any log records that are outside the log file structures usually get missed as well.

The purpose of evtfx is to address the shortfalls of EVTX parsers when it comes to corrupted or partial event logs. As part of its architecture, it is designed to be somewhat independent over the state of the previous records. This, in turn, adds robustness for handling missing/corrupted records.

The downside to the evtfx tool, is it is slower in actual parsing time when compared to evtwalk. In some cases, the evtfx tool misses some of the content of the data. In general, the accuracy of the results of evtfx comes close to that evtwalk. Therefore, if the event log is normal and intact, the evtwalk tool should be the tool of choice. If, however, one wishes to extract corrupted or deleted records from an event log, the evtfx tool is good choice.

Taken to the extreme, the evtfx tool can be used to pull out EVTX type records from any blob of data, assuming the data is uncompressed and unencrypted. The latter condition is very important, since Windows may incorporate NTFS file compression for the event log in question, in which case, the evtfx tool may fail to parse the compressed data. While evtfx makes some attempts to try to perform NTFS file decompression, the results are much more reliable if the raw cluster data is uncompressed.


How to use evtfx (top)

One can display the menu options by typing in the executable's name without parameters. Below is the menu with the various options; details of each option can be found here.

    Usage:

     EVTX Fragment eXtension parser
        evtfx -log <file containing evtx records> [output options]
        evtfx -enumdir <folder> -num_subdirs <#> -filter "*.evtx" -out <results>
      dir <folder> /b /s | evtfx -pipe -out <results>

     Basic output options
      -csv                            = output in CSV format (default)
      -csvl2t                         = log2timeline output
      -bodyfile                       = sleuthkit output
      -sqlite [-db <results] =output results to SQLite db
                             -out <results file>             = output results to a file

     Additional options
      -username <name>                = for -csvl2t output
      -hostname <name>                = for -csvl2t output
      -csv_separator "|"              = use a pipe char for csv separator
      -dateformat mm/dd/yyyy          = "yyyy-mm-dd" is the default
      -timeformat hh:mm:ss.xxx        = "hh:mm:ss" is the default
      -no_whitespace                  = remove whitespace around csv delimiter
      -quiet                          = no progress shown

     Folder Traversing Options
      -pipe                           = pipe files to parse
      -enumdir <dir> -num_subdirs <#> = pull from files from folder

     Query SQLite Options [extract specific records]
      -eventid "id1, id2, .." -db <name> -out <results>

     Experimental options [targeting large data]
      -partition <letter>              = scan Windows volume as raw data
      -image <file> [-offset <#>] [-size <#>] = scan 'dd' image
      -vmdk <vmdk file image>          = scan VMDK image

To process a file with EVTX type event log records, use the -log <file> notation. If sending the output to a file, use the -out <results> option along with the type of desired delimited format (-csv, -csvl2t or -bodyfile). Alternatively, one can send the data to a SQLite database; to do this, use the notation -sqlite -db <database to store results>.

The other options shown above (under Additional options) are the standard ones used in many of the TZWorks tools and contain the same behavior as the other tools. Details about these options are contained in the section on "List of options".

The delimited output formats (with the exception of -csvl2t) generate one record per log entry. The delimiter for all the output file formats can be specified by the user. Some of the more common ones include either a comma character, pipe character or a tab character. If one tells evtfx to send the results to a SQLite database, then the tool will either create a new database if none exists, or, if one exists with the same name, the results will be appended to the existing one. Later on, one can query the database and extract desired records either by using another tool, such as the "DB Browser for SQLite" or using standard SQL commands. There is a rudimentary query option built into evtfx to extract specific event identifiers using the -eventid <#1, #2, …> option.

Processing Multiple Event logs (or log fragments)

If desiring to process many log files in one pass, one can put the artifact logs/fragments into separate a subdirectory and use the pipe option like so:

    dir c:\dump\eventlogs /b /s /a | evtfx -pipe -out results.csv

Alternatively, if not wishing to use the piping option, one can use the -enumdir option along with the sub options -num_subdirs and -filter. This allows one to target a certain level of subdirectories and only files with the desired extension.

    evtfx -enumdir c:\dump\logfrags -num_subdirs 2  -filter "*.bin" -out results.csv

The above command will process all the logs and/or fragments contained in the c:\dump\logfrags folder and subfolders down to two subdirectories with the filename extension ".bin". The output will be stored in the result.csv file.

Extracting Specific Event IDs

If a SQLite database was created to store evtfx results, then one can go back and either query the database using a SQL statement or using the build-in -eventid command. A typical SQL statement to extract a specific event identifier could be (see the evtfx user's guide for the schema used):

    > SELECT * FROM _evtfx_data WHERE event_id IN (4614, 4723, 4724, 4738);

The example above assumes one is in a SQL shell and loaded the database that was generated by evtfx. The following entries in the above command mean:

Alternatively, one could use the -eventid command built into evtfx, and accomplish a similar result. This assumes the security_frag.db was built by evtfx using the -sqlite option.

    evtfx -eventid "4614, 4723, 4724, 4738" -db security_frag.db -out results_pw.csv

The advantage of this last option is one does not need a SQL shell, but can run the extraction directly using the evtfx tool and output the results into an output file with CSV formatting.

To see other categories of event ID combinations associated with system changes, see the section on "Event Category Reports" in the evtwalk user's guide.


List of options (top)

Option Description
-log Identify which event log(s) to operate on. The syntax is: -log <eventlog to analyze>. To operate one more than one at a time, use: -log "<eventlog1> | <eventlog2> | ..."
-csv Outputs the data fields delimited by commas. Since filenames can have commas, to ensure the fields are uniquely separated, any commas in the filenames get converted to spaces.
-csvl2t Outputs the data fields in accordance with the log2timeline format.
-bodyfile Outputs the data fields in accordance with the 'body-file' version3 specified in the SleuthKit. The date/timestamp outputted to the body-file is in terms of UTC. So if using the body-file in conjunction with the mactime.pl utility, one needs to set the environment variable TZ=UTC. The allparams option can be used in conjunction with the bodyfile and tells evtwalk to output all the fields in each record.
-pipe Used to pipe files into the tool via STDIN (standard input). Each file passed in is parsed in sequence.
-enumdir Experimental. Used to process files within a folder and/or subfolders. Each file is parsed in sequence. The syntax is -enumdir <"folder"> -num_subdirs <#>.
-filter Filters data passed in via stdin via the -pipe option. The syntax is filter <"*.ext | *partialname* | ...">. The wildcard character '*' is restricted to either before the name or after the name. An example of using the filter to screen out certain log files could be: dir C:\Windows\System32\winevt\Logs /b /s | evtwalk -pipe -filter "system.evtx|security.evtx|*driverframeworks*" . The first 2 filter terms are explicit files, while the last filter term looks for a partial name that contains driverframeworks.
-no_whitespace Used in conjunction with csv option to remove any whitespace between the field value and the CSV separator.
-csv_separator Used in conjunction with the csv option to change the CSV separator from the default comma to something else. Syntax is csv_separator "|" to change the CSV separator to the pipe character. To use the tab as a separator, one can use the csv_separator "tab" OR csv_separator "\t" options.
-dateformat Output the date using the specified format. Default behavior is -dateformat "yyyy-mm-dd". Using this option allows one to adjust the format to mm/dd/yy, dd/mm/yy, etc. The restriction with this option is the forward slash (/) or dash (-) symbol needs to separate month, day and year and the month is in digit (1-12) form versus abbreviated name form.
-timeformat Output the time using the specified format. Default behavior is timeformat "hh:mm:ss.xxx" One can adjust the format to microseconds, via "hh:mm:ss.xxxxxx" or nanoseconds, via "hh:mm:ss.xxxxxxxxx", or no fractional seconds, via "hh:mm:ss". The restrictions with this option is that a colon (:) symbol needs to separate hours, minutes and seconds, a period (.) symbol needs to separate the seconds and fractional seconds, and the repeating symbol 'x' is used to represent number of fractional seconds. (Note: the fractional seconds applies only to those time formats that have the appropriate precision available. The Windows internal filetime has, for example, 100 nsec unit precision available. The DOS time format and the UNIX 'time_t' format, however, have no fractional seconds). Some of the times represented by this tool may use a time format without fractional seconds and therefore will not show a greater precision beyond seconds when using this option.
-quiet This option suppresses any intermediate progress during a session run
-base10 Ensure all size/address outputs are displayed in base-10 format versus hexadecimal format. Default is hexadecimal format.
-username Option is used to populate the output records with a specified username. The syntax is -username <name to use> .
-hostname Option is used to populate the output records with a specified hostname. The syntax is -hostname <name to use>.
-pipe Used to pipe files into the tool via STDIN (standard input). Each file passed in is parsed in sequence.
-enumdir Experimental. Used to process files within a folder and/or subfolders. Each file is parsed in sequence. The syntax is -enumdir <"folder"> -num_subdirs <#>.
-filter Filters data passed in via stdin via the -pipe option. The syntax is -filter <"*.ext | *partialname* | ...">. The wildcard character '*' is restricted to either before the name or after the name.
-eventid Extract the specified event IDs from the SQLite database (use -db <name> ). If more than one ID is specified, one needs to delimit each ID with a comma. The syntax is -eventid "id1, id2, ...".
-sqlite Outputs the results into a SQLite database. Requires the -db <name> specifier. The format is: -sqlite -db <name>;
-db Specifies which SQLite database to create or to act on. The format is: -db <name> . During creation, one uses the -sqlite command in conjunction with the -db <name>. During query, one uses the -eventid <ids> in conjunction with the -db <name>
-partition Experimental. Extract EVTX records from the specified volume. The format is: -partition <volume letter>.
-image Experimental. Extract EVTX records from the specified file image. This could be a 'dd' image of a volume or a memory dump. As long as the EVTX logs are stored without any compression or encryption, the tool should be able to pull out and parse EVTX records. The format is: -image <file> . If the file is a 'dd' image of a disk one can add the offset/size within the image of the volume to analyze, using the sub options: -offset <#> -size <#>.
-vmdk Experimental. Extract EVTX records from the specified VMDK disk image. The format is: -vmdk <disk image> . One can add the offset/size within the image of the volume to analyze, using the sub options: -offset <#> -size <#>.
-utf8_bom All output is in Unicode UTF-8 format. If desired, one can prefix an UTF-8 byte order mark to the CSV output using this option.

Authentication and License File (top)

This tool has authentication built into the binary. The primary authentication mechanism is the digital X509 code signing certificate embedded into the binary (Windows and macOS).

The other mechanism is the runtime authentication, which applies to all the versions of the tools (Windows, Linux and macOS). The runtime authentication ensures that the tool has a valid license. The license needs to be in the same directory of the tool for it to authenticate. Furthermore, any modification to the license, either to its name or contents, will invalidate the license.

Limited versus Demo versus Full in the tool's output banner

The tools from TZWorks will output header information about the tool's version and whether it is running in limited, demo or full mode. This is directly related to what version of a license the tool authenticates with. The limited and demo keywords indicates some functionality of the tool is not available, and the full keyword indicates all the functionality is available. The lacking functionality in the limited or demo versions may mean one or all of the following: (a) certain options may not be available, (b) certain data may not be outputted in the parsed results, and (c) the license has a finite lifetime before expiring.


Version history (top)


References (top)

  1. Introducing the Microsoft Vista event log format, by Andreas Schuster, 2007
  2. Wikipedia, the free encyclopedia. Event Viewer topic
  3. TechNet, New Tools for Event Management in Windows Vista
  4. Randy Franklin Smith's online encyclopedia.
  5. Windows Event Log Viewer, evtx_view, https://tzworks.com/prototype_page.php?proto_id=4
  6. SleuthKit Body-file format, http://wki.sleuthkit.org/
  7. Log2timeline CSV format, http://log2timeline.net/