TZWorks LLC
System Analysis and Programming
www.tzworks.com
(Version 0.95)
TZWorks LLC software and related documentation ("Software") is governed by separate licenses issued from TZWorks LLC. The User Agreement, Disclaimer, and/or Software may change from time to time. By continuing to use the Software after those changes become effective, you agree to be bound by all such changes. Permission to use the Software is granted provided that (1) use of such Software is in accordance with the license issued to you and (2) the Software is not resold, transferred or distributed to any other person or entity. Refer to your specific EULA issued to for your specific the terms and conditions. There are 3 types of licenses available: (i) for educational purposes, (ii) for demonstration and testing purposes and (iii) business and/or commercial purposes. Contact TZWorks LLC (info@tzworks.com) for more information regarding licensing and/or to obtain a license. To redistribute the Software, prior approval in writing is required from TZWorks LLC. The terms in your specific EULA do not give the user any rights in intellectual property or technology, but only a limited right to use the Software in accordance with the license issued to you. TZWorks LLC retains all rights to ownership of this Software.
The Software is subject to U.S. export control laws, including the U.S. Export Administration Act and its associated regulations. The Export Control Classification Number (ECCN) for the Software is 5D002, subparagraph C.1. The user shall not, directly or indirectly, export, re-export or release the Software to, or make the Software accessible from, any jurisdiction or country to which export, re-export or release is prohibited by law, rule or regulation. The user shall comply with all applicable U.S. federal laws, regulations and rules, and complete all required undertakings (including obtaining any necessary export license or other governmental approval), prior to exporting, re-exporting, releasing, or otherwise making the Software available outside the U.S.
The user agrees that this Software made available by TZWorks LLC is experimental in nature and use of the Software is at user's sole risk. The Software could include technical inaccuracies or errors. Changes are periodically added to the information herein, and TZWorks LLC may make improvements and/or changes to Software and related documentation at any time. TZWorks LLC makes no representations about the accuracy or usability of the Software for any purpose.
ALL SOFTWARE ARE PROVIDED "AS IS" AND "WHERE IS" WITHOUT WARRANTY OF ANY KIND INCLUDING ALL IMPLIED WARRANTIES AND CONDITIONS OF MERCHANTABILITY, FITNESS FOR ANY PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT SHALL TZWORKS LLC BE LIABLE FOR ANY KIND OF DAMAGE RESULTING FROM ANY CAUSE OR REASON, ARISING OUT OF IT IN CONNECTION WITH THE USE OR PERFORMANCE OF INFORMATION AVAILABLE FROM THIS SOFTWARE, INCLUDING BUT NOT LIMITED TO ANY DAMAGES FROM ANY INACCURACIES, ERRORS, OR VIRUSES, FROM OR DURING THE USE OF THE SOFTWARE.
The Software are the original works of TZWorks LLC. However, to be in compliance with the Digital Millennium Copyright Act of 1998 ("DMCA") we agree to investigate and disable any material for infringement of copyright. Contact TZWorks LLC at email address: info@tzworks.com, regarding any DMCA concerns.
ntfswalk is a command line tool that traverses a specified NTFS volume reading all MFT entries and pulling predefined statistics as it runs.
Originally the engine was designed as a widget for other prototypes to help pull data out from targeted categories of files on NTFS partitions. After successfully using the functionality in other tools, it was determined making a standalone tool would be helpful in debugging and understanding the internals of any NTFS volume. This new tool, coined ntfswalk, is named after its ability to walk an entire NTFS volume and output each MFT entry it encounters.
Designed to work with live NTFS partitions, there is also functionality for traversing NTFS images created with the 'dd' utility (as well as some versions of VMWare VMDK files). There are options to filter on file extension, timestamp range, binary signature, partial filenames and directory contents. For the files found, one can list the summary metadata, extract the header bytes, or extract the entire file contents into a designated directory. Since the engine is Windows API agnostic, there are compiled versions for Windows, Linux, and macOS.
To use this tool, an authentication file is required to be in the same directory as the binary in order for the tool to run.
ntfswalk has a number of command line switches, and for the occasional user, it can be confusing which options can be used together and which cannot.
One can display the menu options by typing in the executable name without parameters. Below is the menu with the various options.
ntfswalk -partition <drive letter> [options]usage: Running 'ntfswalk' on an extracted $MFT file ntfswalk -mftfile <name> [-opts] = source is extract $MFT file Running 'ntfswalk' on a disk/partition image captured w/ a 'dd' type tool ntfswalk -image <file> [-offset <vol>] = dd image Running 'ntfswalk' on a VMWare monolithic virtual volume -vmdk "disk1 | disk2 | ..." = VMWare VMDK disk(s) Running 'ntfswalk' on a live volume ntfswalk -partition <drv letter> [opts] ntfswalk -drivenum <#> [-offset <vol>] = Disk# that is mounted ntfswalk -vss <num> [options] = Volume Shadow parse Filter 'OR' logic options -filter_ext "ext1 | ext2 | ..." = extract using exts -filter_name "name1 | name2 | ..." = extract using partial names -filter_fullname "name1 | name2 | ..." = extract using full names -filter_dir "dir1 | dir2 | ..." = extract using dirs -filter_file "path/file1 | ..." = extract specified files -filter_dir_inode "inode1 | inode2 .." = extract using dir inodes -filter_inode "inode1 | inode2 | ..." = extract specified inodes -filter_sig "mz | hive | evt | sqlite" = extract using signatures -filter_max_size <size> = extract if doesn't exceed size Filter 'AND' logic options -filter_start_time <date time> = format "mm/dd/yyyy hh:mm:ss" -filter_stop_time <date time> = format "mm/dd/yyyy hh:mm:ss" -filter_deleted_files = analyzes only filerecs in $MFT -filter_deleted_files_all = analyzes both $MFT and unalloc -filter_unalloc_clusters = analyze unallocated clusters -filter_all_clusters = analyze $MFT & unalloc clusters Extraction options -action_copy_files <dir> [-raw] = extract file into dir [-raw] = incl slack & skip sparse clusters [-skip_sparse_clusters] = don't incl sparse clusters -action_include_header = extracts 0x20 bytes from file -action_find_strings_in_hdr = find strings in first 0x100 bytes -action_include_clusterinfo = show cluster info -action_include_zoneinfo = show any ADS: Zone info -action_include_ownersid = show owner SID Results file format options -csv = csv format, has most output -csvl2t = log2timeline format -bodyfile = bodyfile format -csvperpath = only one csv entry/pathfile -hashfile "md5 | sha1 | sha256" = display hashes in output General purpose options -out <results file> = output results to this file -hide_dos_fntimes = don't include dos 8.3 fn time -hostname <name> = output hostname -base10 = use base10 vice hex -use_orig_ext = only for [-action_copy_files] -script <file> = use file to express options -mftstart <value> [-mftrange <value>] = only process these inodes -filerecord_offset = output the offset of the filerec -quiet = suppress progress during run -dateformat mm/dd/yyyy = "yyyy-mm-dd" is the default -timeformat hh:mm:ss = "hh:mm:ss.xxx" is the default -pair_datetime = combine date/time into 1 field -no_whitespace = remove whitespace between delims -csv_separator "|" = use a pipe char for separator Experimental option [split processing into multiple instances] -cpu <#instances> -out <file> = merge instances into 1 file [-tempdir <folder>] = use this temp dir for file merge [-separate_files] = don't merge; each instance is separate file -guess_path = for seqnum mismatch
The architecture can be broken up into four main areas: (a) source of the data, (b) filter that can be applied, (c) extraction options, and (d) output format.
Starting with the 'source of the data', ntfswalk can handle various types: (a) an $MFT extracted file, (b) a 'dd' image of a drive or volume, (c) a drive or volume currently mounted or (d) a VMWare monolithic NTFS formatted disk. The options available and their syntax are:
Option | Description |
---|---|
-mftfile | Assuming one has an $MFT file that has been copied from a volume, one can analyze the file. The syntax is: -mftfile <extracted $MFT file>. |
-image | Extract artifacts from a volume specified by an image and volume offset. The syntax is -image <filename> -offset <volume offset> |
-drivenum | Extract artifacts from a mounted disk specified by a drive number and volume offset. The syntax is -drivenum <#> -offset <volume offset> |
-partition | Extract artifacts from a mounted Windows volume. The syntax is -partition <drive letter>. |
-vmdk | Extract artifacts from a VMWare monolithic NTFS formatted volume. The syntax is -vmdk <disk name>. For a collection of VMWare disks that include snapshots, one can use the following syntax: -vmdk "disk1 | disk2 | ..." |
-vss | Experimental. Extract artifacts from Volume Shadow. The syntax is -vss <index number of shadow copy>. Only applies to Windows Vista, Win7, Win8 and beyond. Does not apply to Windows XP. |
The second area is filtering. This defines what files (or MFT entries) are analyzed and displayed to the user. One can filter by deleted files/folders, by various extensions, multiple partial names, and/or binary signatures. Also in this area, one can choose to analyze all 'unallocated clusters' instead of the normal 'allocated clusters', or to pull files from a specified directory. For binary signatures, currently ntfswalk allows one to find: registry hives, event logs, or portable executable files. The options available, and their syntax, are:
Option | Description |
---|---|
-filter_ext | Filter on one or more extensions. The syntax is: -filter_ext "<ext1> | "<ext2> | ...". |
-filter_name | Filter on one or more partial names (case insensitive). The syntax is: -filter_name "<partial name1> | "<partial name2> | ...". |
-filter_deleted_files | Filters on deleted files. This option only looks only at the $MFT file to find the deleted files. |
-filter_fullname | Filter on one or more fully qualified path/filename entries. The syntax is: -filter_fullname "<path\name1> | "<path\name1> | ...". |
-filter_inode | Filter on one or more inodes. The syntax is: -filter_inode "<inode1> | "<inode2> | ...". |
-filter_deleted_files_all | Filters on all deleted files in a volume. Looks at both the $MFT file and unallocated clusters to find deleted files. This option is still experimental, in that when scanning unallocated clusters certain boundary conditions may cause ntfswalk to crash, since various metadata will be corrupted. |
-filter_unalloc_clusters | Analyzes files and directories in a volume. Looks at just unallocated clusters and doesn't analyze the $MFT file. This option is still experimental, in that when scanning unallocated clusters certain boundary conditions may cause ntfswalk to crash, since various metadata will be corrupted. |
-filter_all_clusters | Analyzes files and directories in a volume. Looks at both the $MFT file and unallocated clusters to find files and directories. This option is still experimental, in that when scanning unallocated clusters certain boundary conditions may cause ntfswalk to crash, since various metadata will be corrupted. |
-filter_sig | Filter on one or more built in signatures. mz = exes, dlls, and driver files, hive = registry hives, evt = event logs (both .evt and .evtx types), and sqlite = SQLite v3 databases. The syntax is: -filter_sig "mz|hive|evt|sqlite" to filter on all signatures associated with mz, hive, evt and sqlite. |
-filter_dir | Filter on one or more directories. Will filter the first level down in default mode. The syntax is: -filter_dir "<dir1> | <dir2> | ...". To filter beyond one directory, use a wildcard '*' and a number pair to specify the number of directories to scan. For example: -filter_dir "c:\$Recycle.Bin\*3" to enumerate 3 directories down in the recycle bin directory. Use this wildcard carefully, in that if too many directories are specified, ntfswalk will need to take alot time to compute all the subdirectories prior to processing the data. It is usually much faster to do an entire drive in default mode than to specify a deep directory scan. |
-filter_dir_inode | Filter one or more directory inodes. Will filter the first level down. The syntax is: -filter_dir_inode "<inode1> | <inode2> | ...". |
-filter_start_time | Filter on a time start. Time is specified in UTC format using the following notation: mm/dd/yyyy hh:mm:ss, or without time, mm/dd/yyyy. The syntax is: -filter_start_time <date>. |
-filter_stop_time | Filter on a time stop. Time is specified in UTC format using the following notation: mm/dd/yyyy hh:mm:ss, or without time, mm/dd/yyyy. The syntax is: -filter_stop_time <date>. |
-filter_max_size | Filter on the size of the file, so it doesn't exceed the max specified here. Only applies to the 'unnamed' data attribute. |
-filter_min_size | Filter on the size of the file, so it at least has the specified amount of data. Only applies to the 'unnamed' data attribute. |
-filter_min_ads_size | Filter on the size of the ADS (alternate data stream), so it at least has the specified amount of data. |
The third area is the extraction options. All options include a results file. This generated file will contain much of the metadata one needs for forensic analysis. For more detailed analysis, one can add extra data to the results, including: (a) the first run of bytes for each file or (b) the cluster run information. To physically extract the contents of the file, one can specify an archive directory as well as whether to include slack data or not. If one does extract the file data, ntfswalk will compute the MD5 hash of the file and annotate this data to the results file as well. The options available and their syntax are:
Option | Description |
---|---|
-action_copy_files | Extracts the file data into the specified directory. The syntax is: -action_copy_files <directory to store files> [-raw] [-skip_sparse_clusters]. The -raw sub-option says to copy all clusters associated with a file in a bit-for-bit copy operation. This includes slack space as well as not uncompressing any data that may use native NTFS compression. The -skip_sparse_clusters sub-option says to ignore any clusters that are sparse during the copy operation. Applies only to -csv and -csvperpath output options. |
-action_include_header | Extracts the first 32 bytes of data and appends it to the CSV output. Applies only to -csv and -csvperpath output options. |
-action_include_clusterinfo | Shows additional information regarding data types and cluster runs and appends it to the CSV output. Applies only to -csv and -csvperpath output options. |
-action_include_zoneinfo | Extract and display any Zone.Identifier information in the output. Applies only to -csv and -csvperpath output options. |
-action_include_ownersid | Extract and display the owner SID. |
-action_find_strings_in_hdr | Examine the first 256 bytes of data in the file and scan for any strings. Applies only to -csv and -csvperpath output options. |
The fourth area allows one to select how one wishes to see the results. As mentioned above, even if one only wishes to extract data to a directory, there will be a results file that logs all the files passing the filter tests. The default output is plain text, which by itself, has reasonable formatting when viewed in notepad and word wrap is turned off. The other formats are geared for spreadsheet analysis or other post processing tools. Typically, any data containing numbers is defaulted as hexadecimal; however, there is an option to transform the output into base10 notation, if desired. The options available and their syntax are:
Option | Description |
---|---|
-csv | Outputs the data fields delimited by commas. Since filenames can have commas, to ensure the fields are uniquely separated, any commas in the filenames get converted to spaces. |
-csvl2t | Outputs the data fields in accordance with the log2timeline format. |
-bodyfile | Outputs the data fields in accordance with the 'body-file' version3 specified in the SleuthKit. The date/timestamp outputted to the body-file is in terms of UTC. If using the body-file in conjunction with the mactime.pl utility, one needs to set the environment variable TZ=UTC. |
-hashfile | Outputs a hashfile with either MD5, SHA1 or both hashes. The syntax is: -hashfile "md5" to output the MD5 hash, -hashfile "sha1" to output the SHA1 hash, -hashfile "sha256" to output the SHA256 hash, or -hashfile "md5 | sha1 | sha256" to output a combination of them at the same time. |
-csvperpath | Forces one line of CSV output per path/file entry. Since only one line is used, the MACB dates for standard information and filename is also spanned across one entry, which makes for a long record. This is useful for those users wishing to parse the output of ntfswalk into another application where most of the fields are in a separate field. There is an extra option to group the date and time into one field [-pair_datetime]. The default is to put the date and time into separate fields. |
Option | Description |
---|---|
-out | Put the summary information in the specified path/filename. The syntax is -out <results file>. |
-hide_dos_fntimes | Do not include any DOS 8.3 filename timestamps in the output |
-hostname | Option is used to populate the output records with a specified hostname. The syntax is -hostname <name to use>. |
-script | Use the specified file to express which options to use. The syntax is: -script <file>. |
-base10 | Ensure all size/address outputs are displayed in base-10 format versus hexadecimal format. Default is hexadecimal format. |
-use_orig_ext | Normal behavior is to append a .bin extension to any file copied. This option says not to append the .bin, but to use the original extension. |
-mftstart | Filter an inode range. The syntax is: -mftstart <inode> [-mftrange <number of inodes>]. |
-filerecord_offset | Output the absolute offset of the MFT filerecord metadata |
-quiet | This option suppresses status output as each file is processed. |
-no_whitespace | Used in conjunction with -csv option to remove any whitespace between the field value and the CSV separator. |
-csv_separator | Used in conjunction with the -csv option to change the CSV separator from the default comma to something else. Syntax is -csv_separator "|" to change the CSV separator to the pipe character. To use the tab as a separator, one can use the -csv_separator "tab" OR -csv_separator "\t" options. |
-dateformat | Output the date using the specified format. Default behavior is -dateformat "yyyy-mm-dd". Using this option allows one to adjust the format to mm/dd/yy, dd/mm/yy, etc. The restriction with this option is the forward slash (/) or dash (-) symbol needs to separate month, day and year and the month is in digit (1-12) form versus abbreviated name form. |
-timeformat | Output the time using the specified format. Default behavior is -timeformat "hh:mm:ss.xxx" One can adjust the format to microseconds, via "hh:mm:ss.xxxxxx" or nanoseconds, via "hh:mm:ss.xxxxxxxxx", or no fractional seconds, via "hh:mm:ss". The restrictions with this option is that a colon (:) symbol needs to separate hours, minutes and seconds, a period (.) symbol needs to separate the seconds and fractional seconds, and the repeating symbol 'x' is used to represent number of fractional seconds. (Note: the fractional seconds applies only to those time formats that have the appropriate precision available. The Windows internal filetime has, for example, 100 nsec unit precision available. The DOS time format and the UNIX 'time_t' format, however, have no fractional seconds). Some of the times represented by this tool may use a time format without fractional seconds and therefore will not show a greater precision beyond seconds when using this option. |
-pair_datetime | Output the date/time as 1 field versus 2 |
-cpu | This option is experimental and can only be used for processing MFT entries (as opposed to scanning unallocated clusters or other clusters outside the MFT). Its purpose is to take advantage of multiple CPUs to process a desired target. This option also requires one to specify an output file, via -out <result file>. Since this option creates multiple temporary files before merging the results into a final results file, one can also specify a folder where the temporary files can be stored. The option to specify a temporary folder is -tempdir <folder name>. The syntax is: -cpu <#children instances> -out <results file> [-tempdir <folder>] |
-utf8_bom | All output is in Unicode UTF-8 format. If desired, one can prefix an UTF-8 byte order mark to the CSV output using this option. |
1. Analyze a live NTFS partition and copy all PDF files to a specified directory, using the -partition <drive letter> and -action_copy_files <directory to store files> options.
ntfswalk -partition c: -filter_ext "pdf" -action_copy_files "f:\extracted_files\pdf" -out f:\results.txt
2. Analyze all files and extract the header from each file from a specified partition, using the -action_include_header option.
ntfswalk -partition c: -action_include_header -out "f:\mft_summary_output.txt"
3. Analyze a hidden NTFS partition by using the -drivenum option. When using this option, it will not be obvious what the volume offset is. To assist in resolving the volume offset, this option has the ability to find the volume offsets and display them to the user. Therefore, one can do an initial query using the command to query the volume offsets of drive number 0.
ntfswalk -drivenum 0
If your first drive has any NTFS volumes, their offsets will be displayed. Below is the output from my system, when running the above command:
volume offset [ 0x100000 ] : ntfs volume offset [ 0x6500000 ] : ntfs volume offset [ 0x1870500000 ] : ntfs
One can use the returned offset to analyze any volume that is desired. In this case, the first NTFS volume is the hidden. So if one wanted to extract all files from this hidden volume that contain the word "boot" in their filename, one would use the -filter_name <partial name to filter> and -action_copy_files <directory to store files> options.
ntfswalk -drivenum 0 -offset 0x100000 -filter_name "boot" -action_copy_file "f:\extracted_files\boot"
4. Analyze an image of a volume by using the option -image <partition image>, and redirect the output to a file
ntfswalk -image <partition file> > results.txt
5. Analyze an image of a disk via the option -image <disk image> -offset <volume offset> and redirect the output to a file
ntfswalk -image <disk image> -offset <volume offset> > results.txt
One can use the same trick discussed in a previous example to find the offset of the NTFS volumes within a disk image by just using the -image <disk image> command where it will output the NTFS volume offsets.
6. Analyze an Volume Shadow snapshot via the option -vss <snapshot index> and redirect the output to a file
ntfswalk -vss <1> > results.txt
Field | Definition |
---|---|
mft entry | Master File Table (MFT) entry used for this file/folder |
seqnum (after mft entry) | MFT sequence number |
parent mft | Parent MFT of the MFT entry |
seqnum (after parent mft) | Parent MFT sequence number |
type | file, dir or deleted |
ext | Extension used |
ref | Reference count in MFT entry |
date | Date of the MFT entry (refer to MACB to determine type of date) |
time-utc | Time of the MFT entry (refer to MACB to determine type of time) |
MACB | M=Modify, A=Access, C=MFT Changed, B=Birth/create. si = Timestamps associated with the Standard Information Attribute, fn = Timestamps associated with the Filename Attribute, fn8.3 = Timestamps associated with the Short Filename Attribute |
other info | Noteworthy items of interest |
path and filename | -ditto- |
various data types | List of any attributes that contain data |
Field | Definition |
---|---|
SI mdate | Target standard information modify date |
mtime-UTC (after SI mdate) | Target standard information modify time |
SI adate | Target standard information access date |
atime-UTC (after SI adate) | Target standard information access time |
SI cdate | Target standard information MFT change date |
ctime-UTC (after SI cdate) | Target standard information MFT change time |
SI bdate | Target standard information birth/create date |
btime-UTC (after SI bdate) | Target standard information birth/create time |
FN mdate | Target filename modify date |
mtime-UTC (after FN mdate) | Target filename modify time |
FN adate | Target filename access date |
atime-UTC (after FN adate) | Target filename access time |
FN cdate | Target filename MFT change date |
ctime-UTC (after FN cdate) | Target filename MFT change time |
FN bdate | Target filename birth/create date |
btime-UTC (after FN bdate) | Target filename birth/create time |
sym link | Target symbolic link |
object ID | Target object ID |
ADS metadata | Alternate data streams associated with this MFT entry |
Time warning | Any time anomalies that are detected |
8.3 mdate | Target short filename modify date |
mtime-UTC (after 8.3 mdate) | Target short filename modify time |
8.3 adate | Target short filename access date |
atime-UTC (after 8.3 adate) | Target short filename access time |
8.3 cdate | Target short filename MFT change date |
ctime-UTC (after 8.3 cdate) | Target short filename MFT change time |
8.3 bdate | Target short filename birth/create date |
btime-UTC (after 8.3 bdate) | Target short filename birth/create time |
When specifying directories to enumerate, one should use the forward slash when separating subdirectories versus the backslash that is used in Windows... eg.
When running in Linux [or OSX] to list the contents of the drivers directory
<location of app>/ntfswalk -image /<vol mount pt> -filter_dir /Windows/System32/drivers -out results.txt
When specifying directories that start with a dollar sign '$', it needs to be escaped w/ a backslash. For example $Recycle.Bins is a directory in the root directory of an NTFS partition. To list the contents of this directory, one needs to use the following notation:
<location of app>/ntfswalk -image /<vol mount pt> -filter_dir /\$Recycle.Bin/*3 -out results.txt
The first slash in '/\$Recycle.Bin' specifies the root directory. The next character is a backslash to specify the $ is included after the first slash, and the rest of the characters are normal.
This tool has authentication built into the binary. The primary authentication mechanism is the digital X509 code signing certificate embedded into the binary (Windows and macOS).
The other mechanism is the runtime authentication, which applies to all the versions of the tools (Windows, Linux and macOS). The runtime authentication ensures that the tool has a valid license. The license needs to be in the same directory of the tool for it to authenticate. Furthermore, any modification to the license, either to its name or contents, will invalidate the license.
The tools from TZWorks will output header information about the tool's version and whether it is running in limited, demo or full mode. This is directly related to what version of a license the tool authenticates with. The limited and demo keywords indicates some functionality of the tool is not available, and the full keyword indicates all the functionality is available. The lacking functionality in the limited or demo versions may mean one or all of the following: (a) certain options may not be available, (b) certain data may not be outputted in the parsed results, and (c) the license has a finite lifetime before expiring.