TZWorks LLC
System Analysis and Programming
www.tzworks.com
(Version 0.41)
TZWorks LLC software and related documentation ("Software") is governed by separate licenses issued from TZWorks LLC. The User Agreement, Disclaimer, and/or Software may change from time to time. By continuing to use the Software after those changes become effective, you agree to be bound by all such changes. Permission to use the Software is granted provided that (1) use of such Software is in accordance with the license issued to you and (2) the Software is not resold, transferred or distributed to any other person or entity. Refer to your specific EULA issued to for your specific the terms and conditions. There are 3 types of licenses available: (i) for educational purposes, (ii) for demonstration and testing purposes and (iii) business and/or commercial purposes. Contact TZWorks LLC (info@tzworks.com) for more information regarding licensing and/or to obtain a license. To redistribute the Software, prior approval in writing is required from TZWorks LLC. The terms in your specific EULA do not give the user any rights in intellectual property or technology, but only a limited right to use the Software in accordance with the license issued to you. TZWorks LLC retains all rights to ownership of this Software.
The Software is subject to U.S. export control laws, including the U.S. Export Administration Act and its associated regulations. The Export Control Classification Number (ECCN) for the Software is 5D002, subparagraph C.1. The user shall not, directly or indirectly, export, re-export or release the Software to, or make the Software accessible from, any jurisdiction or country to which export, re-export or release is prohibited by law, rule or regulation. The user shall comply with all applicable U.S. federal laws, regulations and rules, and complete all required undertakings (including obtaining any necessary export license or other governmental approval), prior to exporting, re-exporting, releasing, or otherwise making the Software available outside the U.S.
The user agrees that this Software made available by TZWorks LLC is experimental in nature and use of the Software is at user's sole risk. The Software could include technical inaccuracies or errors. Changes are periodically added to the information herein, and TZWorks LLC may make improvements and/or changes to Software and related documentation at any time. TZWorks LLC makes no representations about the accuracy or usability of the Software for any purpose.
ALL SOFTWARE ARE PROVIDED "AS IS" AND "WHERE IS" WITHOUT WARRANTY OF ANY KIND INCLUDING ALL IMPLIED WARRANTIES AND CONDITIONS OF MERCHANTABILITY, FITNESS FOR ANY PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT SHALL TZWORKS LLC BE LIABLE FOR ANY KIND OF DAMAGE RESULTING FROM ANY CAUSE OR REASON, ARISING OUT OF IT IN CONNECTION WITH THE USE OR PERFORMANCE OF INFORMATION AVAILABLE FROM THIS SOFTWARE, INCLUDING BUT NOT LIMITED TO ANY DAMAGES FROM ANY INACCURACIES, ERRORS, OR VIRUSES, FROM OR DURING THE USE OF THE SOFTWARE.
The Software are the original works of TZWorks LLC. However, to be in compliance with the Digital Millennium Copyright Act of 1998 ("DMCA") we agree to investigate and disable any material for infringement of copyright. Contact TZWorks LLC at email address: info@tzworks.com, regarding any DMCA concerns.
csvdx is a command line, support tool that converts delimited data (such as CSV data) into other formats. Currently csvdx supports conversion to: (a) HTML table data, (b) JSON format, and (c) a SQLite database. These formats are useful if desiring to: (a) displaying the data in other viewers (b) importing the original delimited data to other databases, or (c) just trying to merge similar artifacts together.
CSV stands for Comma Separated Values. However, in this readme document, the term CSV will also be used to refer to other delimiters as well, such as tab delimited, pipe character delimited, etc. Currently, csvdx can handle: comma, tab and pipe delimiters.
The default behavior for tools built by TZWorks is to generate a banner at the top of the file before proceeding with any delimited data. This banner contains some additional information that can be useful, if retained, when converting the delimited data to another format. Information such as: (a) the command line options used to parse the original artifact, (b) the timestamp when the parsing was done, (c) the license /organization that conducted the parsing, and (d) which version of the tool was used. csvdx reads this banner data and subsequently embeds it to the converted format so it is preserved.
Other, non-standard CSV data may be present in TZWorks® tools when processing differing artifact types and storing the results in one CSV file. In these cases, the differing artifacts may have column positions which correspond to the different fields of artifact being processed. Good examples of this are when processing registry data via cafae or processing event logs with evtwalk. In both cases, the resulting CSV file will have multiple CSV sections. To handle this, csvdx looks at the banner data and adjusts the parsing logic based on the tool (which is recorded in the banner) that was used to generate the CSV file. When using the SQLite option to store the artifact data from the CSV file, the banner data will allow csvdx to break the data out by artifact within the SQLite database.
One can display the menu options by typing in the executable's name with no parameters. A screen shot of the menu is shown below:
Usage csvdx -src <file> -dst <file> -csv_to_html = convert CSV to HTML csvdx -src <file> -dst <file> -csv_to_json [-flat] = convert CSV to JSON csvdx -src <file> -dst <file> -csv_to_sqlite = convert CSV to SQLite csvdx -src <file> -dst <dir> -split_into_files = split artifacts by file csvdx -src <file> [-dst <file>] [CSV Cleanup/Modifiy Options] CSV Cleanup/Modify Options: -no_whitespace = remove whitespace -pair_fields "1:2, 3:4, .." = group fields together -align_cols = format CSV w/ spacing -change_csv_separator "|" = modify the csv separator JSON Option: -flat = 1 line per entry and no header info Converting back to CSV from SQLite DB created by csvdx: csvdx -src <srcdb> -dst <dir> -sqlite_to_csv = reverse -csv_to_sqlite SQLite utils (note: source file is the SQLite db): csvdx -src <srcdb> -dst <dir> -artifact_tables = outputs to specified dir csvdx -src <srcdb> -metadata_records = outputs to the screen csvdx -src <srcdb> -ref_records = outputs to the screen Generic Options: -pipe = pipe files into csvdx for processing -quiet = don't display status during run -filter <*partial*|*.ext> = filters stdin data from -pipe option -no_header_info = not parsing a CSV file with a TZWorks banner
If one is given a CSV file that has formatting properties that need to be changed, csvdx offers 4 possible options to modify the CSV file via the following switches:
Finally, the last option, which is not so much to modify a CSV file, but to split the contents of the records into separate files, while grouping like artifact types. This is option is -split_into_files. The purpose of this option is to take a CSV file and examine the records for like artifact types and group them into independent and separate CSV files. This is useful for registry hive parsing where cafae to will parse various artifacts from one hive into one CSV file. This option will take that single CSV file and separate all like artifact types into separate CSV files.
When converting from CSV data to a HTML table format, use the -csv_to_html option and either redirect the output to a file or use the -dst <result file> option.
The differing artifacts in the original CSV file are ported over to their respective HTML table, and the banner information is converted as well.
When converting from CSV data to a JSON format, use the -csv_to_json option and either redirect the output to a file or use the -dst <result file> option.
The differing artifacts in the original CSV file are ported over to their respective JSON array, and the banner information is converted as well.
When converting from CSV data to a SQLite format, use the -csv_to_sqlite option. For each artifact that is found in the CSV data, a unique table is dynamically generated for that specific artifact. There are a few functions that should be of note: (a) csvdx has the ability to detect similar artifacts and insert into an existing artifact table if already generated, (b) on subsequent runs of csvdx, if using an existing SQLite database that was generated originally from csvdx, the artifacts will be merged into the appropriate like-artifact tables, and (c) on subsequent runs of csvdx, one can use completely different CSV tool outputs (eg. one from cafae, one from evtwalk, etc) and the artifact tables will be preserved.
While this is nice for those users that are familiar with reviewing a database schema and based on that schema, issuing some SQL select statement to extract those components that are of interest, it may not be useful to those that don't have that background. For those users, we created an option to pull all the merged artifact table data into separate CSV files. This is the -artifact_tables option.
The -artifact_tables option will extract all the artifact data stored in the SQLite database back into a CSV type output. What this option does internally is: (a) reads the SQLite database specified, (b) extracts all the artifact data while merging the banner specific data pulled during the initial CSV parse with the artifact data, and (c) dumps the final output into a separate CSV files at the directory specified. Therefore, if you had 10 artifact tables to start with, you will end up with 10 unique CSV files with the data from those artifact tables. Also the resulting CSV files will have merged similar artifact data into their respective files.
For the SQLite option to work, csvdx makes the of the SQLite library. If one is unfamiliar with SQLite, the official SQLite website ishttp://www.sqlite.org/. It has documentation and details on everything one would ever want to know. Starting with version 0.23, the SQLite library is statically linked into csvdx binary. What this means is the tool is standalone and does not require any external SQLite shared libraries.
Option | Description |
---|---|
-src | Specifies the source CSV file to operate on. This file will not be modified. Format is -src <source file> |
-dst | Specifies the destination file to send the results to. Format is -dst <resuts file>. Without specifying this option, the results will be outputted to the screen (or can be redirected). In some cases, this option will specify the destination folder (not file) to send the output to. |
-csv_to_html | Instructs conversion to be from a CSV file to HTML table format. |
-csv_to_json | Instructs conversion to be from a CSV file to JSON format. There is a sub-option -esc_path to allow for backslashes to be preserved with the escape character. Default behavior converts backslashes to forward slashes. |
-csv_to_sqlite | Instructs conversion to be from a CSV file to SQLite format. With this option, one needs to also specify the -dst <resulting database> parameter to tell what new database base to create. If a previous database was created using csvdx, then that can be specified as well and the data will be added to the database. |
-split_into_files | This is specific to TZWorks tools and their output. Specifically, some artifacts, such as the eventlog and registry hives, will have differing header fields that are specific to differing event IDs or type of registry artifact. This option will take the single CSV file containing these differing artifacts and separate each unique one into a separate CSV file. With this option, one needs to also specify the -dst <folder> parameter to tell which folder to put the new CSV files. |
-pipe | This option allows one to pipe in multiple CSV files from standard input while storing the CSV artifact data into a SQLite database. |
-enumdir | Experimental. Used to process files within a folder and/or subfolders. Each file is parsed in sequence. The syntax is -enumdir <"folder"> -num_subdirs <#>. |
-filter | Filters data passed in via stdin via the -pipe option. The syntax is -filter <"*.ext | *partialname* | ...">. The wildcard character '*' is restricted to either before the name or after the name. |
-quiet | This option is tells csvdx not to display progress status during a run. |
-flat | Suboption to -csv_to_json to force 1 line per entry and no header information |
-no_header_info | Tells tool that you are not parsing a CSV file with a TZWorks banner, so it will not look for it. |
These options are only used to modify the properties of the CSV format. At a minimum the -src <source file> must be specified. If no -dst <resulting file> is specified, the resulting output will be sent to standard output (eg. the screen), which can be redirected into another tool or file. If using the -pipe option, then the -dst <resulting directory> must be specified.
Option | Description |
---|---|
-no_whitespace | This option will remove all white space between the field value and the CSV separator. |
-pair_fields | This option will pair two separate fields into one. Format is: -pair_fields "1:2, 3:4". The 1:2 notation means combine column 2 into column 1. The 3:4 notation means combine column 4 into column 3. This is useful when desired to combine date field and time field into one field containing both date and time. |
-align_cols | This option will space out the delimited fields and try to align the columns. This is useful to view the CSV output in a more readable manner when using text viewer such as notepad. |
-change_csv_separator | This option is to modify the existing CSV delimiter to some other delimiter. Format is: -change_csv_separator "|". |
-utf8_bom | All output is in Unicode UTF-8 format. If desired, one can prefix an UTF-8 byte order mark to the CSV output using this option. |
Option | Description |
---|---|
-ref_records | This option is used to read the resulting ref table and display the records. The format is: -src <SQLite db> -ref_records |
-metadata_records | This option is used to read the resulting metadata table and display the records. The format is: -src <SQLite db> -metadata_records |
-artifact_tables | This option is used to read the resulting SQLite database and output each artifact table into unique CSV files specified at the specified directory. The format is: -src <SQLite db> -dst <directory to store files> -artifact_tables. |
When parsing CSV (or any delimited) data using this tool, the source data needs to be in ASCII or UTF-8 (Unicode) format. Specifically, if the data is another Unicode format (eg. UTF-16, which is common in Windows), then the tool will not parse the source data. All the TZWorks' tools produce their output in UTF-8 format, so if the source data is from one of our tools, then it should work. If the source data is from another tool and if the format is something other than UTF-8, then convert it to UTF-8 first before using this tool.
If using our tools within Powershell and redirecting the output to a file, Powershell's default behavior is to output redirected data in UTF-16. Therefore, one must take steps to output the redirected stream to UTF-8 without a BOM (byte order mark) in the front end of the UTF-8 data.
This tool has authentication built into the binary. The primary authentication mechanism is the digital X509 code signing certificate embedded into the binary (Windows and macOS).
The other mechanism is the runtime authentication, which applies to all the versions of the tools (Windows, Linux and macOS). The runtime authentication ensures that the tool has a valid license. The license needs to be in the same directory of the tool for it to authenticate. Furthermore, any modification to the license, either to its name or contents, will invalidate the license.
The tools from TZWorks will output header information about the tool's version and whether it is running in limited, demo or full mode. This is directly related to what version of a license the tool authenticates with. The limited and demo keywords indicates some functionality of the tool is not available, and the full keyword indicates all the functionality is available. The lacking functionality in the limited or demo versions may mean one or all of the following: (a) certain options may not be available, (b) certain data may not be outputted in the parsed results, and (c) the license has a finite lifetime before expiring.