TZWorks LLC
System Analysis and Programming
www.tzworks.com
(Version 0.21)
TZWorks LLC software and related documentation ("Software") is governed by separate licenses issued from TZWorks LLC. The User Agreement, Disclaimer, and/or Software may change from time to time. By continuing to use the Software after those changes become effective, you agree to be bound by all such changes. Permission to use the Software is granted provided that (1) use of such Software is in accordance with the license issued to you and (2) the Software is not resold, transferred or distributed to any other person or entity. Refer to your specific EULA issued to for your specific the terms and conditions. There are 3 types of licenses available: (i) for educational purposes, (ii) for demonstration and testing purposes and (iii) business and/or commercial purposes. Contact TZWorks LLC (info@tzworks.com) for more information regarding licensing and/or to obtain a license. To redistribute the Software, prior approval in writing is required from TZWorks LLC. The terms in your specific EULA do not give the user any rights in intellectual property or technology, but only a limited right to use the Software in accordance with the license issued to you. TZWorks LLC retains all rights to ownership of this Software.
The Software is subject to U.S. export control laws, including the U.S. Export Administration Act and its associated regulations. The Export Control Classification Number (ECCN) for the Software is 5D002, subparagraph C.1. The user shall not, directly or indirectly, export, re-export or release the Software to, or make the Software accessible from, any jurisdiction or country to which export, re-export or release is prohibited by law, rule or regulation. The user shall comply with all applicable U.S. federal laws, regulations and rules, and complete all required undertakings (including obtaining any necessary export license or other governmental approval), prior to exporting, re-exporting, releasing, or otherwise making the Software available outside the U.S.
The user agrees that this Software made available by TZWorks LLC is experimental in nature and use of the Software is at user's sole risk. The Software could include technical inaccuracies or errors. Changes are periodically added to the information herein, and TZWorks LLC may make improvements and/or changes to Software and related documentation at any time. TZWorks LLC makes no representations about the accuracy or usability of the Software for any purpose.
ALL SOFTWARE ARE PROVIDED "AS IS" AND "WHERE IS" WITHOUT WARRANTY OF ANY KIND INCLUDING ALL IMPLIED WARRANTIES AND CONDITIONS OF MERCHANTABILITY, FITNESS FOR ANY PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. IN NO EVENT SHALL TZWORKS LLC BE LIABLE FOR ANY KIND OF DAMAGE RESULTING FROM ANY CAUSE OR REASON, ARISING OUT OF IT IN CONNECTION WITH THE USE OR PERFORMANCE OF INFORMATION AVAILABLE FROM THIS SOFTWARE, INCLUDING BUT NOT LIMITED TO ANY DAMAGES FROM ANY INACCURACIES, ERRORS, OR VIRUSES, FROM OR DURING THE USE OF THE SOFTWARE.
The Software are the original works of TZWorks LLC. However, to be in compliance with the Digital Millennium Copyright Act of 1998 ("DMCA") we agree to investigate and disable any material for infringement of copyright. Contact TZWorks LLC at email address: info@tzworks.com, regarding any DMCA concerns.
The Mozilla Firefox Browser has many artifacts available that the forensics examiner can use in identifying a user's Internet activity. This includes Firefox's various databases, local storage, JSON formatted text files and cache.
This tool only focuses on certain SQLite databases and specific tables within those databases that are used by the browser that have been deemed useful by the forensics community. Specifically, this tool currently targets the following five databases: (a) places.sqlite, (b) cookies.sqlite, (c) downloads.sqlite, (d) favicons.sqlite, and (e) formhistory.sqlite. Each of these databases are discussed briefly in the user's guide.
When looking across the various versions of the Firefox Browser over time, the schemas of the databases have changed. One can think of the database schema as the roadmap that defines the fields and the type of data in each field that comprise a record in the table (where one or more tables reside in a database). The change in schemas across different versions is something that needed to be taken into account when designing the msp tool. The design that was used was similar to that used on past SQLite parsers developed by TZWorks, in that, the tool can dynamically detect and adjust to varying schemas as they are encountered during the parsing operation.
In addition to the auto-schema detection, the msp tool allows the user to parse a target database in three ways. (1) The first way makes use of the standard SQL (Structure Query Language) to parse the records. The SQL syntax is internal to the tool, so the user is not required to have any knowledge about SQL or its syntax. For this option to be available, the SQLite library was statically linked into the tool, which eliminates the need for a SQLite dynamic library to be present to run the tool. (2) The second approach allows the user to instruct the tool to parse each record by traversing the internal SQLite structures as they are encountered. This option does not use any part of the standard SQLite library, but utilizes the TZWorks' internally designed libraries. The benefit of 'rolling your own' library is multi-faceted; not only does it allow the tool to extract records from a corrupted database, but one can annotate the exact offset of the data where it was found. This enables one to easily validate it later with a hex-editor. (3) The third, and final approach, uses a signature-based parse. While this option is more limited in merging records from one table to another, this turns out to be a unique way in parsing a blob of data whether it be from memory or from a fragment of a database. All three approaches are designed into the tool for the analyst to use. More discussion on these options are discussed later.
Mozilla Firefox SQLite database artifacts are located in the user's directory. This varies depending on the operating system used. Below is a table that breaks out the location by operating system.
| OS | Path |
|---|---|
| Win XP | C:\Documents and Settings\<username>\Local Settings\Application Data\Mozilla\Firefox\Profiles\<random text>.default |
| Post Win XP | C:\Users\<username>\AppData\Roaming\Mozilla\Firefox\Profiles\<random text>.default |
| OSX | /Users/{user acct}/Library/Application Support//Firefox/Profiles/<random text>.default |
| Linux | /home/{user acct}/.mozilla/firefox/<random text>.default |
The menu below shows many of the options available. Details of each option can be found here.
Usage
Mozilla SQLite Parser commands
msp -db <places.sqlite> [options]
dir <location Firefox db> /b /s | msp -pipe [options]
msp -enumdir <location Firefox dbs> -num_subdirs <#> [options]
Parsing options
[default opt] = use standard SQLite 'select' to pull records
-carve [-incl_slack] = carve opt1: by walking db structures
-parse_chunk = carve opt2: by scanning for signatures
Basic output options
-csv = output in CSV format
-csvl2t = log2timeline output
-bodyfile = sleuthkit output
Additional options
-username <name> = for -csvl2t output
-hostname <name> = for -csvl2t output
-csv_separator "|" = use a pipe char for csv separator
-dateformat mm/dd/yyyy = "yyyy-mm-dd" is the default
-timeformat hh:mm:ss = "hh:mm:ss" is the default
-no_whitespace = remove whitespace around csv delimiter
-quiet = no progress shown
Folder Traversing Options
-pipe = pipe files to parse
-enumdir <dir> -num_subdirs <#> = pull from files from folder
-split_sessions = split sessions into separate files
Testing options
-no_table_merge = output without merging tables
-verify [-add_comments] = generate stats on parsing [temporary]
To process SQLite database files, one can either target a folder or individual database files. The tool will automatically determine which database type/schema version to use and adjust the parsing engine accordingly. In fact, when parsing many subdirectories of artifacts where each subdirectory is a different account or machine, the tool will dynamically adjust for the version of the database being parsed at that time and keep the record content data sorted.
If processing a directory of database files (either by using the -pipe option or the -enumdir option), the tool will look for the Mozilla directory structure starting with the "Profile" folder to indicate when to start parsing.
If one wants to target a specific database, use the -db option. Without any specific parsing parameters, the default parser uses the Structured Query Language (SQL) in combination with the statically linked SQLite library to extract the records in the various tables in the database. Below is an example of doing this.
> msp64 -db c:\dump\places.sqlite -out results.csv
The default output is rendered in pipe delimited text and has 11 fields. To allow flexibility with rendering differing data types across the differing tables and databases in the output, some of the fields make use of a quasi-JSON like format; this allows records with different fields across various tables to be rendered in one CSV/delimited format.
The tool will try to show all the associated fields for each record. Those that are not normally looked at, are shown in the 'extra fields' column, where each field is annotated by a 'name of field/value of data' pair. Many of the items of interest such as timestamps and URL have their own dedicated columns.
If running the tool with either the -carve or the -parse_chunk options, the 'data record sources' field will be populated with the offset of the record. For example, running the same command above but specifying carve as the parse algorithm, yields the same data above, but with the data record sources field populated.
> msp64 -db c:\dump\places.sqlite -out results.csv -carve
This gives one the location information necessary to analyze the data in a hex editor to verify the results, if desired.
The benefit of the default option is its usefulness for verification and validation purposes. Given that the tool can produce the same output for any of the three available parsing options, one can use the default option as the base option to compare other parsing algorithm results. In this way, one can easily verify whether the carve option and/or signature-based option works, simply by comparing the results to that of the default SQL-Select option.
In most cases, the carve option (-carve) is a better choice over the default option, simply because is returns the same, if not more, results. If invoking the sub-option -incl_slack, the tool has the ability to detect unused space and switches to a signature-based scan for those areas.
Surprisingly, the signature-base option (-parse_chunk) competes very well with the other two options. Keep in mind, this option relies strictly on unique signatures being accurate for its success. While the other two options can dynamically adjust their parsing engine based on the schema identified in the database, the signature-based option cannot. Depending on the number of recoverable records in the database, it is possible for signature-based option to extract more records than the other options, however, the user is cautioned, that more records do not necessarily mean accurate data. For example, if one passes in a file that contains the contents of a disk volume, with the intent of extracting all the Firefox artifacts from that image, then the user may get multiple false positives on certain table records. The msp tool does a good job of statistically pulling out table entries that have many fields versus those tables that only have a few fields. Therefore, certain table entries will have less false positives than others.
The other issue to consider with the signature-base option is the merging operation from data in one table to another table (based on some relationship between the tables) may or may not make sense. For example, if a timestamp from one table is merged with data from another table, and the data is not in sync (from a chronological point of view), then the resulting merged record will mislead the investigator of an event's occurrence time-wise. The other pitfall with the signature-based scans, which was mentioned earlier, is that approach will truncate the data if a record overflows into multiple databases pages; the signature-based scan will only report on data found in the initial page.
To handle the data accuracy issue, refer to the section on "Merging of Data between Tables" in the User's Guide. In conclusion, despite the negatives for the signature-based parse, it is the only choice if analyzing partial chunks of database fragments, whether from memory or disk images.
In the output there is a column that identifies records by a type. These types are listed below along with where the data comes from.
| Record Type | Table(s) where the data resides | Database where the table(s) reside |
|---|---|---|
| Download | moz_annos or moz_downloads | places.sqlite or downloads.sqlite |
| Favicon | moz_icons or moz_favicons | favicons.sqlite or places.sqlite |
| Keyword | moz_keywords | places.sqlite |
| Bookmark | moz_bookmarks | places.sqlite |
| Url | moz_places, moz_historyvisits | places.sqlite |
| Origin | moz_origins | places.sqlite |
| Cookie | moz_cookies | cookies.sqlite |
| Formhistory | moz_formhistory | formhistory.sqlite |
In addition to the record types shown above, there are some cases were the type is supplemented with some extra words.
All tools need to do some form of verification to ensure their results are accurate. Part of that testing is to validate the tool's functionality across different artifact versions. If the tool developer can automate this testing, then it allows the developer to test the tool across many datasets quickly. This in turn quickly identifies inconsistencies and problems so a wide range of bugs can be diagnosed and fixed.
Normally, the developer tries to do as much of this testing before sending a tool out to clients. In the case of Mozilla, however, since it has a history of changing the schema's across versions such that they are not backwards compatible, we decided to temporarily add an option for clients run this type of verification on their own, if they so choose. To this end, the msp tool incorporates the -verify option to aid in this purpose.
The -verify option internally invokes all three parsing engines in sequence to parse the same database so it can compare the results of all three. Simplistically, if all the results match, then the confidence is very high the tool is working as designed. If the results do not match, it will be because a version of Mozilla is being analyzed where the tool may work on one of parsing engines, but not the others. The first parsing engine most likely to have problems will be the signature-based parsing, since it more sensitive to schema changes. In contrast, the default SQL-Select type parsing engine should be the most robust if there are schema changes, because it will key off of specific field names, which typically are more consistent across versioning. Either way, the purpose of the -verify option is an internal test to alert a user if any issues are found.
The nice thing about the way this option was implemented, is not only does it check the internal parsers against themselves, but it also outputs critical diagnostic data that can be used by TZWorks to help improve the tool. To ensure no personal information is outputted, the -verify option sanitizes the results so it contains no, or unintelligible, private/confidential information from the raw artifact. The output only contains metadata from the SQLite internal structures. This causes the data generated to be cryptic and only useful for machine type learning/statistics. An additional sub-option was added (-add_comments) to annotate some additional commentary to the results; this provides some extra information for the user if a test passed or failed and why.
If one wants to test an entire collection of SQLite artifacts from Mozilla, one can run this option with the -pipe< switch and point it at the folder/subfolders of Mozilla database artifacts. The tool will process all the database/records it finds and incorporate the results into a final report. Below is an example of doing this.
e:\ dir e:\testcase\sqlite\mozilla\tests /b /s | msp64 -pipe -verify -all_comments > results.txt
As mentioned earlier, the data produced is mostly cryptic since it contains statistical information about the database and records being parsed. This statistical information, if sent back to TZWorks, will help us improve our parsing engines for future releases.
| Option | Description |
|---|---|
| -db | Specifies which database file to act on. The format is: -db <SQLite db to parse>. |
| -csv | Outputs the data fields delimited by commas. Since filenames can have commas, to ensure the fields are uniquely separated, any commas in the filenames get converted to spaces. |
| -csvl2t | Outputs the data fields in accordance with the log2timeline format. |
| -bodyfile | Outputs the data fields in accordance with the 'body-file' version3 specified in the SleuthKit. The date/timestamp outputted to the body-file is in terms of UTC. So if using the body-file in conjunction with the mactime.pl utility, one needs to set the environment variable TZ=UTC. |
| -username | Option is used to populate the output records with a specified username. The syntax is -username <name to use> . |
| -hostname | Option is used to populate the output records with a specified hostname. The syntax is -hostname <name to use>. |
| -pipe | Used to pipe files into the tool via STDIN (standard input). Each file passed in is parsed in sequence. |
| -enumdir | Experimental. Used to process files within a folder and/or subfolders. Each file is parsed in sequence. The syntax is -enumdir <"folder"> -num_subdirs <#>. |
| -filter | Filters data passed in via stdin via the -pipe option. The syntax is -filter <"*.ext | *partialname* | ...">. The wildcard character '*' is restricted to either before the name or after the name. |
| -no_whitespace | Used in conjunction with -csv option to remove any whitespace between the field value and the CSV separator. |
| -csv_separator | Used in conjunction with the -csv option to change the CSV separator from the default comma to something else. Syntax is -csv_separator "|" to change the CSV separator to the pipe character. To use the tab as a separator, one can use the -csv_separator "tab" OR -csv_separator "\t" options. |
| -dateformat | Output the date using the specified format. Default behavior is -dateformat "yyyy-mm-dd". Using this option allows one to adjust the format to mm/dd/yy, dd/mm/yy, etc. The restriction with this option is the forward slash (/) or dash (-) symbol needs to separate month, day and year and the month is in digit (1-12) form versus abbreviated name form. |
| -timeformat | Output the time using the specified format. Default behavior is -timeformat "hh:mm:ss.xxxxxxxxx" One can adjust the format to microseconds, via "hh:mm:ss.xxxxxx" or milliseconds, via "hh:mm:ss.xxx", or no fractional seconds, via "hh:mm:ss". The restrictions with this option is that a colon (:) symbol needs to separate hours, minutes and seconds, a period (.) symbol needs to separate the seconds and fractional seconds, and the repeating symbol 'x' is used to represent number of fractional seconds. |
| -quiet | Show no progress during the parsing operation |
| -carve | Experimental option. Bypass the SQLite embedded library and parse using TZWorks internal algorithms. This is for the situation where the database to be parsed is corrupted and the SQLite library has trouble parsing it. |
| -incl_slack | Experimental option to look at unused space to see if any records are present. Not required with the -parse_chunk option. Use this in conjunction with -carve option to look for deleted records. |
| -parse_chunk | Experimental option. Given a portion (chunk) of the database this option will examine the data to see if any records exist and parse out the contents. This is a signature-based parse so it can parse out records from chunks of memory or slack space (in the form of a file). |
| -no_table_merge | This option is for pulling records from a image. It is also used for testing and debugging purposes. If you want to see all the tables that were parsed without merging any relationships, use this option. |
| -verify | This option is for testing and debugging purposes only. This option runs all 3 parsing engines in the tool (SQL Select parse, Carve parse and Signature-based parse) and reports whether the parsers work at least up to the level of the SQL Select parse. Metadata is generated that can be used to help develop more robust parsing algorithms. |
| -split_sessions | Split the Mozilla sessions into separate files. |
This tool has authentication built into the binary. The primary authentication mechanism is the digital X509 code signing certificate embedded into the binary (Windows and macOS).
The other mechanism is the runtime authentication, which applies to all the versions of the tools (Windows, Linux and macOS). The runtime authentication ensures that the tool has a valid license. The license needs to be in the same directory of the tool for it to authenticate. Furthermore, any modification to the license, either to its name or contents, will invalidate the license.
The tools from TZWorks will output header information about the tool's version and whether it is running in limited, demo or full mode. This is directly related to what version of a license the tool authenticates with. The limited and demo keywords indicates some functionality of the tool is not available, and the full keyword indicates all the functionality is available. The lacking functionality in the limited or demo versions may mean one or all of the following: (a) certain options may not be available, (b) certain data may not be outputted in the parsed results, and (c) the license has a finite lifetime before expiring.