NSZ - NSP/XCI compressor/decompressor by Nico Bosshard

NSZ is an open source MIT licensed python application with optional GUI to lossless compress/decompress NSP files in order to save a lot of storage while all NCA files keep their exact same hash. The NSZ file format for compressed Nintendo Switch games is already widely adopted and supported by popular homebrews like Awoo-Installer, GoldBricks, OG Tinfoil, Tinfoil, Lithium and many more.

Currently it improves size, speed and bandwidth used for storing and transferring NSZ files while still leading to the same size when installed but integration into Yuzu and CFW to directly play compressed games is planned. NSZ's developments started by blawar based on the idea of nsZip and its NSPZ/XCIZ file format which unfortunately was too complex to be implemented by homebrew developers. I soon took over his project and am closely working together with all homebrew developers interested in the NSZ format.

How to install:
Put dumped prod.keys to %userprofile%/.switch, install python, execute "pip install nsz" and use "nsz" like every other cmd command.

You can get the latest release of NSZ under https://github.com/nicoboss/nsz/releases
To read the readme and take a look at its source code visit https://github.com/nicoboss/nsz

Results how well NSZ performs: https://gbatemp.net/threads/nsz-title-compression-results.549831/
An easy to use tool for NSZ compression: https://github.com/julesontheroad/NSC_BUILDER
Anoter tool for NSZ compression: https://gbatemp.net/threads/ryjin-a-nsz-converter-mod.550174/

Here some screenshots of the GUI:

Installation:

There are several ways the install the script. You can find details on installation for all of them below.

You need to have a hactool compatible keys file in a suitable directory to use the script.
The keys file must be located as prod.keys file in %USERPROFILE%/.switch/(Windows)/$HOME/.switch/(UNIX) or keys.txt in the working directory.

It can be dumped with Lockpick_RCM.

Windows Builds

You can also use the Windows binaries. They do not require any external libraries to be installed and can be run without installing anything. You can find the binaries in the release page.

Methods listed below requires you to have Python 3.6+ installed.

PIP Package

Simplest way to install would be using the following command in a terminal or a command prompt. This works on every operating system with Python 3.6 and later.
pip3 install --upgrade nsz

If you are interested in installing the GUI for the script, you can do so by running one of the following commands. On Linux it’s highly recommended to follow Running from source on Linux instead.
Python 3.6 and Python 3.7:
pip3 install --upgrade nsz[gui]

Python 3.8 and later: Download requirements-gui.txt and execute:
pip3 install -r requirements-gui.txt

Running from source on Linux

On Linux just clone and execute pip3 install -r requirements.txt for the no-GUI version and ./install_linux.sh if you want GUI.

Running from source on Windows

The script can also be run by cloning the repo locally. You need to install the dependencies by running the following command.
pip3 install -r requirements.txt

GUI is optional and requires extra modules to run with GUI. To install the modules required to run GUI, run the following command on Python 3.6 and Python 3.7 on Windows:
pip3 install -r requirements-gui.txt

Usage

nsz --help
usage: nsz.py [-h] [-C] [-D] [-l LEVEL] [-B] [-S] [-s BS] [-V] [-p] [-P]
              [-t THREADS] [-m MULTI] [-o [OUTPUT]] [-w] [-r] [--rm-source]
              [-i] [--depth DEPTH] [-x] [--extractregex EXTRACTREGEX]
              [--titlekeys] [--undupe] [--undupe-dryrun] [--undupe-rename]
              [--undupe-hardlink] [--undupe-prioritylist UNDUPE_PRIORITYLIST]
              [--undupe-whitelist UNDUPE_WHITELIST]
              [--undupe-blacklist UNDUPE_BLACKLIST] [--undupe-old-versions]
              [-c CREATE]
              [file ...]

positional arguments:
  file

optional arguments:
  -h, --help            show this help message and exit
  -C                    Compress NSP/XCI
  -D                    Decompress NSZ/XCZ/NCZ
  -l LEVEL, --level LEVEL
                        Compression Level: Trade-off between compression speed
                        and compression ratio. Default: 18, Max: 22
  -L, --long            Enables zStandard long distance mode for even better
                        compression
  -B, --block           Use block compression option. This mode allows highly
                        multi-threaded compression/decompression with random
                        read access allowing compressed games to be played
                        without decompression in the future however this comes
                        with a slightly lower compression ratio cost. This is
                        the default option for XCZ.
  -S, --solid           Use solid compression option. Slightly higher
                        compression ratio but won't allow for random read
                        access. File compressed this way will never be
                        mountable (have to be installed or decompressed first
                        to run). This is the default option for NSZ.
  -s BS, --bs BS        Block Size for random read access 2^x while x between
                        14 and 32. Default: 20 => 1 MB
  -V, --verify          Verifies files after compression raising an unhandled
                        exception on hash mismatch and verify existing NSP and
                        NSZ files when given as parameter
  -p, --parseCnmt       Extract TitleId/Version from Cnmt if this information
                        cannot be obtained from the filename. Required for
                        skipping/overwriting existing files and --rm-old-
                        version to work properly if some not every file is
                        named properly. Supported filenames:
                        *TitleID*[vVersion]*
  -P, --alwaysParseCnmt
                        Always extract TitleId/Version from Cnmt and never
                        trust filenames
  -t THREADS, --threads THREADS
                        Number of threads to compress with. Numbers < 1
                        corresponds to the number of logical CPU cores for
                        block compression and 3 for solid compression
  -m MULTI, --multi MULTI
                        Executes multiple compression tasks in parallel. Take
                        a look at available RAM especially if compression
                        level is over 18.
  -o [OUTPUT], --output [OUTPUT]
                        Directory to save the output NSZ files
  -w, --overwrite       Continues even if there already is a file with the
                        same name or title id inside the output directory
  -r, --rm-old-version  Removes older versions if found
  --rm-source           Deletes source file/s after compressing/decompressing.
                        It's recommended to only use this in combination with
                        --verify
  -i, --info            Show info about title or file
  --depth DEPTH         Max depth for file info and extraction
  -x, --extract         Extract a NSP/XCI/NSZ/XCZ/NSPZ
  --extractregex EXTRACTREGEX
                        Regex specifying which files inside the container
                        should be extracted. Example: "^.*\.(cert|tik)$"
  --titlekeys           Extracts titlekeys from your NSP/NSZ files and adds
                        missing keys to ./titlekeys.txt and JSON files inside
                        ./titledb/ (obtainable from
                        https://github.com/blawar/titledb). Titlekeys can be
                        used to unlock updates using NUT OG (OG fork
                        obtainable from https://github.com/plato79/nut). There
                        is currently no publicly known way of optioning NSX
                        files. To MitM: Apply disable_ca_verification &
                        disable_browser_ca_verification patches, use your
                        device's nx_tls_client_cert.pfx (Password: switch,
                        Install to OS and import for Fiddler or import into
                        Charles/OWASP ZAP). Use it for aauth-
                        lp1.ndas.srv.nintendo.net:443, dauth-
                        lp1.ndas.srv.nintendo.net:443 and
                        app-b01-lp1.npns.srv.nintendo.net:443. Try with your
                        WiiU first as there you won't get banned if you mess
                        up.
  --undupe              Deleted all duplicates (games with same ID and
                        Version). The Files folder will get parsed in order so
                        the later in the argument list the more likely the
                        file is to be deleted
  --undupe-dryrun       Shows what files would get deleted using --undupe
  --undupe-rename       Renames files to minimal standard:
                        [TitleId][vVersion].nsz
  --undupe-hardlink     Hardlinks files to minimal standard:
                        [TitleId][vVersion].nsz
  --undupe-prioritylist UNDUPE_PRIORITYLIST
                        Regex specifying which dublicates delegtion should be
                        prioritized before following the normal deletion
                        order. Example: "^.*\.(nsp|xci)$"
  --undupe-whitelist UNDUPE_WHITELIST
                        Regex specifying which dublicates should under no
                        circumstances be deleted. Example: "^.*\.(nsz|xcz)$"
  --undupe-blacklist UNDUPE_BLACKLIST
                        Regex specifying which files should always be deleted
                        - even if they are not even a dublicate! Be careful!
                        Example: "^.*\.(nsp|xci)$"
  --undupe-old-versions
                        Removes every old version as long there is a newer one
                        of the same titleID.
  -c CREATE, --create CREATE
                        create / pack a NSP

Few Usage Examples

To compress all files in a folder: nsz -C /path/to/folder/with/roms/
To compress all files in a folder and verifying integrity of compressed files: nsz --verify -C /path/to/folder/with/roms/
To compress all files in a folder with 8 threads and outputting resulting files to a new directory: nsz --threads 8 --output /path/to/out/dir/ -C /path/to/folder/with/roms/
To compress all files in a folder with level 22 compression level: nsz --level 22 -C /path/to/folder/with/roms/
To decompress all files in a folder: nsz -D /path/to/folder/with/roms/

To view all the possible flags and a description on what each flag, check the Usage section.

File Format Details

NSZ

NSZ files are functionally identical to NSP files. Their sole purpose to alert the user that it contains compressed NCZ files. NCZ files can be mixed with NCA files in the same container.

As an alternative to this tool NSC_Builder also supports compressing NSP to NSZ, and decompressing NSZ to NSP. NSC_Builder can be downloaded at https://github.com/julesontheroad/NSC_BUILDER

XCZ

XCZ files are functionally identical to XCI files. Their sole purpose to alert the user that it contains compressed NCZ files. NCZ files can be mixed with NCA files in the same container.

NCZ

These are compressed NCA files. The NCA’s are decrypted, and then compressed using zStandard.

The first 0x4000 bytes of an NCZ file is exactly the same as the original NCA (and still encrypted). This applies even if the first section doesn’t start at 0x4000.

At 0x4000, there is the variable sized NCZ Header. It contains a list of sections which tell the decompressor how to re-encrypt the NCA data after decompression. It can also contain an optional block compression header allowing random read access.

All of the information in the header can be derived from the original NCA + Ticket, however it is provided pre-parsed to make decompression as easy as possible for third parties.

Directly after the NCZ header, the zStandard stream begins and ends at EOF. The stream is decompressed to offset 0x4000. If block compression is used the stream is splitted into independent blocks and can be decompressed as shown in https://github.com/nicoboss/nsz/blob/master/nsz/BlockDecompressorReader.py

class Section:
	def __init__(self, f):
		self.magic = f.read(8) # b'NCZSECTN'
		self.offset = f.readInt64()
		self.size = f.readInt64()
		self.cryptoType = f.readInt64()
		f.readInt64() # padding
		self.cryptoKey = f.read(16)
		self.cryptoCounter = f.read(16)

class Block:
	def __init__(self, f):
		self.magic = f.read(8) # b'NCZBLOCK'
		self.version = f.readInt8()
		self.type = f.readInt8()
		self.unused = f.readInt8()
		self.blockSizeExponent = f.readInt8()
		self.numberOfBlocks = f.readInt32()
		self.decompressedSize = f.readInt64()
		self.compressedBlockSizeList = []
		for i in range(self.numberOfBlocks):
			self.compressedBlockSizeList.append(f.readInt32())

nspf.seek(0x4000)
sectionCount = nspf.readInt64()
for i in range(sectionCount):
	sections.append(Section(nspf))

if blockCompression:
	BlockHeader = Block(nspf)

References

NSZ pip package: https://pypi.org/project/nsz/
Forum thread: https://gbatemp.net/threads/nsz-homebrew-compatible-nsp-xci-compressor-decompressor.550556/

Credits

SciresM for his hardware crypto functions; the blazing install speeds (50 MB/sec +) achieved here would not be possible without this.

Thanks to our contributors: nicoboss, blawar, plato79, eXhumer, Taorni, gabest11, thatch, pR0Ps, maki-chan

Changelog:

NSZ 4.0:

Implemented Drag & Drop support as requested in #51
Implemented CRC32 key validation and added support for future masterkeys
- Fixed the issue of master_key_0a not being recognized
- Added CRC32 for master_key_0a
Windows 7 support for Windows builds
Improved GUI font size scaling
Set the NSZ GUI window to be TopMost (always on top) on Windows so Drag & Drop gets much more convenient
- Added a setting to specify if the Kivy window should be always on top or not
XCI/XCZ finally extracts to folders containing the NCA/NCZ files instead of HFS0 partition dumps
Fixed a major XCI compatibility bug by implementing compression/decompression support for NCA files with the first section having a smaller or larger offset then 0x4000. This fixes #49
Added NSPZ (nsZip legacy file format) extraction support
Make GUI an optional install
Fixed #59 ncz decompression is not working
Cleaned up imports for nsz package
Stop bar manager to avoid broken shells
Added pywin32 as GUI dependency for Windows. This fixes #56
Fixed BlockDecompressorReader.seek with whence = 2 (seek relative to the file's end). This fixes #64
Starting nsz.py from within a different working directory finally works
- Fixes the current Azure Pipeline issue
Added solid decompression, block decompression, solid compression and block compression tests to azure-pipelines.yml
Made NSZ new returning with error code 1 if there are any exceptions
Fixed deadlock in BlockCompressor.py
Highly improved block decompression speed by caching the current block
Added Visual Studio 2019 Python Project
Added titleId and version to the file list and highly improved its design
The SelectableLabel items inside the game list now properly scales its height according to the available width and text length of the file path. This fixes #50
Added a multi-language supporting open source font for #61
Implemented input folder as output folder by default for #61
Waiting for Enter before exit when started over GUI so errors and console output can be seen before it closes for #74
Fixed install failing on Kubuntu 20.04 and a lot of other modern Linux distributions by improving install_linux.sh for #75
Removed code that manipulated the XCI header size for absolutely no reason which fixes #77
Improved decompression speed by 400% by heavily reducing the amount of performance intensive status bar refresh calls
The decision if the last block should be decompressed or copied now matches the file format specifications by comparing the decompressed block size of that specific block with its compressed size. This issue was caused by missing the (unlikely) edge case that the last block can be larger when compressed without exceeding the general block size. This fixes #79
Improved BlockCompression performance and overall CPU usage by reducing the amount of performance intensive status bar refresh calls.
Ensure that the line right to the curser is clean when the application terminates
Implemented undupe, undupe-dryrun, undupe-prioritylist, undupe-whitelist and undupe-old-versions to remove duplicate games
Highly improved the missing prod.keys/keys.txt error message by not showing the stack trace and waiting for a user input before exiting
Highly improved README.md
General system stability improvements to enhance the user's experience.

NSZ 3.1:

Fixed broken decompression in v3.0.0 due to an optional argument not marked as such and an outdated file extension comparison
Allow the selection of individual files inside the OpenFileDialog
Made it visible which drive is currently selected inside the OpenFileDialog and SaveFileDialog
Replaced the background_down image with the background_normal one as it looks ugly inside the FileDialogs
Removed SaveFileDialog as its unused and a pain to maintain
Highly improved the OpenFileDialog by allowing switching between the icon and list layout
Added filter to the OpenFileDialog so it only displays nsp, nsz, xci, xcz and ncz files
No longer showing the empty placeholder for the device selection on non-Windows platforms an made per OpenFileDialog argument specifiable file filters possible
Finally enabled the selection of folders and selecting multiple items at the same time
Automatically creating the empty gui folder required for settings to be saved in nsz portable which is exactly the fix manually applied to nsz_v3.0.0_hotfix1_win64_portable.zip
Fixed the warnings that appeared in the console on every GUI launch
Made the file browser view mode buttons in the same design as the Windows device selection buttons
Implemented deletion of selected GameList items using the delete or backspace key
Highly improved the GameList item selection and deletion
Vixed verification only mode not executing when called from GUI
Set the default amount of threads for solid compression to 3 while keeping the number of logical cores the default for block compression because the majority is now using task parallelization for solid compression
Fixed: AttributeError: 'RootWidget' object has no attribute 'verify'
Dirty fixed Kivy throwing "Error in sys.excepthook" during shutdown if the amount of RecycleView data was ever decreasing
- I tried multiple hours fixing this the proper way without success. I also tried removing all children what makes it to throw the same exception without even decreasing the amount of data and tried filling it with dummy data on shutdown in self.rootWidget.gameList.recycleView.shutdown() which didn’t worked because it’s already too late to spawn everything needed to avoid this exception.
Improved the wording of the error summary
Updated testing and deployment scripts
Fixed orphan processes remaining after the main process terminated after block compressing a very short task resulting in the orphan processes spamming "AttributeError: 'ForkAwareLocal' object has no attribute 'connection'" exceptions
Fixed NSZ GUI icon not showing
General system stability improvements to enhance the user's experience.

NSZ 3.0:

GUI:
- Contains all functions available using command line arguments
XCZ support
- Block compressed (default)
- Solid compressed
- XCZ to XCI decompression
Implemented task parallel solid compression. See --multi
Implemented titlekey extraction with titlekeys.txt and titledb support. See --titlekeys
Added regex support to the extract option to allow the user to specify exactly which files should be extracted from the container. See --extractregex
NCZ decompression directly in nsz #38
Updated IndependentNczDecompressor to the latest version inside nsz
Fully replace tqdm with enlighten
- Implemented multithreaded multiple process bars communication system for solid compression and verification while avoiding stdout race conditions
Endless decompression/verification on a few block compressed games #25
Fixed a major bug causing the space between block compressed NCZ files to be filled up with 0x00 to fit their uncompressed size
Major Scripts Cleanup
Fixed NSZ Decompressor TQDM Progress Bar
Fixed --rm-old-version
fixed path bug with decompression
Removed pycryptodome v3.9.0 restriction as its latest v3.9.3 works fine
Improved installation guide
Fixed an exception that could occur when trying to receive keys under special circumstances during debugging
Fixed a major bug inside the enhanced file existing check leading to files with the same name as a file already existing inside the output directory being overwritten without specifying this behavior using the –overwrite command line argument. This bug was cause by comparing the output file path instead of the output filename with existing filenames.
Made nsz pip package building Kivy compatible
Switched from Nuitka to PyInstaller due to Kivy compatibility
Changed how --extract and --verify arguments are handled internally
Set up CI with Azure Pipelines using self-hosted server
Fully switched to pathlib. This fixes #41 and a lot of other file path related issues
Fixed Enhanced File Existing Check. Adapting to pathlib and finally fixing --overwrite and --rm-old-version
Improved exception handling related to outdated keys.txt which fixes issue #29 and #40
General system stability improvements to enhance the user's experience.

NSZ 2.1.1:

Fixed block compression for pip and Nuitka (nsz_win64_portable) by using sys.argv[0] instead of __main__.__file__ so threads no longer need to be prevented from initializing their own nut environment
Made installation instructions easier to see and understand
Scripts to automate testing and publishing

NSZ 2.1:

98 commits worth of changes since v2.0
Added pip support
Added Nuitka standalone windows build support
Enhanced File Existing Check #20
- Skip already compressed/decompressed files by default
- --overwrite
- --rm-old-version
- Extracting TitleIDs and Versions from filename if possible (#17 and #19)
  - The titleID checking is now immensely faster than when extracting from Cnmt
- --parseCnmt to get TitleID/version from Cnmt if not extractable from filename
  - Otherwise it falls back to simple filename checking which is much faster
Batch error handling with tracebacks (#16)
- Some debugging codes to find out erroneous files
- Prevent batch process raising errors
- Batch error handling with tracebacks
The --thread option now works even for solid compression however the progress bar still has some visual glitches
NSP/NSZ file hash verification now uses the hashes inside Cnmt instead of the nca filename (#22)
Fixed decompression memory leak (#21)
- The memory leak only occurs for dctx.stream_reader and was fixed by switching to the simple decompressing API which makes more sense for block decompression anyways
Improved pageReadSize calculation speed by using math instead of a while loop
Added option --remove-source that deletes the source file after compression or decompression (#24)
- For this we finally properly closed file containers too
Fix huge compression memory leak (#13)
Fixed wrong working directory when starting nut.py from a different directory (#18)
Improved exception handling during TitleID/Version extraction
- Added support for prod.keys
Reorganized file structure
Fixed keys.txt path to always be the folder containing nsz.py
Fixed default thread amount to be cpu_count()
Implemented python version checking to prevent python from showing the users confusing compatibility related exceptions
- Compatible and tested with Python 3.6 and later
General system stability improvements to enhance the user's experience.

NSZ 2.0:

Fully implemented block compression which can be enabled using the --block option
- Supports for random read access on compressed files
- Highly multithreaded compression when block compressing
- Technically supports playing compressed games in the future
- Current title installers do not support this yet
- Comes with a low compression ratio cost
Implemented NSP/NSZ file hash verification
Overwrite/Duplicate protection
Reorganized the project's folder structure and enhanced code readability
Improved user feedback in form of better understandable messages and errors
Fixed a lot of bugs and non-working features
Added MIT License so all code inside this project can be used for whatever you like
General system stability improvements to enhance the user's experience.

NSZ 1.0:

Scripts to compress and decompress NSZ files.

Differences between NSZ and NSPZ:

NSZ/XCZ:

GitHub Project: https://github.com/nicoboss/nsz
Uses solid compression by default. Block compression can be enabled using the -B option. Block compression will be the default for XCZ
Decrypts all sections while keeping the first 0x4000 bytes encrypted. Puts information needed to encrypt inside the header.
Deleted NDV0 fragments as they have no use for end users as they only exist to save CDN bandwidth
Already widely used. Supported by Tinfoil, SX Installer v3.0.0 and probably a lot of other software in the future

NSPZ/XCIZ:

GitHub Project: https://github.com/nicoboss/nsZip
Always uses Block compression allowing random read access to play compressed games in the future
Decrypts the whole NCA
Trims NDV0 fragments to their header and reconstructs them
Only supported by nsZip and unfortunately doesn't really have a future