SWCompression

In this update several crashes were fixed that could occur in certain edge cases. These include situations when trying to process a very small or clearly invalid data. In these cases an error is now thrown instead. Unfortunately, there aren't suitable errors for these types of situations in SWCompression and it is impossible to add new error cases to the existing enums in a patch release, and as such the errors that are thrown may look a bit out of place.

• Fixed a crash when trying to open a very small file as a 7-Zip container, a BZip2 "archive", a GZip archive (either when using unarchive or multiUnarchive), a LZMA "archive", a XZ archive (either when using unarchive or splitUnarchive), a ZIP container, or a Zlib archive.
• Fixed a crash when trying to open an unexpectedly truncated file as a GZip or XZ archive.
• Fixed a crash when trying to open an invalid file as a ZIP container. The ZipError.notFoundCentralDirectoryEnd error is now thrown instead as was originally intended.
• Fixed a crash when trying to read a GZip or Zlib header from a very small file.
• Fixed a compilation warning in BZip2.

• Fixed an issue where SWCompression couldn't be built on Linux using Swift 5.3 (#13).

Please note, that there is currently an issue with Carthage not working properly with Xcode 12. Refer to this issue thread for more details and a workaround.

• Fixed a crash when opening ZIP files created by Finder on certain versions of macOS.

Comment: ZIP files produced by Finder (or whatever it uses under the hood) on certain versions of macOS sometimes contain an Extended Timestamp extra field in their Central Directory which doesn't strictly conform to the specifications. More specifically, extended timestamp located in CD can only contain modification time, but Finder puts there access and creation times as well. When trying to open such files, SWCompression was crashing. Now, instead, SWCompression ignores these extra additions.

• Fixed Swift Package Manager and Xcode sometimes being unable to add SWCompression as a package dependency (#9).

Comment: This was commonly happening when specific settings related to Git LFS were present in the git config, but SPM/Xcode were unable to find Git LFS in their PATH environment variable for any reason. Fixing this required changing the development workflow of SWCompression, so if you are working on SWCompression itself, please read the updated "Executing tests locally" section in the README.

• Fixed incompatibility with Swift Package Manager from Swift 4.2.

Comment: At the moment SWCompression supports Swift 4.2 (only via CocoaPods and SPM) and Swift 5.X.

Several changes have been made to improve compatibility with Swift 5.x and Xcode versions newer than 10.1 (including Xcode 11).

Testing that nothing is broken before releasing 4.5.3.

• Increased the lowest required version of BitByteData dependency to 1.4.1.

Comment: This version of BBD fixes its gross incompatibility with Swift 5.

• Minimum required version of BitByteData is now 1.4.0.
• Updated to support Swift 4.2.
• Added default values to the properties of LZMAProperties struct based on the documentation from LZMA SDK.
• Added init() to LZMAProperties struct which sets lc, lp, pb, and dictionarySize properties to their default values.
• Improved the detection of Swift versions less than 4.2 in the workaround for the crash in Data.prefix(upTo:).
• Documentation updates:
  • Fixed an outdated example in README (PR #4 by @brianantonelli).
  • Fixed grammar issues related to the usage of articles, during/while, and others.
• swcomp changes:
  • Minimum required version of SwiftCLI is now 5.2.0.
  • Improved the layout of output of benchmark commands.

• Added LZMAProperties struct with simple member-wise initializer.
• Added LZMA.decompress(data:properties:uncompressedSize:) function (with uncompressedSize argument being optional) which allows to specify LZMA properties.
  • Useful in situations when properties are known from some external source instead of being encoded at the beginning of data.
  • Note, that these new APIs are intended to be used by expert users and as such no validation is performed on LZMA properties values.
• Added support for Delta "filter" in both XZ archives and 7-Zip containers.
• Added support for SHA-256 check type in XZ archives.
  • As a result XZError.checkTypeSHA256 is now never thrown and will be removed in the next major update.
• Added ZipEntryInfo.crc property.
• Fixed a problem where XZArchive.unarchive and XZArchive.splitUnarchive functions would produce incorrect result when more than one "filter" was used (though it was practically impossible to encounter this issue since only one filter was supported (LZMA2) until this update).
• Reduced in-memory size of ZipEntryInfo instances.
  • Some rough estimates indicate that the reduction is up to 68%.
• Clarified documentation for LZMA.decompress(data:) to explain expectation about data argument.
  • Particularly, it is explained that it expects LZMA properties encoded with standard LZMA encoding scheme at the beginning of data.
• swcomp changes:
  • zip -i command now also prints CRC32 for all entries.
  • -v is now accepted as an alias for --verbose option.

This is the first and only test release for the upcoming 4.5.0 update. It includes new LZMAProperties APIs, support for SHA-256 check for XZ archives and support for delta filter in 7-ZIP and XZ, as well as a couple of fixes.

Known issue: no documentation for new APIs.

A couple of side notes before diving into release notes:

1. I've started a github project board where I am going to track and plan changes and additions for 5.0 Update.
2. If you ever wanted to financially support either this project or me you can now do so using this link.

Creating TAR containers

The main addition in this update is a set of APIs which allow to create a new TAR container.

• Added TarContainer.create(from:) function which creates a new TAR container with provided array of TarEntry objects as its content and generates container's Data.
• Added TarCreateError error type with a single case utf8NonEncodable. Comment: This enum is planned to be merged with TarError in 5.0. A new enum had to be created since otherwise it would be a breaking change to introduce a new case to already existing enum.

To enable reasonable usage scenarios for these new APIs, additional changes to existing APIs have been made:

• TarEntry.info and TarEntry.data are now var-properties (instead of let).
• Accessing setter of TarEntry.data now automatically updates TarEntry.info.size with data.count value (or 0 if data is nil). Comment: Maintaining consistency between these two properties is extremely important for producing correct and valid containers.
• Added (or, rather, made public) TarEntry.init(info:data:) initializer.
• Most public properties of TarEntryInfo are now var-properties (instead of let). Exceptions: size and type. Comment: Property size is kept read-only for reasons mentioned above. The reason for not allowing mutating type property is more vague: it is hard to imagine usage scenario where changing the type of an entry makes sense. Moreover, there are some concerns about (potential future) behavior in more generic context with type-erased ContainerEntryInfo objects, etc.
• Added TarEntryInfo.init(name:type:) initializer.

I do realize that this set of APIs is somewhat limited. For example, it is not easy to convert ZIP container (array of ZipEntry objects) to TAR using these new additions. But rest assured, there are plans to provide more generic functionality for creating new containers in the future (something like TarContainer.create(from entries: [ContainerEntry]) throws -> Data).

Other Changes

• Improved compatibility with other TAR implementations:
  • All string fields of TAR headers are now treated as UTF-8 strings. Comment: This is compatible with previous behavior since ASCII strings are UTF-8 strings.
  • Non-well-formed numeric fields of TAR headers no longer cause TarError.wrongField to be thrown and instead result in nil values of corresponding properties of TarEntryInfo (exception: size field). Comment: Mainly, this change was made to accommodate situations when a TAR header field is absent (i.e. filled with NULLs). Absent size field is still not accepted since its value impacts the structure of the container. This particular behavior is consistent with other implementations.
  • Base-256 encoding of numeric fields, which is sometimes used for very big or negative values, is now supported.
  • Leading NULLs and whitespaces in numeric fields are now correctly skipped.
  • Sun Extended Headers are now processed as local PAX extended headers instead of being considered entries with .unknown type.
  • GNU TAR format features for incremental backups are now partially supported (access and creation time).
• TarContainer.formatOf now correctly returns TarFormat.gnu when GNU format "magic" field is encountered.
• A new (copy) Data object is now created for TarEntry.data property instead of using a slice of input container data. Comment: This change makes indices of TarEntry.data zero-based which is consistent with other containers. This should also prevent keeping in memory Data for the entire container until the TarEntry object is destroyed.
• Fixed incorrect file name of TAR entries from containers with GNU TAR format-specific features being used.
• Fixed TarError.wrongPaxHeaderEntry error being thrown when header with multi-byte UTF-8 characters is encountered.
• Fixed incorrect values of TarEntryInfo.ownerID, groupID, deviceMajorNumber and deviceMinorNumber properties (previously, they were assumed to be encoded as decimal numbers).
• Slightly improved performance of LZMA/LZMA2 operations by making internal classes declared as final.
• swcomp changes:
  • Added -c, --create option to tar command which creates a new TAR container.
  • Output of bencmark commands is now properly flushed on non-Linux platforms.
  • Results for omitted iterations of benchmark commands are now also printed.
  • Iteration number in benchmark commands is now printed with leading zeros.
  • Fixed compilation error on Linux platforms due to ObjCBool no longer being an alias for Bool.

This is the first and only test release for the upcoming 4.4.0 update. It includes functionality for creating new TAR containers as well as numerous fixes for TAR open/info functions.

Known issue: no documentation for new APIs.

ZIP Custom Extra Fields

ZIP format provides capabilities to define third-party extra fields, so it is impossible for SWCompression to support all possible extra fields. In this update several APIs were added which allow users to define their own extra fields (aka "custom extra fields") and make SWCompression recognize them. All extra fields previously supported by SWCompression (aka "standard extra fields") are still supported.

• Added ZipExtraField protocol.
• Added ZipExtraFieldLocation enum.
• Added ZipContainer.customExtraFields property.
• Added ZipEntryInfo.customExtraFields property.

To add support of a custom extra field one must first create a type which conforms to ZipExtraField protocol. Then it must be added to ZipContainer.customExtraFields dictionary with the key equal to the id property of the type being added. If during execution of open(container:) or info(container:) functions custom extra field is found it will be processed using initializer of the provided type and stored in ZipEntryInfo.customExtraFields property of entry where this extra field was found.

Note: It is impossible to define custom extra field with the same ID as any of the standard extra fields and make SWCompression use user-defined extra field instead of the standard one (i.e. SWCompression first checks if ID is one of the standard IDs and then tries to find it in ZipContainer.customExtraFields dictionary).

TAR Formats

• Added TarContainer.Format enum which represents various formats of TAR containers.
• Added TarContainer.formatOf(container:) function which returns format of the TAR container.
• Added -f, --format option to swcomp's tar command which prints format of the TAR container.

Comment: In the context of TAR containers "format" means a set of extensions to the basic TAR container layout which must be supported to successfully process given container.

Benchmark changes

• Number of benchmark iterations increased from 6 to 10.
• Benchmarks now have a zeroth iteration which is excluded from averages.

Comment: For some reason when benchmarked functions are being executed for the first time they perform significantly worse than any of the following iterations. So it was decided to drop this "zeroth" iteration from calculating of averages. This change, of course, artificially improves benchmark results, but, hopefully, makes them more reliable. On the other hand, the increase in number of iterations aims to improve accuracy of benchmarks in general.

Other changes

• Updated to support Swift 4.1.
• Minuimum required version of BitByteData is now 1.2.0.
• Added TarEntryInfo.compressionMethod property which is always equal to .copy.
• Added documenation for Container.Entry and ContainerEntry.Info associated types.
• Reverted "disable symbol stripping" change from 4.2.0 update, since underlying problem was fixed in Carthage.

In this test release a new function and enum have been added to TarContainer which provide information about formats of TAR containers.

This is the first test release of 4.3.0 which includes support for ZIP custom extra fields, updates for Swift 4.1 and upcoming 1.2.0 release of BitByteData.

• Fixed a problem when in some rare cases (when SevenZipSubstreamInfo isn't found in a container) 7-Zip Container entries were skipped during open(container:), instead of being appended with nil data to the result array, as was always intended.
• Fixed a crash in SevenZipContainer.info(container:) when either entry's size or CRC32 isn't present in the container.
• Updated documentation.

• All 7-Zip minor format versions from 1 to 4 are now accepted (previously, it was only 4; for all other versions SevenZipError.wrongFormatVersion was thrown).

• Minuimum required version of BitByteData is now 1.1.0.
• Added ownerID and groupID properties to ZipEntryInfo, which provide access to uid and gid from Info-ZIP New Unix and Info-Zip Unix extra fields.
• Added unknownExtendedHeaderRecords property to TarEntryInfo which includes unrecognized entries from PAX global and local extended headers.
• Prevent double slashes in TarEntryInfo.name when prefix header's field was used, but it had trailing slash.
• Improved BZip2 compression performance: Now additional Huffman trees aren't created when they cannot be used, since maximum amount of Huffman tables was generated.
• Disabled symbol stripping in archives generated by Carthage and published on GitHub Releases.
• swcomp changes:
  • Replaced 9 block size options of bz2 command with a single one: -b (--block-size).
  • Now prints entry type-specific properties in output of tar, zip and 7z commands with -i (--info) option.
  • Renamed perf-test command group to benchmark.

The most important thing to note about this test version is the removal of "several APIs which work with ByteReaders and BitReaders instead of Data". As it was said, their addition in the previous test release was an experimental feature. Unfortunately, there were some problems with these APIs. For example, they were incosistent: some of them were changing byte(bit) reader that the user passed as an argument, while others weren't. As such, it was decided to remove them (or, rather, make them internal).

This test release of the upcoming 4.2.0 update includes following improvements:

• Added several APIs which work with ByteReaders and BitReaders instead of Data (experimental feature).
• Added support for Info-ZIP's extra fields which store uids and gids of entries.
• Now requires BitByteData version at least 1.1.0.

• Fixed incorrect value of TarEntryInfo.name in certain situations. Comment: This problem was occurring when file name was long enough to not fit into "name" field of TAR's header, but no PAX or GNU extensions were used, so the "prefix" field was used instead.
• Updated documentation for TarEntryInfo.

• Functionality related to reading and writing bits and bytes was published as a separate project, BitByteData. As a result SWCompression now has BitByteData as a dependency.
• Several performance improvements have been made. See updated test results for more information.
• Added a workaround to prevent situations when some BZip2 archives created by SWCompression could not be opened by original BZip2 implementation. Comment: It seems that some BZip2 implementations (including original one) don't account for the fact that the initial Run-Length Encoding can actually cause expansion of the input data. To prevent this from happening SWCompression now uses BZip2 blocks smaller than intended by the format.
• Modification time stored in ZIP's "native" field is now calculated relative to current system's calendar and time zone. Comment: Since standard ZIP's fields don't have any way to save time zone data it is more likely that archivers stored modification time relative to current system's settings rather than UTC-relative (which was assumed by SWCompression before). Behaviour of NTFS timestamp and Extended timestamp extra fields hasn't been changed.
• Additions to swcomp command-line tool:
  • Added -1...-9 options to bz2 command which specifies what block size to use for BZip2 compression.
  • Added -i (--info) option to gz command which prints GZip's header.
  • Added comp-deflate and comp-bz2 subcommands to perf-test command which can be used for performance testing of Deflate and BZip2 compression.
  • Updated descriptions of -e options for zip, tar and 7z commands.
  • Now sets permissions for extracted files only if they are greater than 0.

• Performance improvements:
  • Preallocate byte arrays when output size is known.
  • Inverse Burrows-Wheeler transform without sorting.
  • Store ntfs reference date as a static constant.
• "Fixed" a problem when some BZip2 archives created by SWCompression could not be opened by original BZip2 implementation.
• Modification time stored in ZIP's "native" field is now calculated relative to current system's calendar and time zone.
• Added options (-1...-9) to swcomp bz2 -c for specifying block size .
• Added option (-i, --info) to swcomp gz for printing GZip's header.
• Added commands (comp-deflate, comp-bz2) to swcomp for performance testing of Deflate and BZip2 compression.
• Corrected descriptions of extract options of swcomp's container-related commands.
• swcomp now sets permissions for extracted files only if they are greater than 0.

The first part of changes planned for 4.1.0 is extraction of some code from SWCompression into a separate project. This was done recently and you can now check out the new project, BitByteData.

The purpose of this test release is to test integration of this new external dependency into SWCompression.

• Starting with this update, git tags for releases no longer have "v" prefix, since absence of such prefix is a common practice among Swift developers.
• Fixed incorrectly thrown XZError.wrongDataSize without actually trying to decompress anything.
• Fixed crash when opening 7-Zip containers with more than 255 entries with empty streams.
• Reduced memory usage by Deflate compression.
• Improved performance in some extreme cases (e.g. containers with an enormous amount of entries).
• No longer verify if ZIP string field needs UTF-8 encoding, if language encoding flag is set for the entry.
• Added "perf-test" command to swcomp, which is used for measuring performance. See Results document for the new tests results.

Reworked "Container" API

There is a couple of ideas behind this rework:

1. Enforce in API the idea that one can get information about entries without acquiring their data, but not vice versa.
2. Provide a unified set of API for all three formats: ZIP, TAR and 7-Zip.
3. Try to fix mistakes that were made in the development of previous versions.

Implementation of these ideas led to a lot of changes and here I will try to highlight most of them.

• New protocol: ContainerEntryInfo.
  • It contains some informational properties from the previous version of ContainerEntry.
  • There are also new properties such as access/creation/modificationTime and type.

Comment: I would like to encourage you to check out this protocol's documentation as well as types that implement it: SevenZipEntryInfo, TarEntryInfo and ZipEntryInfo. These types not only have protocol's properties, but also contain their own format-specific members which may be useful in certain cases.

• ContainerEntry now has only two members: info and data properties.

Comment: All properties that existed in previous versions were either removed or moved to ContainerEntryInfo. One may also note that entry's data is now acquired through constant property rather than function. This means that it is no longer possible to asynchronously unpack ZIP containers, but it worth mentioning, that it was never intended in the first place.

• ContainerEntry now has an associated type Info: ContainerEntryInfo.
• ContainerEntry.entryAttributes removed without replacement.

Comment: One may note that it is not a SWCompression's job to prepare entry's properties for the file system. Besides that, its existence was causing duplication of entry's information, so, in the end, it is for the best for this property to go.

• Container now has an associated type Entry: ContainerEntry.
• open function now returns an array of associated type Entry.
• Added new function info which returns an array of Entry.Info.

Comment: One of the useful consequences of these changes is that it is no longer necessary to cast the result of open(container:) to specific entry type (such as ZipEntry).

• All existing ZIP, TAR and 7-Zip types conform to these protocols.
• Added missing types for ZIP, TAR and 7-Zip with conformances to these protocols.
• ZipEntry, TarEntry and SevenZipEntry now have only two members: info and data (in accordance with ContainerEntry protocol).
• Standardised behaviour when ContainerEntry.data can be nil:
  • If entry is a directory.
  • If data is unavailable, but error wasn't thrown for some reason.
  • 7-Zip only: if entry is an anti-file.
• Removed SevenZipEntryInfo.isDirectory. Use type property instead.
• SevenZipEntryInfo.name is no longer Optional.
  • Now throws SevenZipError.internalStructureError when file names cannot be properly processed.
  • Entries now have empty strings as names when no names were found in container.

Common Types

Added several new common types which are used across the framework:

• CompressionMethod.
  • Used as a type of compressionMethod property of GzipHeader, ZlibHeader and ZipEntryInfo.
  • Removed GzipHeader.CompressionMethod.
• ContainerEntryType.
  • Used as a type of type property of ContainerEntryInfo.
  • Removed TarEntry.EntryType.
• DosAttributes.
  • It is the same as SevenZipEntryInfo.DosAttributes type from previous version.
  • Used as a type of dosAttributes property of SevenZipEntryInfo and ZipEntryInfo.
• Permissions.
  • It is the same as SevenZipEntryInfo.Permissions type from previous version.
  • Used as a type of permissions property of ContainerEntryInfo.
• FileSystemType.
  • Used as a type of GzipHeader.osType and ZipEntryInfo.fileSystemType properties.
  • Removed GzipHeader.FileSystemType.

Errors

• Removed following errors:
  • SevenZipError.dataIsUnavailable
  • LZMAError.decoderIsNotInitialised
  • LZMA2Error.wrongProperties (LZMA2Error.wrongDictionarySize is thrown instead).
  • TarError.wrongUstarVersion.
  • TarError.notAsciiString (TarError.wrongField is thrown instead).
  • XZError.wrongFieldValue (XZError.wrongField is thrown instead).
• Renamed following errors:
  • BZip2Error.wrongHuffmanLengthCode to BZip2Error.wrongHuffmanCodeLength.
  • BZip2Error.wrongCompressionMethod to BZip2Error.wrongVersion.
  • TarError.fieldIsNotNumber to TarError.wrongField.
  • XZError.reservedFieldValue to XZError.wrongField.
• Standardised behaviour for errors with names similar to wrongCRC:
  • These errors mean that everything went well, except for comparing the checksum.
  • Their associated values now contain all "successfully" unpacked data, including the one which caused checksum error.
    • This change affects BZip2.decompress, GzipArchive.multiUnarchive, XZArchive.unarchive, XZArchive.splitUnarchive, ZipContainer.open.
  • Some of these errors now have arrays as associated values to account for the situations with unpacked data being split.
    • This change affects GzipArchive.multiUnarchive, XZArchive.unarchive, XZArchive.splitUnarchive, ZipContainer.open.

General changes

• Renamed XZArchive.multiUnarchive to XZArchive.splitUnarchive.
• XZArchive.unarchive now processes all XZ streams similar to splitUnarchive, but combines them into one output Data.

Comment: These two changes are made to fix intended but apparently incorrect behaviour from previous versions, which was causing inconveniences with some archives.

• Fixed "bitReader is not aligned" precondition crash in Zlib.
• Fixed potential incorrect behavior when opening ZIP containers with size bigger than 4 GB.
• Updated to use Swift 4.
• Various improvements to documentation.
• "swcomp" is now included as part of this repository.

Comment: This was done to make it easier to synchronise changes between SWCompression and swcomp, if necessary. "swcomp" is not built by default.

• Changes to Errors:
  • Removed:
    • SevenZipError.dataIsUnavailable
    • LZMAError.decoderIsNotInitialised
    • LZMA2Error.wrongProperties (LZMA2Error.wrongDictionarySize is thrown instead.)
  • Renamed BZip2Error.wrongHuffmanLengthCode to BZip2Error.wrongHuffmanCodeLength.
• ContainerEntryInfo.name is no longer Optional.
  • 7z now throws internalStructureError when it is unable to process file names.
• Various FileSystemType variables are no longer Optional.
  • They have .other value where it was nil previously.
• TarEntryInfo.linkName is no longer Optional.
• Fixed "bitReader is not aligned" precondition crash in Zlib.
• ZlibHeader.compressionMethod now uses common CompressionMethod enum instead of its own.

Upcoming 4.0.0 update will include major rework of "Container API" as well as changes here and there. It will also have some internal changes which are tested by this test release.

• Implementation of BZip2 compression algorithm.
  • There is also an auxiliary function BZip2.compress(data:blockSize:) if you need to specify size of input data block.
  • Default block size is 100 Kilobytes.
• Added CompressionAlgorithm protocol.
• Deflate now conforms to CompressionAlgorithm protocol (as well as BZip2).
• Deflate.compress(data:) and ZlibArchive.archive(data:) no longer throw, but can crash with fatalError(). Comment: Though unlikely to happen, it seems more logical to crash with fatalError() when problems occur during compression instead of error throwing because it indicates there is a problem with the code itself, not with the input data.
• Fixed crash in some rare cases for corrupted BZip2 archives. It now throws BZip2Error instead.

• Fixed compile problems on some platforms (iOS, watchOS). Comment: These problems are related to the fact that on those platforms Int is 32-bit instead of 64-bit.