SWCompression

A couple of side notes before diving into release notes:

1. I've started a github project board where I am going to track and plan changes and additions for 5.0 Update.
2. If you ever wanted to financially support either this project or me you can now do so using this link.

Creating TAR containers

The main addition in this update is a set of APIs which allow to create a new TAR container.

• Added TarContainer.create(from:) function which creates a new TAR container with provided array of TarEntry objects as its content and generates container's Data.
• Added TarCreateError error type with a single case utf8NonEncodable. Comment: This enum is planned to be merged with TarError in 5.0. A new enum had to be created since otherwise it would be a breaking change to introduce a new case to already existing enum.

To enable reasonable usage scenarios for these new APIs, additional changes to existing APIs have been made:

• TarEntry.info and TarEntry.data are now var-properties (instead of let).
• Accessing setter of TarEntry.data now automatically updates TarEntry.info.size with data.count value (or 0 if data is nil). Comment: Maintaining consistency between these two properties is extremely important for producing correct and valid containers.
• Added (or, rather, made public) TarEntry.init(info:data:) initializer.
• Most public properties of TarEntryInfo are now var-properties (instead of let). Exceptions: size and type. Comment: Property size is kept read-only for reasons mentioned above. The reason for not allowing mutating type property is more vague: it is hard to imagine usage scenario where changing the type of an entry makes sense. Moreover, there are some concerns about (potential future) behavior in more generic context with type-erased ContainerEntryInfo objects, etc.
• Added TarEntryInfo.init(name:type:) initializer.

I do realize that this set of APIs is somewhat limited. For example, it is not easy to convert ZIP container (array of ZipEntry objects) to TAR using these new additions. But rest assured, there are plans to provide more generic functionality for creating new containers in the future (something like TarContainer.create(from entries: [ContainerEntry]) throws -> Data).

Other Changes

• Improved compatibility with other TAR implementations:
  • All string fields of TAR headers are now treated as UTF-8 strings. Comment: This is compatible with previous behavior since ASCII strings are UTF-8 strings.
  • Non-well-formed numeric fields of TAR headers no longer cause TarError.wrongField to be thrown and instead result in nil values of corresponding properties of TarEntryInfo (exception: size field). Comment: Mainly, this change was made to accommodate situations when a TAR header field is absent (i.e. filled with NULLs). Absent size field is still not accepted since its value impacts the structure of the container. This particular behavior is consistent with other implementations.
  • Base-256 encoding of numeric fields, which is sometimes used for very big or negative values, is now supported.
  • Leading NULLs and whitespaces in numeric fields are now correctly skipped.
  • Sun Extended Headers are now processed as local PAX extended headers instead of being considered entries with .unknown type.
  • GNU TAR format features for incremental backups are now partially supported (access and creation time).
• TarContainer.formatOf now correctly returns TarFormat.gnu when GNU format "magic" field is encountered.
• A new (copy) Data object is now created for TarEntry.data property instead of using a slice of input container data. Comment: This change makes indices of TarEntry.data zero-based which is consistent with other containers. This should also prevent keeping in memory Data for the entire container until the TarEntry object is destroyed.
• Fixed incorrect file name of TAR entries from containers with GNU TAR format-specific features being used.
• Fixed TarError.wrongPaxHeaderEntry error being thrown when header with multi-byte UTF-8 characters is encountered.
• Fixed incorrect values of TarEntryInfo.ownerID, groupID, deviceMajorNumber and deviceMinorNumber properties (previously, they were assumed to be encoded as decimal numbers).
• Slightly improved performance of LZMA/LZMA2 operations by making internal classes declared as final.
• swcomp changes:
  • Added -c, --create option to tar command which creates a new TAR container.
  • Output of bencmark commands is now properly flushed on non-Linux platforms.
  • Results for omitted iterations of benchmark commands are now also printed.
  • Iteration number in benchmark commands is now printed with leading zeros.
  • Fixed compilation error on Linux platforms due to ObjCBool no longer being an alias for Bool.

This is the first and only test release for the upcoming 4.4.0 update. It includes functionality for creating new TAR containers as well as numerous fixes for TAR open/info functions.

Known issue: no documentation for new APIs.

ZIP Custom Extra Fields

ZIP format provides capabilities to define third-party extra fields, so it is impossible for SWCompression to support all possible extra fields. In this update several APIs were added which allow users to define their own extra fields (aka "custom extra fields") and make SWCompression recognize them. All extra fields previously supported by SWCompression (aka "standard extra fields") are still supported.

• Added ZipExtraField protocol.
• Added ZipExtraFieldLocation enum.
• Added ZipContainer.customExtraFields property.
• Added ZipEntryInfo.customExtraFields property.

To add support of a custom extra field one must first create a type which conforms to ZipExtraField protocol. Then it must be added to ZipContainer.customExtraFields dictionary with the key equal to the id property of the type being added. If during execution of open(container:) or info(container:) functions custom extra field is found it will be processed using initializer of the provided type and stored in ZipEntryInfo.customExtraFields property of entry where this extra field was found.

Note: It is impossible to define custom extra field with the same ID as any of the standard extra fields and make SWCompression use user-defined extra field instead of the standard one (i.e. SWCompression first checks if ID is one of the standard IDs and then tries to find it in ZipContainer.customExtraFields dictionary).

TAR Formats

• Added TarContainer.Format enum which represents various formats of TAR containers.
• Added TarContainer.formatOf(container:) function which returns format of the TAR container.
• Added -f, --format option to swcomp's tar command which prints format of the TAR container.

Comment: In the context of TAR containers "format" means a set of extensions to the basic TAR container layout which must be supported to successfully process given container.

Benchmark changes

• Number of benchmark iterations increased from 6 to 10.
• Benchmarks now have a zeroth iteration which is excluded from averages.

Comment: For some reason when benchmarked functions are being executed for the first time they perform significantly worse than any of the following iterations. So it was decided to drop this "zeroth" iteration from calculating of averages. This change, of course, artificially improves benchmark results, but, hopefully, makes them more reliable. On the other hand, the increase in number of iterations aims to improve accuracy of benchmarks in general.

Other changes

• Updated to support Swift 4.1.
• Minuimum required version of BitByteData is now 1.2.0.
• Added TarEntryInfo.compressionMethod property which is always equal to .copy.
• Added documenation for Container.Entry and ContainerEntry.Info associated types.
• Reverted "disable symbol stripping" change from 4.2.0 update, since underlying problem was fixed in Carthage.

In this test release a new function and enum have been added to TarContainer which provide information about formats of TAR containers.

This is the first test release of 4.3.0 which includes support for ZIP custom extra fields, updates for Swift 4.1 and upcoming 1.2.0 release of BitByteData.