Swiftpack.co - Package - holzschu/ios_system

ios_system: Drop-in replacement for system() in iOS programs

Platform: iOS Build Status
Twitter

When porting Unix utilities to iOS (vim, TeX, python...), sometimes the source code executes system commands, using system() calls. These calls are rejected at compile time, with: error: 'system' is unavailable: not available on iOS.

This project provides a drop-in replacement for system(). Simply add the following lines at the beginning of you header file:

extern int ios_system(char* cmd);
#define system ios_system

link with the ios_system.framework, and your calls to system() will be handled by this framework.

Commands available: shell commands (ls, cp, rm...), archive commands (curl, scp, sftp, tar, gzip, compress...) plus a few interpreted languages (python, lua, TeX). Scripts written in one of the interpreted languages are also executed, if they are in the $PATH.

The commands available are defined in two dictionaries, Resources/commandDictionary.plist and Resources/extraCommandsDictionary.plist. At startup time, ios_system loads these dictionaries and enables the commands defined inside. You will need to add these two dictionaries to the "Copy Bundle Resources" step in your Xcode project.

Each command is defined inside a framework. The framework is loaded when the command is called, and released after the command exits. Frameworks for small commands are in this project. Frameworks for interpreted languages are larger, and available separately: python, lua and TeX. Some commands (curl, python) require OpenSSH and libssl2, which you will have to download and compile separately (see https://github.com/holzschu/libssh2-for-iOS, for example).

Network-based commands (nslookup, dig, host, ping, telnet) are also available as a separate framework, network_ios. Place the compiled library with the other libraries and add it to the embedded libraries of your application.

This ios_system framework has been successfully integrated into four shells, Blink, OpenTerm, Pisth and LibTerm as well as an editor, iVim. Each time, it provides a Unix look-and-feel (well, mostly feel).

Issues: In iOS, you cannot write in the ~ directory, only in ~/Documents/, ~/Library/ and ~/tmp. Most Unix programs assume the configuration files are in $HOME. So either you redefine $HOME to ~/Documents/ or you set configuration variables (using setenv) to some other place. This is done in the initializeEnvironment() function.

Here's what I have:

setenv PATH = $PATH:~/Library/bin:~/Documents/bin
setenv PYTHONHOME = $HOME/Library/
setenv SSH_HOME = $HOME/Documents/
setenv CURL_HOME = $HOME/Documents/
setenv HGRCPATH = $HOME/Documents/.hgrc/
setenv SSL_CERT_FILE = $HOME/Documents/cacert.pem

Your Mileage May Vary. Note that iOS already defines $HOME and $PATH.

scp and sftp:

scpand sftp are implemented by rewriting them as curl commands. For example, scp user@host:~/distantfile localfile becomes (internally) curl scp://user@host/~/distantFile -o localFile. This was done to keep the size of the framework as small as possible. It will work for most of the users and most of the commands. However, it has consequences:

  • scp from distant file to distant file probably won't work, only from local to distant or distant to local.
  • The flags are those from curl, not scp. Except -q (quiet) which I remapped to -s (silent).
  • The config file is .curlrc, not .ssh/config.
  • The library used internally is libssh2, not OpenSSH.

Installation:

The easy way: (Xcode 12 and above) ios_system is available as a set of binary frameworks. Add this project as "Swift Package dependency", and link and embed the frameworks as you need them.

The hard way:

  • Open the Xcode project ios_system.xcodeproj and hit build. This will create the ios_system framework, ready to be included in your own projects.
  • Compile the other targets as well: files, tar, curl, awk, shell, text, ssh_cmd. This will create the corresponding frameworks.
  • Alternatively, type xcodebuild -project ios_system.xcodeproj -alltargets -sdk iphoneos -configuration Release -quiet to build all the frameworks.
  • If you need python, lua, TeX or network_ios, download the corresponding projects and compile them. All these projects need the ios_system framework to compile.

Integration with your app:

  • Link your application with the ios_system.framework framework.
  • Embed (but don't link) the frameworks corresponding to the commands you need (libtar.dylib if you need tar, libfiles.dylib for cp, rm, mv...).
  • Add the two dictionaries, Resources/commandDictionary.plist and Resources/extraCommandsDictionary.plist to the "Copy Bundle Resources" step in your Xcode project.

Basic commands:

The simplest way to integrate ios_system into your app is to just replace all calls to system() with calls to ios_system(). If you need more control and information, the following functions are available:

  • initializeEnvironment() sets environment variables to sensible defaults.
  • ios_executable(char* inputCmd) returns true if inputCmd is one of the commands defined inside ios_system.
  • NSArray* commandsAsArray() returns an array with all the commands available, if you need them for helping users.
  • NSString* commandsAsString() same, but with a NSString*.
  • NSString* getoptString(NSString* command) returns a string containing all accepted flags for a given command ("dfiPRrvW" for "rm", for example). Letters are followed by ":" if the flag cannot be combined with others.
  • NSString* operatesOn(NSString* command) tells you what this command expects as arguments, so you can auto-complete accordingly. Return values are "file", "directory" or "no". For example, "cd" returns "directory".
  • int ios_setMiniRoot(NSString* mRoot) lets you set the sandbox directory, so users are not exposed to files outside the sandbox. The argument is the path to a directory. It will not be possible to cd to directories above this one. Returns 1 if succesful, 0 if not.
  • FILE* ios_popen(const char* inputCmd, const char* type) opens a pipe between the current command and inputCmd. (drop-in replacement for popen).

More advance control:

replaceCommand: replaceCommand(NSString* commandName, int (*newFunction)(int argc, char *argv[]), bool allOccurences) lets you replace an existing command implementation with your own, or add new commands without editing the source.

Sample use: replaceCommand(@"ls", gnu_ls_main, true);: Replaces all calls to ls to calls to gnu_ls_main. The last argument tells whether you want to replace only the function associated with ls (if false) or all the commands that used the function previously associated with ls(if true). For example, compress and uncompress are both done with the same function, compress_main (and the actual behaviour depends on argv[0]). Only you can know whether your replacement function handles both roles, or only one of them.

If the command does not already exist, your command is simply added to the list.

addCommandList: NSError* addCommandList(NSString* fileLocation) loads several commands at once, and adds them to the list of existing commands. fileLocation points to a plist file, with the same syntax as Resources/extraCommandsDictionary.plist: the key is the command name, and is followed by an Array of 4 Strings: name of the framework, name of the function to call, list of options (in getopt() format) and what the command expects as argument (file, directory, nothing). The last two can be used for autocomplete. The name of the framework can be MAIN if your command is defined in your main program (equivalent to the RTLD_MAIN_ONLY option for dlsym()), or SELF if it is defined inside ios_system.framework (equivalent to RTLD_SELF).

Example:

<key>rlogin</key>
  <array>
    <string>network_ios.framework/network_ios</string>
    <string>rlogin_main</string>
    <string>468EKLNS:X:acde:fFk:l:n:rs:uxy</string>
    <string>no</string>
  </array>

ios_execv(const char path, char const argv[]): executes the command in argv[0] with the arguments argv (it doesn't use path). It is not a drop-in replacement for execv because it does not terminate the current process. execv is usually called after fork(), and execv terminates the child process. This is not possible in iOS. If dup2 was called before execv to set stdin and stdout, ios_execv tries to do the right thing and pass these streams to the process started by execv.

ios_execve also exists, but is just a pointer to ios_execv (we don't do anything with the environment for now).

Adding more commands:

ios_system is OpenSource; you can extend it in any way you want. Keep in mind the intrinsic limitations:

  • Sandbox and API limitations still apply. Commands that require root privilege (like traceroute) are impossible.
  • Inside terminals we have limited interaction. Apps that require user input are unlikely to get it, or with no visual feedback. That could be solved, but it is hard.

To add a command:

  • (Optional) create an issue: https://github.com/holzschu/ios_system/issues That will let others know you're working on it, and possibly join forces with you (that's the beauty of OpenSource).
  • find the source code for the command, preferrably with BSD license. Apple OpenSource is a good place to start. Compile it first for OSX, to see if it works, and go through configuration.
  • make the following changes to the code:
    • change the main() function into command_main().
    • include ios_error.h.
    • link with ios_system.framework; this will replace most function calls by ios_system version (exit, warn, err, errx, warnx, printf, write...)
    • replace calls to isatty() with calls to ios_isatty().
    • usually, this is enough for your command to compile, and sometimes to run. Check that it works.
    • if you have no output: find where the output happens. Within ios_system, standard output must go to thread_stout. libc_replacement.c intercepts most of the output functions, but not all.
    • if you have issues with input: find where it happens. Standard input comes from thread_stdin.
    • make sure you initialize all variables at startup, and release all memory on exit.
    • make all global variables thread-local with __thread, make sure local variables are marked with static.
    • make sure your code doesn't use commands that don't work in a sandbox: fork, exec, system, popen, isExecutableFileAtPath, access... (some of these fail at compile time, others fail silently at run time).
    • compile the digital library, add it to the embedded frameworks of your app.
    • Edit the Resources/extraCommandsDictionary.plist to add your command, and run.
    • That's it.
    • Test a lot. Side effects can appear after several launches.

Frequently asked commands: here is a list of commands that are often requested, and my experience with them:

  • ping, nslookup, telnet: now provided in the network_ios package.
  • traceroute and most network analysis tools: require root privilege, so impossible inside a sandbox.
  • unzip: use tar -xz.
  • nano, ed: require user interaction, so currently impossible.
  • vim: like ed, but even more difficult (needs to access the entire screen, need to add lines to the keyboard for Escape, Tab... iVim is on the App Store, and can be accessed from inside OpenTerm using the share command. My fork of iVim can launch shell commands with :!. It's easier to make an editor start commands than to make a terminal run an editor.
  • sh, bash, zsh: shells are hard to compile, even without the sandbox/API limitations. They also tend to take a lot of memory, which is a limited asset.
  • git: WorkingCopy does it very well, and you can transfer directories to your app, then transfer back to WorkingCopy. Also difficult to compile.

Licensing:

ios_system itself is released under the Revised BSD License (3-clause BSD license). Foe the other tools, I've used the BSD version as often as possible:

Using BSD versions has consequences on the flags and how they work. For example, there are two versions of sed, the BSD version and the GNU version. They have roughly the same behaviour, but differ on -i (in place): the GNU version overwrites the file if you don't provide an extension, the BSD version won't work unless you provide the extension to use on the backup file (and will backup the input file with that extension).

Github

link
Stars: 448

Dependencies

Used By

Total: 0

Releases

Binary frameworks for Xcode 12 GM - 2020-07-06 21:10:57

In this version, we move to binary frameworks and Swift packages, for easier integration in your code.

Release 2.5 of ios_system and libraries - 2020-06-10 15:16:10

Contains a precompiled version of all the libraries and frameworks associated with ios_system:

https://github.com/holzschu/ios_system https://github.com/holzschu/lib-tex https://github.com/holzschu/ctags https://github.com/holzschu/python_ios https://github.com/holzschu/python3_ios https://github.com/holzschu/network_ios https://github.com/holzschu/lua_ios https://github.com/holzschu/bc/

Changes compared to version 2.4: Focus on the ability to run several commands simultaneously. Command parsing is thread-safe, waitpid() is more robust. Somme commands have been added: vim, taskwarrior, lex, ar.

Python3 is designed so that Jupyter can run. This required creating several copies of the Python3 framework, so several versions of Python3 can run simultaneously.

2 versions of pre-compiled binaries available:

release = iOS_system.framework and all the associated libraries. Useful when you're building an application, such as a shell, ivim... smallRelease = ios_system.framework + openssl.framework + libssh2.framework. Useful when you need to compile a new library.

Release 2.4 of the ios_system libraries - 2019-06-25 11:03:17

Contains a precompiled version of all the libraries and frameworks associated with ios_system:

https://github.com/holzschu/ios_system https://github.com/holzschu/lib-tex https://github.com/holzschu/ctags https://github.com/holzschu/python_ios https://github.com/holzschu/python3_ios https://github.com/holzschu/network_ios https://github.com/holzschu/lua_ios https://github.com/holzschu/bc/

Changes compared to version 2.3: several updates to ios_system, making it more robust. Includes a fake process id system, with ios_fork() and ios_waitpid().

Python3 is designed so that Jupyter can run. This required creating several copies of the Python3 framework, so several versions of Python3 can run simultaneously.

2 versions of pre-compiled binaries available:

  • release = iOS_system.framework and all the associated libraries. Useful when you're building an application, such as a shell, ivim...
  • smallRelease = ios_system.framework + openssl.framework + libssh2.framework. Useful when you need to compile a new library.

The TeX libraries have been converted to frameworks, so it should be possible to use them in an AppStore application. ctags is now the only library not converted to frameworks.

New commands, Python3, Jupyter. - 2019-01-18 15:46:46

Release 2.3 of the ios_system libraries:

Contains a precompiled version of all the libraries and frameworks associated with ios_system:

https://github.com/holzschu/ios_system https://github.com/holzschu/lib-tex https://github.com/holzschu/ctags https://github.com/holzschu/python_ios https://github.com/holzschu/python3_ios https://github.com/holzschu/network_ios https://github.com/holzschu/lua_ios https://github.com/holzschu/bc/

Changes compared to version 2.2: new commands ifconfig, openurl, python3.

Python3 is designed so that Jupyter can run. This required creating several copies of the Python3 framework, so several versions of Python3 can run simultaneously.

2 versions of pre-compiled binaries available:

  • release = iOS_system.framework and all the associated libraries. Useful when you're building an application, such as a shell, ivim...
  • smallRelease = ios_system.framework + openssl.framework + libssh2.framework. Useful when you need to compile a new library. The TeX libraries and ctags have not been converted to frameworks, as no application on the AppStore is using them currently. It might change in the future.

Updated commands, bug fixes. - 2018-12-09 15:08:10

Release 2.2 of the ios_system libraries:

Contains a precompiled version of all the libraries and frameworks associated with ios_system:

https://github.com/holzschu/ios_system https://github.com/holzschu/lib-tex https://github.com/holzschu/ctags https://github.com/holzschu/python_ios https://github.com/holzschu/network_ios https://github.com/holzschu/lua_ios https://github.com/holzschu/bc/

Three major changes:

  • we now use frameworks instead of dynamic libraries; apparently naked dynamic libraries are forbidden on the AppStore (see issue #103 in OpenTerm: https://github.com/louisdh/terminal/issues/103 )
  • the list of commands and their associated libraries has been moved outside of the source code. So adding new commands can be done without having to recompile the ios_system framework.
  • we removed calls to private APIs (there is now an automatic check on app submission).

Smaller changes:

  • New commands: bc, dc, xargs, say.

  • New global variable: sideLoading. It is supposed to be false if you plan to release through the AppStore, true if you're downloading software to your own device. If false, we load a smaller number of commands and output less debugging information.

  • The command "groups" was removed.

  • There are two lists of commands, both of them under Resources: commandDictionary.plist and extraCommandsDictionary.plist. The former is always loaded, the latter only if sideLoading is true.

Both files have the following format:

        <key>chksum</key>
        <array>
                <string>files.framework/files</string>
                <string>chksum_main</string>
                <string>o:</string>
                <string>file</string>
        </array>

As before, 2 versions of pre-compiled binaries available:

  • release = iOS_system.framework and all the associated libraries. Useful when you're building an application, like shell, ivim...
  • smallRelease = ios_system.framework + openssl.framework + libssh2.framework. Useful when you need to compile a new library.

The TeX libraries and ctags have not been converted to frameworks, as no application on the AppStore is using them currently. It might change in the future.

Moved to frameworks + extract list of commands - 2018-03-27 20:11:05

Release 2.1 of the ios_system libraries:

Contains a precompiled version of all the libraries and frameworks associated with ios_system:

https://github.com/holzschu/ios_system https://github.com/holzschu/lib-tex https://github.com/holzschu/ctags https://github.com/holzschu/python_ios https://github.com/holzschu/network_ios https://github.com/holzschu/lua_ios

Three major changes:

  • we now use frameworks instead of dynamic libraries; apparently naked dynamic libraries are forbidden on the AppStore (see issue #103 in OpenTerm: https://github.com/louisdh/terminal/issues/103 )
  • the list of commands and their associated libraries has been moved outside of the source code. So adding new commands can be done without having to recompile the ios_system framework.
  • we removed calls to private APIs (there is now an automatic check on app submission).

Smaller changes:

  • New global variable: sideLoading. It is supposed to be false if you plan to release through the AppStore, true if you're downloading software to your own device. If false, we load a smaller number of commands and output less debugging information.

  • The command "groups" was removed.

  • There are two lists of commands, both of them under Resources: commandDictionary.plist and extraCommandsDictionary.plist. The former is always loaded, the latter only if sideLoading is true.

Both files have the following format:

        <key>chksum</key>
        <array>
                <string>files.framework/files</string>
                <string>chksum_main</string>
                <string>o:</string>
                <string>file</string>
        </array>

As before, 2 versions of pre-compiled binaries available:

  • release = iOS_system.framework and all the associated libraries. Useful when you're building an application, like shell, ivim...
  • smallRelease = ios_system.framework + openssl.framework + libssh2.framework. Useful when you need to compile a new library.

The TeX libraries and ctags have not been converted to frameworks, as no application on the AppStore is using them currently. It might change in the future.

Multi-session ability - 2018-03-27 14:36:19

Release 1.3 of the ios_system libraries.

Contains a precompiled version of all the libraries and frameworks associated with ios_system:

https://github.com/holzschu/ios_system https://github.com/holzschu/lib-tex https://github.com/holzschu/ctags https://github.com/holzschu/python_ios https://github.com/holzschu/network_ios https://github.com/holzschu/lua_ios

Compared to v1.2: If you want to run ios_system in multiple sessions (tabs, etc): all variables are stored in a structure (sessionParameters). New command:

  • switchSession(void* identifier): tells ios_system that you're moving to a different session. Identifier can be anything, as long as it's unique to the session and stays valid while the session is valid.
  • closeSession(void* identifier): tells ios_system that the session has been closed, and to release the associated data.

This release is API-compatible with previous releases: if you never call switchSession, ios_system works just like before.

As before, 2 versions of pre-compiled binaries available:

  • release = iOS_system.framework and all the associated libraries. Useful when you're building an application, like shell, ivim...
  • smallRelease = ios_system.framework + openssl.framework + libssh2.framework. Useful when you need to compile a new library.

More commands: ping, nslookup, ctags... - 2018-03-19 16:34:25

Release 1.2 of the ios_system libraries.

Contains a precompiled version of all the libraries and frameworks associated with ios_system:

  • https://github.com/holzschu/ios_system
  • https://github.com/holzschu/lib-tex
  • https://github.com/holzschu/ctags
  • https://github.com/holzschu/python_ios
  • https://github.com/holzschu/network_ios
  • https://github.com/holzschu/lua_ios

Compared to v1.1:

  • commands now return the proper error code.
  • ios_system handles files with spaces in their names (provided the spaces are escaped (new\ file) or the filename is enclosed within quotes ("new file"). Backslashes have to be escaped.
  • fixed a bug where the "awk" command had disappeared.
  • added commands: ctags, nslookup, ping, dig, host.

As before, 2 versions of pre-compiled binaries available:

  • release = iOS_system.framework and all the associated libraries. Useful when you're building an application, like shell, ivim...
  • smallRelease = ios_system.framework + openssl.framework + libssh2.framework. Useful when you need to compile a new library.

Added pipes, color ls, bug fixes - 2018-02-11 19:54:15

2 packages available:

  • release = iOS_system.framework and all the associated libraries. For applications, like shell, ivim...
  • smallRelease = ios_system. framework + openssl. framework + libssh2.framework. Useful when you need to compile a new library.

First release. All dynamic frameworks. - 2018-02-02 11:52:37

First release, using dynamic libraries and frameworks. Simply link with iOS_system.framework, embed all the other dynamic libraries and run.