of SIMPHONE (version 0.8)

1. Directory Structure




Please refer to build/BUILD.txt for a description.


Third-party asynchronous DNS resolver library (c-ares).


Third-party C++ cryptographic library (crypto++).


Third-party mainline DHT implementation (from transmission-bt).


Documentation in HTML format.


Third-party memory debugger for Windows (not used by default).


Third-party XML parser library (libexpat).


Third-party UPnP router traversal library.


Third-party coroutines implementation (refer to npth/README).


Third-party TLS implementation (libcrypto + libssl).


Please refer to patches/PATCHES.txt for a description.


Third-party real-time audio library (Windows, Mac OS and unix).


Source code to the simphone GUI (using Qt).


This directory will be created when you compile qtbase.


You need to unpack the Qt GUI toolkit source code here.


Third-party Qt library that implements "single start".


Source code that implements all functions of Simphone.


Third-party audio codec used by Simphone.


Third-party software DSP functions used with the speex codec.

2. qsimphone

The first few lines of each .cpp file contain a comment describing which classes are implemented by that source file and what purpose they serve.

Each source file usually has an associated .h file of the same base name, which defines those classes. There are some comments that describe the structure fields there.

The rest of the files (.ui) contain Qt resources (definitions of GUI elements in XML format).

The flags subdirectory contains all country flags in PNG format.

The icons subdirectory contains all other icons used by the GUI, mostly in PNG format.

The resources subdirectory contains the IP address database in binary format.

When changing the source code of the GUI or simcore, please follow the existing coding style. clang-format must be run on the source code to format it properly.

3. simcore Overview

The first two lines of each .c file contain a comment describing the purpose of that module. Each module usually has an associated .h file of the same base name, which defines functions exported by the module, as well as data structures (if any). Structure fields are commented; most of the function definitions also include a short comment describing the purpose of each function.

Interaction between the GUI and simcore is done through the simcore public API (please refer to api.html for a description).

The .h files define the internal API of simcore; it is used for interaction between different modules. Functions are called to perform actions, while data structures are passed as arguments to those functions and are used to pass data between different modules. Usually only the module which defines a structure has the job of writing to it. In object-oriented terminology, some of these structures would define the "classes" of simcore.

4. Special simcore Modules

spth.c and npth

The nPth library used by simcore provides pseudo-threads (which are really coroutines) using real (preemptive) threads, as provided by the operating system. It implements functionality of another library called Pth, but not its API. In order to allow simcore to use either nPth or Pth, the glue code in spth.c implements some of the Pth API, using nPth as an implementation.

Additionally, it provides access to the npth_protect and npth_unprotect functions (not available under Pth) and brings the Pth API in line with the simcore "underscore" naming convention (described under The simcore Naming Convention), for better code readability.

Multiple threads are used by simcore to perform various tasks, but only a single thread can usually run at a given time; this happens with both the single-threaded and the multi-threaded version. While this may seem an unnecessary restriction, it allows to dispense with complicated synchronization between threads and avoids risks of unforeseen race conditions and deadlocks. simcore threads usually perform only control tasks that do not take a lot of CPU time; a thread is suspended automatically by nPth/Pth when it has to wait for an event (such as receiving data from the network), while other threads can run in the mean time. simcore threads can also allow other threads to run at the same time, while they are performing a CPU-intensive task (such as a TLS handshake), thus taking advantage of a multi-core CPU. This is done by calling pth_unprotect and pth_protect_ explicitly, two functions which do nothing in the Pth (fully single-threaded) version. One can still enable time-sharing for the Pth version by calling pth_thread_yield_.

You can think of nPth/Pth as a magic device that turns a multi-threaded program into a single-threaded select loop at run-time (and in case of Pth, it really uses select and does not use more than one system thread).


While not really a module, this header file defines all SIMTABLE keys of all simphone network protocols. A comment for each key describes the meaning of its SIMTABLE value.


This pseudo-module includes all .h files that contain simcore resources (large read-only byte arrays, such as built-in sounds).


The only special thing about the crypto module is that its written partly in C++, as it has to interface to the crypto++ library, and makes heavy use of templates. The rest of simcore is written in pure ANSI C.

Note that the crypto++ library uses static constructors; this means that linking to simcore will insert 'invisible' function calls that are invoked by your program prior to entering its main function.


The console module contains code that uses mostly the simcore public API, as most of the module runs at UNLOCK level. It provides an example of how to call the API from C.


Implementation of simtypes. The sim_table_read and sim_table_write functions serialize simtypes (convert them to/from a byte sequence) and take an opaque pointer to a user-defined context structure which simstreamer callbacks can access.

simc, simsize, simdump and simxmls

These are in the simcore subdirectory, but they are really stand-alone programs, not simcore modules.

simc is the main testing program for simcore. It uses the console module and has batch capability, but not scripting capability, because it lacks an interface to any scripting language. You can use it to execute 'scripts' consisting of multiple console commands either at startup or at a later time (on receipt of a unix signal). The GUI has the same capability via the qtsingleapplication RPC (also available on Windows), but for security reasons, this is enabled only in the DEBUG build.

To compile simc, execute:

   cd simcore
   make simc

Most simphone functions are available with simc, so it can be abused as a poor man's Simphone in environments where the GUI cannot be used.

simxmls is a command-line program that allows you to manually merge multiple chat history files, at the same time cleaning them up by skipping duplicate, corrupted and removed messages. Edited messages remain in place (the original text as well as any and all edits of it).

module hierarchy

There is no clear hierarchy of simcore modules, as many of them are inter-dependent. However, modules can be logically grouped in the following (approximately bottom-to-top) way:

  1. basic services:
  2. network protocol: limit/proxy, proxies, nat, server, client
  3. network users: msg, audio
  4. simcore interface: api
  5. simcore users: console, simc

5. simcore Threads

When configuring simcore, you choose which threading model to use:

   ./configure --with-threads

Multi-threaded (nPth): Your eventually multi-threaded program can call the simcore API from any thread; simcore threads run simultaneously with your thread(s). Threads started by libraries used by simcore run simultaneously with simcore threads and your thread(s). This is the default mode.

   ./configure --without-threads --without-pth

Single-threaded (nPth): Your eventually multi-threaded program can call the simcore API from a single thread of your choice. That thread will be blocked while any simcore thread is running, but libraries used by simcore will still run simultaneously with simcore threads and with your thread(s), so some responsiveness is provided.

   ./configure --without-threads --with-pth

Fully single-threaded (Pth): Your eventually multi-threaded program can call simcore from a single thread of your choice. That thread will be blocked while any simcore thread is running; libraries used by simcore will block simcore threads AND the thread from which you called simcore. Avoid using this version with a GUI.

The Pth version may use less CPU time than the nPth version, but it has restrictions on the total number of socket descriptors that can be used. If you configure --without-threads but neither --with-pth nor --without-pth, the default is to use Pth if it is available from the system, else use nPth.

Each simcore thread has a single argument (passed to it on creation), which is usually a pointer to a structure. Threads that require interactivity communicate with other threads using Pth message queues. These are specially implemented by spth.c in a system-dependent way, so that writing to a message queue is non-blocking (not possible with the nPth API). The only other lock (synchronization primitive) used by simcore, apart from the main nPth lock and the logger lock, is the socket lock (one for each simsocket). The socket lock is needed because simcore lacks a sender thread, so multiple threads can send to a single socket at the "same" time, and this needs to be synchronized.

Here's an overview of all simcore threads:




started by

joined by




Monitor audio device plug and unplug.





Generate a permanent secret key.


struct keygen_arg



Try to maintain open ports through the UPnP protocol.



struct network_upnp_queue



Monitor status of connection to mainline DHT and try to connect when not connected.

sim_init_user_, sim_status_set_

sim_exit_user_, sim_status_set_



Forward TCP packets from a local incoming (proxy customer) connection to proxy control connection.





Try to make a reverse connection from proxy to one of its customers.





Find an available proxy, connect to it and receive TCP packets in a loop.





Try to perform connection reversal or NAT traversal with a simclient (connected contact)

thread_connect_, thread_server_

thread_connect_, thread_server_

struct nat_queue



Try to connect directly to a contact at a single IP address and port number.



struct nat_reverse_arg



Listen for incoming TCP connections.

sim_init_user_, sim_status_set_

sim_exit_user_, sim_status_set_



Receive incoming UDP packets.

sim_init_user_, sim_status_set_

sim_exit_user_, sim_status_set_



Handle a single incoming TCP connection and receive TCP packets in a loop.





Periodically send status probes to all of your contacts.





Send pending status probes to contacts.



struct client_probe



Send a single status probe to a contact.


struct client_probe



Try to make a single outgoing TCP connection to a contact and receive TCP packets in a loop.





Send pending chat messages to a contact.

thread_connect_, thread_server_

thread_connect_, thread_server_, sim_exit_user_

struct msg_queue



Perform an audio call or an audio test.

sim_audio_call_, audio_ring_, audio_reopen_




Join thread_audio_ when an audio device error occurs or audio test finishes.





Play a built-in sound.


thread_sound_, audio_init_, sim_exit_user_, sim_exit_

struct sound_arg


simcore threads use up to 32K stack space each, but the thread stack limit is set to 64K to provide extra safety. If you modify simcore, avoid writing recursive functions and putting large arrays on the stack.

Note that libraries used by simcore (such as Portaudio) also start background threads when necessary.

6. Notable simcore Structures

The most important simcore structure is simtype as it is heavily used internally, and is also used by the simcore public API. simtype defines a dynamically-typed (and usually dynamically-allocated) value, as described by api.html.

You can think of it as a typed pointer with a "length". For the SIMNUMBER type, it contains the value of a number (not a pointer to it), but this should be considered an implementation detail.

Another important structure associated with simtype is simwalker (used as an opaque context when iterating over all values of a SIMTABLE type). struct _simelement is the basic building block of a SIMTABLE.

simtypes can be nested recursively (used as containers of dynamically-typed values). Note that if such a container (SIMTABLE or SIMARRAY) contains a SIMPOINTER, it does NOT "own" the string that the SIMPOINTER points to, and does not free it from memory when destroyed. If it contains any other type, the simtype container owns it and frees it when destroyed. simtypes can be detached from a simtype container in order to remove them from the container AND transfer ownership from the container simtype to the caller at the same time.

The following table lists some more or less important public (simcore internal API) structures:





struct _system_qos


IPv4 type-of-service field.

part of simsocket



State of a cryptographic random number generator.

single globals random_public, random_session, random_private, random_seeded

struct _socket_crypt


Encryption keys and state of an active TCP connection. Note that a proxy connection at the client side contains two of these for a single socket.

part of simsocket

struct _ssl_master


Everything simcore has learned about a TLS handshake.

owned by simsocket



Data about any kind of an active TCP connection.

typedef pointer

struct _socket_stat


Data transfer statistics.

part of simsocket

struct _network_ip_port


An IPv4 address and port number.

part of simcontact

struct _main_param


State of mainline DHT searches.

part of simcontact



Everything simcore knows about a user in your list of contacts.

typedef pointer; accessible also as SIMTABLE or SIMARRAY

struct _contacts


General information about your list of contacts.

single global contact_list



Data about an active proxy connection. These can be of different types (#defined as PROXY_TYPE_xxx). Note that a proxy connection at the client side involves a only a simclient, while the server side uses a simclient and a simcustomer.

typedef pointer; reference-counted; linked list

struct _nat_param


IP addresses and port numbers for NAT traversal and connection reversal.

part of simclient



Data about an active TCP connection to one of your contacts.

typedef pointer; reference-counted; linked list

struct _message


Complete contents of a single chat message currently loaded into memory.

struct _messages contains a pointer to array of struct _message

struct _messages


List of all chat messages currently loaded into memory for a single contact.

part of simcontact

struct _audio_param


Current state of an audio call with one of your contacts.

part of simclient

struct _audio_status


General information about an audio call currently in progress.

single global audio_status

struct _status_self


General information about your current connection to the simphone network (status, flags, IP address, etc.)

single global simself

The next table lists some relatively important structures that are private to each of the listed modules:





struct log_buffer_entry


Complete contents of a single log line currently loaded into memory.

accessible as SIMARRAY

struct type_context


Internally used when serializing (reading or writing) simtypes.


struct _simelementalign


Internally used to define struct _simelement.

only used with sizeof



Header of each dynamically-allocated memory block; present only if compiled --with-memory-check.

typedef pointer

struct thread_info


Debug information about a currently running thread.

reference-counted; linked list; accessible also as SIMTABLE

struct system_register


Name and number of a machine (CPU) register.

const; single global system_registers

struct crypt_cipher


List of all available symmetric ciphers and their properties.

const; single global crypt_ciphers

struct file_txt_buffer


Currently loading chat history or log file.

temporary; linked list

struct file_xml_buffer


Currently loading chat history file.


struct file_context


Internally used when serializing (reading or writing) simtypes from/to files.


struct socket_context


Internally used when serializing (reading or writing) simtypes from/to network.


struct key_time


Start time of key generation operation.


struct miniupnpc


Last (current) UPnP state.

single global network_upnp

struct main_search


Data about a currently running search on the mainline DHT.

accessible also as SIMTABLE



Define default values and limits and provide descriptions of all predefined configuration parameters.

const; single globals param_number, param_string, param_strings

struct proxies


List of seemingly available proxies of a certain type (#defined as PROXY_LIST_xxx).

linked list (single per type)

struct nat


State of NAT traversal or connection reversal attempts with a connected contact.

accessible only through simclient

struct client_probe


Information about a currently running status probe to one of your non-connected contacts.

linked list

struct audio_speex


Codec state of an audio call currently in progress.

single global audio_speex

7. The simcore Naming Convention

The main purpose of this is to reveal functions that can "block" or "unblock" and possibly switch to another thread, as well to know which function can be called from where; following the rules prevents deadlocks and unsynchronized access to data.

The main nPth lock levels are LOCK (when acquired) and UNLOCK (when released). The recursive logger lock levels are LOG (when acquired) and NOLOG (when released).

name pattern

lock on entry

log on entry

can do

can call






log_xxx_, sim_xxx
while LOCK: xxx_, xxx

Blocking; public API functions (should not block for a long time).





log_xxx_, sim_xxx
while UNLOCK: sim_xxx_, sim_xxx__

Blocking (internal wrappers for other blocking functions).





log_xxx_, sim_xxx






log_xxx_, sim_xxx
while LOG: log_xxx

Logger blocking.





log_xxx_, sim_xxx, log_xxx

Logger non-blocking.





log_xxx_, sim_xxx
while LOCK: xxx_, xxx
while UNLOCK: sim_xxx__

Unblocking; may block for a very long time (these are called by background threads).





log_xxx_, sim_xxx, xxx


_xxx_, _xxx, _sim_xxx

These have file and line arguments to help with debug and error logging. Their locking behavior is defined by their name ending (as described above).

Note that some functions are defined as macros, and they usually follow the same naming convention. Macros defined in table.h are aliases for sim_xxx functions but do not follow the naming convention strictly, as their macro names do not start with sim. This is just so that simcore can use shorter function names; external users should NOT include table.h, but use the sim_xxx versions defined in simapi.h instead.

Macro names are normally defined as UPPERCASE, but macros that are callable safely as functions (for example, a macro that just calls a function with some default arguments) may have lowercase names.

8. Debugging and Testing

The LOG_xxx macros that are used throughout simcore differ from the log_xxx functions described by api.html in one respect: their arguments are evaluated only if the current log level is set below or equal to the one used by the macros. This allows the use of very slow functions as arguments of LOG_xxx macros; actually, code that has to do only with logging should not be written outside these macro calls. However, only 'pure' functions (ones that have no side effects) can be used here. Using a function or an expression that has side effects as an argument of a LOG_xxx macro is a big mistake that will cause your code to behave differently depending on the currently set log level.

One can even compile simcore with a preset log level by adding -DSIM_LOG_LEVEL=n to the Makefile in simcore; this eliminates all code that logs to a lower level at compile time.

The compilation process (as described by compiling.html) creates two executables: simphone and qsimphone. Both are identical except that qsimphone contains debug information for gdb (or lldb if you use clang instead of gcc).

You can ./configure --with-debug to obtain a "debug" build (an executable compiled without optimization). This makes debugging easier, but you can debug the "release" build, too. There are some small differences between the code of "debug" and "release" build in the GUI (but not in simcore).

To enable debug logging, type the following to the simphone console:

   set log debug

A lot of information will be logged, so this should not be enabled for "production" use. It all goes to simcore.log in your user directory; only ERRORs and FATAL errors (crashes) will be logged there, unless you have enabled debug logging.

If you also want to watch the debug log inside the simphone console, type:

   set log.console debug

To find out the reason for a crash, it is usually enough to look at the fatal error log (in simcore.log or in the crash message box). This contains a register dump and a call trace; if you have also enabled debug logging, simcore.log (but not the crash message box) will contain a "full" stack dump.

For better crash debugging, type the following to the simphone console:

   set system.crash 5

This will create a core dump on unix (if allowed by the kernel; usually you need to have set ulimit -c unlimited in advance) or invoke the system debugger on Windows or Mac OS. Note that depending on your operating system, a crash may send sensitive information to the company, which created the operating system; that is why this option is not enabled by default.

Configuration parameters can be used to modify the behavior of Simphone and turn features off or on; most of these parameters are only for testing and debugging. To see a list of available parameters, type to the simphone console:



   dump params

To see a brief description and values that can be set for a predefined configuration parameter, type:

   help set param

To see the currently running threads and open file descriptors, type:

   dump threads

This command also reports the total size of memory allocated by simcore, if you use the single-threaded version or if you have configured --with-memory-check. You can also see the total size of allocated (not freed) memory on quit in simcore.log after having enabled:

   set log.api note

Three mutually-exclusive methods are currently available for debugging memory problems, as described below. Note that all of these cause a major slow down, and electric fence also increases memory usage a lot (up to a point where you may run out of memory). Memory debugging may or may not work, depending on your platform and setup.

electric fence

This is a simple but reliable malloc replacement that can catch invalid access to dynamically-allocated memory. To enable, you need to ./configure --with-efence before compiling Simphone.

Gtk does not seem to work with electric fence, so after you have linked with electric fence on unix, you need to use windows-style by starting:

   qsimphone -style windows

Also, you need to use a larger malloc alignment, by having set an environment variable:

   export EF_ALIGNMENT=8
   export EF_ALLOW_MALLOC_0=1

simc can be used without malloc alignment for byte-exact memory checking, by setting:

   export EF_ALIGNMENT=1

Electric fence is also available on Windows, but only for memory allocated by simcore (not by other libraries or the GUI).

bounds-checking gcc

Simphone can be compiled with bounds-checking gcc, which can catch invalid access to memory (not just to dynamically-allocated memory).

You need to apply this patch to gcc and then compile it from source with commands like:

   mkdir build
   cd build
   ../gcc-4.0.4/configure --srcdir=../gcc-4.0.4 --enable-languages=c,c++ --prefix=/usr/local
   make BOOT_CFLAGS="-O2" STAGE1_CFLAGS="-O2" CFLAGS="-O2" bootstrap
   sudo make install

Afterwards, ./configure --with-bounds-check before compiling Simphone.

Bounds-checking will work only for the C code (this means simcore), and you may need to use -style windows to avoid otherwise unavoidable errors.

glibc malloc tracing

Can be used to detect memory leaks. To enable, ./configure --with-memory-check --without-threads before compiling.

Before running, set an environment variable:

   export MALLOC_TRACE=mtrace.out

After the program has finished, you can run:

   mtrace mtrace.out

to see memory blocks that have been allocated but not freed.

9. Byte Code

The simcore byte-code is used when serializing simtypes (writing to files or sending them over the network). Understanding of the byte-code is not essential to working with simcore, but is still provided here for reference.

hex codes

binary encoding (bits and bytes)

contents (decimal)


Reserved (not used).


010lllll ...

String short length = lllll (0..31) bytes; string bytes follow.


Reserved for longer short strings (not used).


100lllll ...

Table short number of key:value pairs = lllll (0..31); byte-code encoded key:value pairs follow.


Reserved for longer short tables (not used).


11000000 00000100

Empty (zero-length) array of numbers.


11000000 00000101

Empty (zero-length) array of strings.


11000000 00000110

Empty (zero-length) array of tables.


11000000 00000111

Empty (zero-length) array of arrays.


1100llll ...

Array short number of elements = llll (1..15); byte-code encoded elements follow.


Reserved for longer short arrays (not used).


11100000 llllllll ...

String length = llllllll (32..255) bytes; string bytes follow.


11100001 hhhhhhhh llllllll ...

String length = (hhhhhhhh<<8)+llllllll (256..65535) bytes; string bytes follow.


11100010 hhhhhhhh iiiiiiii llllllll ...

String length = (hhhhhhhh<<16)+(iiiiiiii<<8)+llllllll (65536..16777215) bytes; string bytes follow.


Reserved for longer strings (not used).


11101000 llllllll ...

Table number of key:value pairs = llllllll (32..255); byte-code encoded key:value pairs follow.


11101001 hhhhhhhh llllllll ...

Table number of key:value pairs = (hhhhhhhh<<8)+llllllll (256..65535); byte-code encoded key:value pairs follow.


11101010 hhhhhhhh iiiiiiii llllllll ...

Table number of key:value pairs = (hhhhhhhh<<16)+(iiiiiiii<<8)+llllllll (65536..16777215); byte-code encoded key:value pairs follow.


Reserved for longer tables (not used).


11101100 llllllll ...

Array number of elements = llllllll (16..255); byte-code encoded elements follow.


11101101 hhhhhhhh llllllll ...

Array number of elements = (hhhhhhhh<<8)+llllllll (256..65535); byte-code encoded elements follow.


11101110 hhhhhhhh iiiiiiii llllllll ...

Array number of elements = (hhhhhhhh<<16)+(iiiiiiii<<8)+llllllll (65536..16777215); byte-code encoded elements follow.


Reserved for longer arrays (not used).



Zero number; the value is 0.


11110lll ...

Positive number length = lll bytes; big-endian number bytes follow. Value is in range 1..72057594037927935


11111000 ...

Number length = 8 bytes; big-endian number bytes follow. Value is in range 72057594037927936..4611686018427387904 or -4611686018427387905..-256.


Reserved for negative number values in range -72057594037927936..-256 (not used).


11111111 nnnnnnnn

Short negative number; the value of the number is -nnnnnnnn. Value is in range -1..-255.

Note that byte-code is always encrypted when sent over the network, so it cannot be seen directly. To see the contents of network packets in a human-readable form, type:

   set log.table xtra

to the simphone console. You can then review simcore.log for the contents of these packets.

Binary files in the user directory can be encrypted or not; in the latter case, you can use simdump to show the contents of these files in a human-readable form. If the files are encrypted, you can decrypt them by typing:

   passwd none

to the simphone console. This command decrypts your private key file and enables auto-login, too.