C++ Correct multi threaded singleton initialisation

Correct double-checked locking pattern

1. Pointer must be atomic
2. Check, lock, check, construct

std::atomic<Foo*> foo { nullptr };

Foo* instance()
{
    Foo* f = foo; // single load of foo
    if (!f)
    {
        std::lock_guard<std::mutex> l(foo_lock);
        if (!foo)
        {
            foo = f = new Foo(); // assign both foo and f
        }
    }
    return f;
}

Even better, use std::unique_ptr and std::once to get automatic cleanup and less scaffolding

class Foo

{

public:

    static Foo& instance()

    {

        std::call_once(_create, [=]{

            _instance = std::make_unique<Foo>();

        });

        return *_instance;

    }

private:

    static std::unique_ptr<Foo> _instance;

    static std::once_flag       _create;

};

Or just use a function local static

Foo& Foo::instance()

{

    static Foo foo;

    return foo;

}

Linux tools

byobu - text-based window manager and terminal multiplexer, enhanced screen
htop - interactive process viewer
nethogs - process bandwidth utilisation
nload - realtime bandwidth utilisation graph
iptraf - IP network traffic monitor
    IP traffic monitor:
        top pane: TCP traffic: source addresses, packets/bytes received, link status, interface
        bottom pane: UDP / ICMP / broadcast traffic
    Statistical breakdown by TCP/UDP
        view traffic by protocol (only the common ports by default)
    LAN station monitor:
        IP traffic by MAC address

atomics & fences

Acquire
    cannot move anything up beyond an acquire

Release
    cannot move anything down beyond a release

Note
    acquire/release cannot be reordered with respect to each other
 
What does this mean?

instructions can be reordered from before an acquire to after an acquire
instructions can be reordered from after a release to before a release
acquire cannot be reordered before or after a release
release cannot be reordered before or after an acquire

std::atomic
    read = load_acquire --> read the value == acquire the value
    write = store_release --> write the value == release the value

Sequential Consistency
Transitivity / Causality
Total Store Order

using /proc/irq/#/smp_affinity to shield cpus from IRQs

IRQs (interrupt requests) are a request for service at a hardware level from the kernel.

When an IRQ arrives the kernel switches to interrupt context and loads the ISR (interrupt service routine) for the IRQ number which will process the interrupt.

IRQs have an affinity mask which defines which CPUs the ISR can run on. This is defined in /proc/irq/#/smp_affinity

The irqbalance service distributes IRQs across the processors in their associated affinity masks on a multiprocessor system. It uses /proc/irq/#/smp_affinity if it exists, or otherwise falls back to /proc/irq/default_smp_affinity/

The affinity mask in /proc/irq/#/smp_affinity and /proc/irq/default_smp_affinity is a 32-bit hex bitmask of CPUs.
    eg: ffffffff means all 32 CPUs 0-31
If there are more than 32 cores, we build up multiple comma separated 32-bit hex masks.
    eg: ffffffff,00000000 means CPUs 32-63

The irqbalance service uses the environment variable IRQBALANCE_BANNED_CPUS to tell it which CPUs it can't use for ISRs and IRQBALANCE_BANNED_INTERRUPTS to tell it which IRQs to ignore,

IRQBALANCE_BANNED_CPUS follows the same comma separated 32-bit hex format as /proc/irq/#/smp_affinity
IRQBALANCE_BANNED_INTERRUPTS is a space separated list of integer IRQs.

We can shield some CPUs from being interrupted in order to dedicate them to our own tasks.

We can also pin a network data consuming process onto a certain CPU and set the associated NIC (network interface controller) IRQs affinity mask to the same CPU so that they can share cache lines.

Notes on performance counters and profiling with PAPI

Performance counters

2 main types of profiling applications with performance counters: aggregate (direct) and statistical (indirect).

• Aggregate: Involves reading the counters before and after the execution of a region of code and recording the difference. This usage model permits explicit, highly accurate, fine-grained measurements. There are two sub-cases of aggregate counter usage: Summation of the data from multiple executions of an instrumented location, and trace generation, where the counter values are recorded for every execution of the instrumentation.
• Statistical: The PM hardware is set to generate an interrupt when a performance counter reaches a preset value. This interrupt carries with it important contextual information about the state of the processor at the time of the event. Specifically, it includes the program counter (PC), the text address at which the interrupt occurred. By populating a histogram with this data, users obtain a probabilistic distribution of PM interrupt events across the address space of the application. This kind of profiling facilitates a good high-level understanding of where and why the bottlenecks are occurring. For instance, the questions, "What code is responsible for most of the cache misses?" and "Where is the branch prediction hardware performing poorly?" can quickly be answered by generating a statistical profile.

PAPI

PAPI supports two types of events, preset and native. 

• Preset events have a symbolic name associated with them that is the same for every processor supported by PAPI. 
• Native events, on the other hand, provide a means to access every possible event on a particular platform, regardless of there being a predefined PAPI event name for it

PAPI supports measurements per-thread; that is, each measurement only contains counts generated by the thread performing the PAPI calls

int events[2] = { PAPI_L1_DCM, PAPI_FP_OPS }; // L1 data cache misses; hardware flops

long_long values[2];

PAPI_start_counters(events, 2);

// do work

PAPI_read_counters(values, 2);

Taken from a Dr Dobbs article

Voluntary/involuntary context switches

$ cat /prod/$PID/status

Voluntary context switches are when your application is blocked in a system call and the kernel decide to give it's time slice to another process.

Non voluntary context switches are when your application has used the entire timeslice the scheduler has attributed to it

Boost Compute - GPGPU programming

Pre-release version of boost compute by Kyle Lutz

http://kylelutz.github.com/compute/index.html
https://github.com/kylelutz/compute

C++ implementation of the Disruptor pattern

original: https://github.com/fsaintjacques/disruptor--

fork: https://github.com/jwakely/disruptor--

Kernel performance tuning

Kernel performance tuning is art not science
http://blog.kreyolys.com/2011/03/05/kernel-performance-tuning-is-art-not-science-part-1/

RedHat summit: RHEL Kernel Performance Optimization, Characterization and Tuning
http://www.redhat.com/promo/summit/2008/downloads/pdf/Wednesday_1015am_John_Shakshober_and_Larry_Woodman_Decoding_the_Code.pdf

TCP slow start

Slow start is a congestion control strategy used by TCP.

On startup, and after an idle period, since the session doesn't know what the congestion on the network is, the session will start with a TCP congestion window of only 2 segments. This means the session will only send 2 segments and then wait for an ACK before exponentially increasing the congestion window.

With today's modern 10Gb networks, congestion is rarely an issue; and in a low-latency environment practitioners will often say "congestion be damned, send as much as we can and deal with the issues later!"

We can override the default setting on a route by route basis in linux by setting the initial congestion window size:


$ ip route show
10.80.32.0/22 dev eth0  proto kernel  scope link  src 10.80.33.247 
169.254.0.0/16 dev eth0  scope link 
127.0.0.0/8 dev lo  scope link 
default via 10.80.32.1 dev eth0 

Now we can change the congestion window size on the default route as follows:

$ sudo ip route change default via 10.80.32.1 dev eth0 proto static initcwnd 10

NUMA local PCI-Express interfaces

With Sandybridge PCI-Express slots are now local to a particular socket and therefore NUMA node.

If you're connecting to a specific NIC then it makes sense to make sure you're on the same NUMA node as the NIC is, thereby preventing having to shuttle data over the QPI between the kernel buffers and your user-space buffers.

Details on what NUMA nodes a particular NIC is on can be found as follows:

/sys/class/net/eth0/device $ cat numa_node
0

/sys/class/net/eth0/device $ cat local_cpus
00005555

/sys/class/net/eth0/device $ cat local_cpulist
0,2,4,6,8,10,12,14

PAPI: Performance API

PAPI aims to provide the tool designer and application engineer with a consistent interface and methodology for use of the performance counter hardware found in most major microprocessors. PAPI enables software engineers to see, in near real time, the relation between software performance and processor events.

http://icl.cs.utk.edu/papi/

User Guide: http://icl.cs.utk.edu/projects/papi/files/documentation/PAPI_USER_GUIDE_23.htm

boost lockfree

Currently being reviewed for release - here is the documentation up for review:

http://tim.klingt.org/boost_lockfree/

Using strace & top to debug a multi threaded app

Displays all the system calls being made in each thread

    strace -f ./AppName

Displays all threads for the pid in question.

    top -p pid -H

Use f,j in top's interactive mode to show which cpus each thread is running on

You can get the same information from the /proc/pid filesystem
This displays each thread and the last cpu it ran on

    /proc/pid/task $ for i in `ls -1`; do cat $i/stat | awk '{print $1 " is on " $(NF - 5)}'; done

cset set/proc - finer control of cpusets

http://code.google.com/p/cpuset/

Set

Create, adjust, rename, move and destroy cpusets

Commands

Create a cpuset, using cpus 1-3, use NUMA node 1 and call it "my_cpuset1"

    $ cset set --cpu=1-3 --mem=1 --set=my_cpuset1

Change "my_cpuset1" to only use cpus 1 and 3

    $ cset set --cpu=1,3 --mem=1 --set=my_cpuset1

Destroy a cpuset

    $ cset set --destroy --set=my_cpuset1

Rename an existing cpuset

    $ cset set --set=my_cpuset1 --newname=your_cpuset1

Create a hierarchical cpuset

    $ cset set --cpu=3 --mem=1 --set=my_cpuset1/my_subset1

List existing cpusets (depth of level 1)

    $ cset set --list

List existing cpuset and its children

    $ cset set --list --set=my_cpuset1

List all existing cpusets

    $ cset set --list --recurse

Proc

Manage threads and processes

Commands

List tasks running in a cpuset

    $ cset proc --list --set=my_cpuset1 --verbose

Execute a task in a cpuset

    $ cset proc --set=my_cpuset1 --exec myApp -- --arg1 --arg2

Moving a task

    $ cset proc --toset=my_cpuset1 --move --pid 1234
    $ cset proc --toset=my_cpuset1 --move --pid 1234,1236
    $ cset proc --toset=my_cpuset1 --move --pid 1238-1340

Moving a task and all its siblings

    $ cset proc --move --toset=my_cpuset1 --pid 1234 --threads

Move all tasks from one cpuset to another

    $ cset proc --move --fromset=my_cpuset1 --toset=system

Move unpinned kernel threads into a cpuset

    $ cset proc --kthread --fromset=root --toset=system

Forcibly move kernel threads (including those that are pinned to a specific cpu) into a cpuset (note: this may have dire consequences for the system - make sure you know what you're doing)

    $ cset proc --kthread --fromset=root --toset=system --force

Hierarchy

Using hierarchical cpusets to create prioritised groupings

Example
1. Create a system cpuset with 1 cpu (0)

2. Create a prio_low cpuset with 1 cpu (1)

3. Create a prio_met cpuset with 2 cpus (1-2)

4. Create a prio_high cpuset with 3 cpus (1-3)

5. Create a prio_all cpuset with all 4 cpus (0-3) (note this the same as root;  it is considered good practice to keep a separation from root)

To achieve the above you create prio_all, and then create subset prio_high under prio_all, etc

    $ cset set --cpu=0 --set=system

    $ cset set --cpu=0-3 --set=prio_all

    $ cset set --cpu=1-3 --set=/prio_all/prio_high

    $ cset set --cpu=1-2 --set=/prio_all/prio_high/prio_med

    $ cset set --cpu=1 --set=/prio_all/prio_high/prio_med/prio_low

cset shield - easily configure cpusets

http://code.google.com/p/cpuset/

shield

Basic concept - 3 cpusets
root: present in all configurations and contains all cpus (unshielded)
system: contains cpus used for system tasks - the ones which need to run but aren't "important" (unshielded)
user: contains cpus used for "important" tasks - the ones we want to run in "realtime" mode (shielded)

The shield command manages these 3 cpusets.

During setup it moves all movable tasks into the unshielded cpuset (system) and during teardown it moves all movable tasks into the root cpuset.
After setup, the subcommand lets you move tasks into the shield (user) cpuset, and additionally, to move special tasks (kernel threads) from root to system.

Commands:
Create a shield (Example: 4-core non-NUMA machine: we want to dedicate 3 cores to the shield, and leave 1 core for unimportant tasks; since it is non-NUMA we don't need to specify any memory node parameters; we leave the kernel threads running in the root cpuset)

    $ cset shield --cpu 1-3

Some kernel threads (those which aren't bound to specific cpus) can be moved into the system cpuset. (In general it is not a good idea to move kernel threads which have been bound to a specific cpu)

    $ cset shield --kthread on

List what's running in the shield (user) or unshield (system) (-v for verbose, list the process names) (2nd -v to display more than 80 characters)

    $ cset shield --shield -v
    $ cset shield --unshield -v -v

Stop the shield (teardown)

    $ cset shield --reset

Execute a process in the shield (commands following '--' are passed to the command to be executed, not to cset)

    $ cset shield --exec mycommand -- -arg1 -arg2

Move a running process into the shield (move multiple processes by passing a comma separated list, or ranges (any process in the range will be moved, even if there are gaps))

    $ cset shield --shield --pid 1234
    $ cset shield --shield --pid 1234,1236
    $ cset shield --shield --pid 1234,1237,1238-1240

Dr Dobb's Go Parallel

A multicore application performance blog hosted on Dr Dobbs:

http://www.drdobbs.com/go-parallel

Folly benchmarking

Facebook recently open sourced an internal C++ library called Folly.

They have an interesting benchmarking library.

Check it out here: https://github.com/facebook/folly/blob/master/folly/docs/Benchmark.md

TTAS spin lock

http://askldjd.wordpress.com/2010/03/31/test-and-test-and-set-lock/

Kenel bypass & zero-copy

http://ttthebear.blogspot.com/2008/07/linux-kernel-bypass-and-performance.html

kernel bypass
http://www.networkworld.com/news/tech/2005/013105techupdate.html

zero-copy
http://lwn.net/2001/0419/kernel.php3http://www.linuxjournal.com/article/6345

Interrupt Coalescence
It is possible to reduce mean latency, but will most likely increase the min latency

RDMA
http://www.networkworld.com/news/tech/2003/0324tech.html
http://www.networkworld.com/newsletters/lans/2002/01556276.html

Request completions might be processed either entirely in user space (by polling a user-level completion queue) or through the kernel in cases where the application wishes to sleep until a completion occurs.

A Fast Read/Write Process to Reduce RDMA Communication Latencyhttp://www.people.vcu.edu/~xhe2/publications/Conferences/FRRWP_NAS06.pdf

--- implementation of user-space waiting on rdma ---
create a condition variable based on a futex
have rdma completion handler wake the condition variable