HPX Frequently Asked Questions

HPX-5 Home / Applications / Documentation / Frequently Asked Questions /
GIT Repository /Media / Presentations / Publications /
Users Guide / Vision / CREST Home
 The detailed documentation for HPX-5 is: here. Please feel free to contact the Mailing List if you need additional help. 

Debugging

Can I pause execution on an error?

Why yes, you can. Make sure that you have configured with –enable-debug and then you can use the –hpx-dbg-waitonabort runtime option to have the system print out the process id to attach to when an error is detected. You can also use –hpx-dbg-waitonsegv though this can be unreliable.

How do I know what the local virtual address is for a global address?

In SMP execution (you’re debugging in SMP, right?) you can simply cast the hpxaddrt to a pointer type to do the address translation. With PGAS or AGAS this is more complicated.

In PGAS, you may use the libhpx internal function pgasgpatolva. (gdb) p *p $3 = {ustack = 0x792c90, next = 0x0, src = 0, size = 4, state = 1, offset = 0, action = 64, caction = 0, target = 1069687200, ctarget = 0, pid = 0, credit = 0, buffer = 0x792c80 “\001”} (gdb) p pgasgpato_lva (p→target) $4 = (void *) 0x13fc221a0 In AGAS1) .

You can use the bttlookup function.
(gdb) p *p
$5 = {ustack = 0x783c90, next = 0x0, src = 0, size = 4, state = 1, offset = 0, action = 64, c_action = 0, target = 21990232691008, c_target = 0, pid = 0, credit = 0, buffer = 0x783c80 “”}
(gdb) p btt_lookup(((agas_t*)here->gas)->btt, p->target)
$6 = (void *) 0x7fffed021140

If you commonly need to perform this translation in AGAS you could create your own function to do this and use that instead.

What configuration should I use for debugging?

When debugging you want to make sure that debug symbols are available, and that you have access to the HPX-5 debug options. Logging can also be helpful.

A typical configuration line for an SMP build would look something like the following:
./configure –enable-debug –enable-instrumentation CFLAGS=”-O0 -g”

 

How do I know what function an action id corresponds to?
The global action table, accessible through the here pointer, tracks this mapping.
$ gdb ./hello
[…] (gdb) break _hello_action
Breakpoint 1 at 0x400a26: file ../../../../../hpx/examples/hello.c, line 18.
(gdb) r –hpx-threads=1
[…] Breakpoint 1, _hello_action (args=0x0, size=0) at ../../../../../hpx/examples/hello.c:18
18 printf(“Hello World from %u.\n”, hpx_get_my_rank());
(gdb) bt
#0 _hello_action (args=0x0, size=0) at ../../../../../hpx/examples/hello.c:18
#1 0x00007ffff795eb0d in action_execute (p=0x7ffff6494040) at ../../../../../hpx/libhpx/action.c:321
#2 0x00007ffff7992048 in _execute_thread (p=0x7ffff6494040) at ../../../../../../hpx/libhpx/scheduler/worker.c:153
#3 0x0000000000000000 in ?? ()
(gdb) up 2
#2 0x00007ffff7992048 in _execute_thread (p=0x7ffff6494040) at ../../../../../../hpx/libhpx/scheduler/worker.c:153
153 int e = action_execute(p);
(gdb) p *p
$1 = {ustack = 0x7ffff64a3000, next = 0x0, src = 0, size = 0, state = 1, offset = 0, action = 48, c_action = 0, target = 140737325133888, c_target = 0, pid = 0, credit = 0, buffer = 0x7ffff6494080 “\377\377”}
(gdb) p actions[p->action] $2 = {handler = 0x400a16 <_hello_action>, id = 0x7fffffffda7e, key = 0x400bf8 “../../../../../hpx/examples/hello.c:_hello_action”, type = HPX_DEFAULT, attr = 1, cif = 0x0}
(gdb) quit
How do I debug my application?

First things first, read the documentation on debugging in the user guide.

Debugging distributed applications is hard. Debugging HPX-5 applications is harder. This is generally due to two factors.

  • Large distributed applications have a large amount of runtime complexity.
  • Debugging tools are thread centric.

HPX-5 applications are often event driven and have large numbers of user-level threads, parcels, and conditions expressed as LCOs. When you run in a traditional thread-centric debugger, you will have to infer the execution of your application by inspecting the libhpx runtime state. This requires at least a bit of understanding of the libhpx source code.

Your first task is to find the minimal execution in terms of number of localities and number of threads that your problem manifests with. If you are lucky, you will be able to debug an SMP single-threaded execution.

  • Each locality has a global here pointer that points to a locality_t, which tracks the overall state of the locality. In addition, each thread has a global self pointer that keeps the state of the worker thread.
  • Breakpoints and watchpoints are vital.
  • Do not be afraid to do printf debuggingit may permute your execution but if your bug still manifests you will get application-level tracing information.
  • Use logging where necessary. When you compile with –enable-debug you have access to libhpx’s logging interface, including –hpx-log-at to restrict the localities that log, and –hpx-log-level= to log certain subsystems. Logging is ad-hoc, for more regularized information consider the tracing interface.

Contact Us on the HPX user’s mailing list. We often (think we) have helpful advice.

General Information

I would like to add a publication or presentation using HPX-5 to your website. How do I sent it?

If you have a paper for publication / presentation using HPX-5, please email the HPX-5 Developer’s mailing list:- “hpx-devel[at]crest[dot]iu[dot]edu” with a Subject: line that begins with the word “Publication”.

Are there other HPX implementations?

Yes! There is a C++ runtime system available centered on the ParalleX execution model from Louisiana State University (LSU).

These are two different implementations of the ParalleX execution model.

What if my contribution was not accepted into the HPX-5 code base?

If your contribution was not accepted into the main HPX-5 code base, there are likely to be good reasons for it (perhaps technical, perhaps due to licensing restrictions, etc.).

If you wrote a standalone component, you can still distribute this component independent of the main HPX-5 distribution. As such, you can distribute your component even if it is closed source (e.g., distributed as binary-only) via any mechanism you choose, such as on a web site, FTP site, etc.

How should I report a bug?

First check that this does not exist as a known issue in FAQ or the mailing list archives. If you can’t find your problem mentioned anywhere, create a new issue with details as to how it can be replicated.

How are 3rd party contributions handled?

HPX-5 is a free and open source code base. We require an original signed contribution agreement from the 3rd party before accepting any code. We need to have an established intellectual property pedigree of the code in HPX-5. The agreement asserts that the contributor has the right to donate the code and allows the HPX-5 project to distribute it under HPX-5 project’s license. We must have a copy of this agreement on file before we can accept code into the HPX-5 code base. The agreement document can be downloaded from here dashmmuserguide-1.2.0. Please send the signed copy to Jayashree Ajay Candadai [jayaajayindiana.edu].

How can I contribute to HPX-5?

There are many ways to contribute to HPX-5:

  • Read through the HPX Developer’s policy.
  • Subscribe to the Mailing Lists and become an active member in the discussions.
  • Obtain a source code checkout of HPX-5 code base and start going through the code.
  • Write your code and contribute to the main code base.
  • Contribute bug fixes and feature enhancements to the main code base.
Can I redistribute HPX-5?

Absolutely. Please read the HPX-5 license (the BSD license). It contains extremely liberal provisions for redistribution.

How is this software licensed?

The HPX-5 code base is licensed under the BSD license. Having said that, although we are an open source project, we recognize that not everyone can provide free, open source software. We allow and encourage 3rd parties to write and distribute their own components perhaps with a different license.

Can I contribute to HPX-5?

Yes. HPX is based on Open Source model. It would be our pleasure to bring together smart researchers and developers to work on a common product. Please see our HPX Developer’s policy for more information.

What are the components HPX-5 provides?

HPX-5 supports large-scale irregular applications with localities, Active Global Address Space (AGAS), ParalleX Processes, Complexes (ParalleX Threads and Thread Management), Parcel Transport and Parcel Management and Local Control Objects (LCOs).

Where can I learn about HPX-5?

Please see our documentation, tutorials, and Publications sections. If you have any additional questions, you can contact us by using our mailing list.

What is HPX-5?

HPX-5 is the High Performance ParalleX runtime library from CREST at Indiana University. HPX-5 is a reduction to practice of the revolutionary ParalleX execution model, which establishes roles and responsibilities between layers in an exascale system and supports dynamic and adaptive resource management and scheduling. It is organized around a cooperative lightweight thread scheduler, a global address space, an active-message parcel transport, and a group of globally addressable local synchronization object classes. Internally, the infrastructure is built on scalable concurrent data structures to minimize shared-memory synchronization overhead. The global address space and parcel transport are based on the innovative Photon network transport library, which supports low-level access to network hardware and provides RDMA with remote completion events for low overhead signaling. An alternative ISend/IRecv network layer is included for portability, along with a reference MPI implementation. HPX-5 is compatible with Linux running on Intel x86, MAC OSX and Xeon Phi processors and various ARM core platforms (including both ARMv7 and ARMv8/Aarch64).

Getting Started with HPX-5

How do I use the HPX-5 C++ interface?

Experimental support for HPX C++ bindings (HPX++) is available and can be enabled with the configure option –enable-hpx++. Users of the interface can use the API using the hpx/hpx++.h header. The API provides classes for creating and registering actions, global pointers, and LCOs. More information can be found in the User Guide.

I’m still having problems / my problem is not listed here. What do I do?

If you have a problem, then:

  1. Please go through the user’s guide and developer’s guide in the documentation to see if your problem is listed there.
  2. Read through the mailing list archives to see if the problem was already discussed.
  3. Check in https://gitlab.crest.iu.edu/extreme/hpx/issues to see if the problem is already known and is in open status.

If the problem was not listed anywhere, then send a mail to hpx-users mailing list for support on problems building, installing, and using HPX-5. You can subscribe to the hpx-users mailing list.

What are the generalities of testing?

A test suite harness of HPX-5 is a software component that executes all (or part of) the defined test cases in unit and perf folders, analyzes their outcomes, and report these outcomes appropriately.

PASS: A test is said to pass when it can determine that the condition or behaviour it means to verify holds, and is said to fail (FAIL) when it can determine that such condition of behaviour does not hold.

Sometimes, tests are skipped (SKIP), if the tests rely on non-portable tools or prerequisites, or simply make no sense on a given system. These tests are not run, or their result is anyway ignored for what concerns the count of failures an successes. Skips are usually explicitly reported though, so that the user will be aware that not all of the testsuite has really run.

It’s not uncommon, especially during early development stages, that some tests fail for known reasons, and that the developer doesn’t want to tackle these failures immediately (this is especially true when the failing tests deal with corner cases). In this situation, the better policy is to declare that each of those failures is an expected failure (or XFAIL). In case a test that is expected to fail ends up passing instead, many testing environments will flag the result as a special kind of failure called unexpected pass (or XPASS).

An hard error (ERROR) happens when e.g., the set-up of a test case scenario fails, or when some other unexpected or highly undesirable condition is encountered (for example, the program under test experiences a segmentation fault).

How do I build the applications in hpx-apps folder?
Run the following commands to run applications in hpx-apps folder
shell$ export PKG_CONFIG_PATH=$HOME/repos/hpx-install/lib/pkgconfig:$PKG_CONFIG_PATH
shell$ export LD_LIBRARY_PATH=$HOME/repos/hpx-install/lib:$LD_LIBRARY_PATH
shell$ ./bootstrap
shell$ ./configure
shell$ make

and run the application.

When compiling HPX-5 what do I do if I get errors from Id about libdarshan-mpi-io.a saying could not read symbols: Bad value?
If you find the error:
/usr/bin/ld: /usr/common/usg/darshan/2.3.0/lib/libdarshan-mpi-io.a(darshan-mpi-io.o): relocation R_X86_64_32 against `.rodata’ can not be used when making a shared object; recompile with -fPIC
/usr/common/usg/darshan/2.3.0/lib/libdarshan-mpi-io.a: could not read symbols: Bad value
collect2: error: ld returned 1 exit status
Then:
module unload darshan

 

How do I compile HPX-5 application?

This is detailed in the Users Guide in documentation.

What are the known limitations to build on ARM boards?

These are the current known limitations to build on ARM:

  1. Since, ARMv7 boards have limited memory, some of the apps and examples might not work at higher scale due to memory limitation.
  2. AGAS does NOT work with ARM.
  3. Aarch64 also know as Armv8 is supported with HPX has the same above limitations.
What is the main tree layout of the HPX-5 source tree?

One useful source of information about the HPX source base is the HPX doxygen . The following is a brief introduction to code layout:

hpx/config – M4 scripts supporting the top-level configure script.

hpx/contrib – consists of 3rd party dependencies that are distributed with HPX-5.

  • hpx/contrib/hwloc-1.11.4 – hwloc library helps to obtain the hierarchical map of key computing elements, such as: NUMA memory nodes, shared caches, processor sockets, processor cores, processing units (logical processors or “threads”) and even I/O devices. It gathers various attributes such as cache and memory information. Additionally it also assembles the topology information of the entire cluster at once.
  • hpx/contrib/jemalloc – It is a general purpose malloc implementation that emphasizes fragmentation avoidance and scalable concurrent support.
  • hpx/contrib/libcuckoo – provides a high-performance, compact hash table that allows multiple concurrent reader and writer threads.
  • hpx/contrib/photon – Photon is a network backend for HPX-5. Photon can take advantage of RDMA over IB resulting in better performance.
  • hpx/contrib/valgrind – valgrind is a memory error detector library. It can detect accessing memory one shouldn’t, using undefined values, incorrect freeing of memory, overlapping src and dest pointers in memcpy and related functions, passing a negative value to the size parameter of a memory allocation function and memory leaks.
  • hpx/contrib/libffi – is a foreign function interface library. It provides a C programming language interface for calling naively compiled functions given information about the target function at run time instead of compile time. It also implements opposite functionality: lib can produce a pointer to a function that can accept and decode any combination of arguments defined at run time. libffi is most ofter used as bridging technology between compiled and interpreted language implementations. It can also be used to implement plugin’s, where the plugin’s function signature are not known at the time of creating the host application. hpx/contrib/libffi-mic – When compiling on Xeon-Phi HPX-5 is built with libffi-mic interface library automatically.

hpx/examples This directory contains some simple examples of how to use the HPX-5. cxx folder has c++ versions of some of the same examples.

hpx/include This directory contains public header files used for HPX-5. The three main subdirectories of this directory are:

hpx/include/hpx This directory consist all of the HPX-5 specific header files. This folder has types and functions defined for HPX-5 actions, global addresses, attribute definitions, built-in compiler functions, futures, functions for allocating and using memory in the HPX-5 global address space, HPX-5 LCO interface, HPX-5 parcel interface, HPX-5 parallel loop interface, HPX-5 process creation, HPX-5 remote procedure call interface, HPX-5 system interface, HPX-5 thread interface, HPX-5 high-resolution timer interface and HPX-5 topology interface. cxx folder has the c++ bindings.

hpx/include/libhpx This directory has types and functions defined for libhpx source directory. It deals with header files for bootstrapper, configuring HPX-5 at run-time, locality, interface to the parcel network subsystem, gas, internal interface to the scheduler and HPX scheduler statistics, system, transport, worker threads, and util.

hpx/include/libapex This directory consists of types and functions defined for libapex source directory.

hpx/libhpx – Contains source code for HPX implementation – HPX-5 actions, bootstrapping, HPX-5 calls, functions for debugging HPX-5, GAS (Global Address Space – smp, pgas), network layer, scheduler code (LCOs – collectives, cvar, future, gencount, semaphores, thread and worker code), and system related code.

hpx/libapex – This consist of code for support for APEX library for instrumentation purposes.

hpx/scripts – Consists of script for installing the latest autotools and valgrind warning suppression script for photon.

hpx/tests – This consist of unit and performance testsuite for HPX-5.

hpx/tutorial – This consist of tutorial examples for HPX-5.

What are the additional optional configure options?
Some additional optional features that HPX supports are the following:
enable-parallel-config : Speed-up configure [off] enable-hpx++ : Enable C++ bindings [off] enable-photon : Build with Photon networking [off] enable-mpi : Build with MPI networking and mpirun bootstrap [off] enable-pmi : Build with PMI bootstrap [off] enable-jemalloc : Use jemalloc for memory allocation [on] enable-tbbmalloc : Use tbbmalloc for memory allocation (requires C++11) [off] enable-dlmalloc : Use Doug Leas malloc for GAS allocation [off] enable-hugetlbfs : Enable support for explicit huge pages [off] enable-agas : Enable AGAS (requires C++11) [off] enable-docs : Build doxygen documentation [off] enable-debug : Enable debug checks (hpx-dbg-{waitat,waitonabort,waitonsegv} and logging (hpx-log-{at,level}) [off] enable-instrumentation : Enable instrumentation (can affect performance) [off] enable-testsuite : Build the testsuite [on] enable-lengthy-tests : Enable long running tests in the test suite [off]
How do I set the Photon IB device and backend for HPX-5?

The Photon default is to use the first detected IB device and active port.

This behavior can be overridden with the following environment variable:
shell$ export HPX_PHOTON_IBDEV=”mlx4_0″

This string also acts as a device filter. For example, device names can be retrieved with ibvdevinfo on systems with IB Verbs support. If HPXPHOTON_IBDEV is set to be blank, Photon will try to automatically select the right device.

These parameters can also be set at run time using the command line option –hpx-photon-ibdev

The Photon default backend is also set to verbs.

On Cray system with a uGNI supported interconnect, set the following environment variable:
shell$ export HPX_PHOTON_BACKEND=ugni
OR, use –hpx-photon-backend=ugni as a command line flag.
How do I run the HPX-5 application?
HPX programs can be run using any of the MPI pr PMI launchers such as mpirun/mpiexec. e.g. to run the pingpong example
shell$ mpirun -np 2 examples/pingpong 10 -m -v

For more detailed information refer to User’s Guide in the documentation.

How do I know what the runtime options with HPX-5 are?

HPX-5 provides runtime options that can be specified on the command line or in the environment. The list of options can be obtained from any HPX program by adding –hpx-help option to the job launcher.

For detailed information on options refer to User’s Guide in the documentation.

How do I complete the build and install HPX-5?
To build and install HPX-5 use the commands:
shell$ make
shell$ make install
How do I configure HPX-5 with the test-suite enabled?
To build and run the unit and performance testsuite enabled:
shell$ ./configure TESTS_CMD=”mpirun -np 2 –map-by node:PE=16″ –enable-mpi
shell$ make check

TESTS_CMD is used to specify the application launch command on various systems. This has to be set appropriately for your system.

How do I configure HPX-5 with Photon?

The Photon transport is included in HPX within the contrib directory.

To configure HPX with Photon use:
shell$ ./configure –prefix=/path/to/install –enable-mpi –with-mpi=mpich CC=cc

On stampede pass PHOTONCARGS to ofed path: shell$ ./configure enable-mpi CC=mpicc enable-photon PHOTONCARGS=“LDFLAGS=-L/opt/ofed/lib64 CPPFLAGS=-I/opt/ofed/include”

HPX-5 can be configured with Photon on CRAY System using:
shell$ ./configure –prefix=/path/to/install/ CC=cc –enable-photon –enable-pmi –enable-mpi PHOTON_CARGS=”–enable-ugni –disable-libfabric”
How do I configure HPX-5 with MPI?

By default, HPX-5 will be configured in SMP modewithout a high-speed network. To configure with MPI, use the –enable-mpi option. When MPI is enabled, the configuration will search for the appropriate way to include and link to MPI.

HPX5 will try and see if hpx.h and libmpi.so are available with no additional flags (e.g., for CC=mpicc or on Cray CC=cc).

HPX5 will test for an hpx.h and -lmpi in the current C_INCLUDE_PATH and {LD_}LIBARARY_PATH respectively.

HPX5 will look for an ompi pkg-config package.

To configure HPX-5 with MPI use the following command:
shell$ ./configure –prefix=/path/to/install –enable-mpi
On CRAY System use If you have MPICH installed, you need to specify:
shell$ ./configure –prefix=/path/to/install –enable-mpi –with-mpi=mpich CC=cc

This will instruct ‘configure’ to look for ‘mpich.pc’.

How do I configure HPX-5 on CRAY Systems?
The HPX-5 can be configured on CRAY Systems using:
shell$ module unload PrgEnv-cray
shell$ module unload PrgEnv-intel/5.2.40
shell$ module load PrgEnv-gnu
shell$ setenv CRAYPE_LINK_TYPE dynamic
shell$ ./configure –prefix=/path/to/install/ CC=cc
You might need hugepages module depending on your module and environment configuration E.g.,
shell$ module load craype-hugepages8M
shell$ ./configure –enable-pmi –enable-hugetlbfs –enable-photon CC=cc
How do I set explicit path to Open MPI during configure?

If you want to specify which MPI should be used or the build system cannot find the .pc file you may need to use the –with-mpi=pkg option. The pkg parameter can be either the prefix for the .pc file (ompi, mpich, etc.) or a complete path to the .pc file.

What network transports does HPX-5 support?

HPX-5 can be built with two network transport options: MPI and Photon. HPX-5 can be built with one, both, or none of the network transports depending on application needs. Each transport is runtime configurable. MPI and PMI are currently used as job launchers and bootstrap mechanisms for HPX-5. Note that if you are building with Photon, the libraries for the given network interconnect you are targeting need to be present on the build system. The two supported interconnects are InfiniBand (libibverbs and librdmacm) and Cray’s GEMINI and ARIES via uGNI (libugni). You may build with IBV and/or uGNI support on a workstation where the development packages are installed, but launching an HPX application with Photon requires that the actual network devices be present so that the library can initialize.

HPX5 2.0.0 includes experimental support for sockets and psm networking through the included libfabrics package, which may have additional dependencies on your system. Libfabric is enabled by default, but if photon detects a “native” backend, like verbs or ugni, it will use that by default only falling back to libfabric/sockets if neither is detected.

If you build with Photon and/or MPI on a system without networking, you may still use the SMP transport to run applications that are not distributed.

How do I set path to install HPX-5?

HPX-5 by default installs to /usr/local directory.

If you would like to set a different path, it can be specified at configuration time using the command:
shell$ ./configure –prefix=$path/to/install/directory
shell$ make
shell$ make install
How can I build HPX without network transport in SMP Mode?

Use the following commands for configuring HPX and building it.

This is the simplest configure option.
shell$ ./configure
shell$ make

In next sections we will get you started with various configure options.

How to set the configure environment dependencies

The HPX configuration uses pkg-config to find dependencies. For custom builds using non-default configuration options, you need to

  • point to the appropriate pkg-config (.pc) files or
  • set the appropriate LIBS and CFLAGS environment variables.
For further details see:
shell$./configure –help
I updated the autotools using the script but still get libtool version error. What do I do?
If autotools were installed correctly and path was set right, and still you get the following error:

shell$ libtool –version
libtool (GNU libtool) 2.4.6
Written by Gordon Matzigkeit, 1996

Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

and/or
shell$ ./bootstrap
configure.ac:39: error: Libtool version 2.4.6 or higher is required
config/libtool.m4:46: LT_PREREQ is expanded from…
configure.ac:39: the top level
autom4te: /global/homes/j/jayaajay/autotools/bin/m4 failed with exit status: 63
aclocal: error: echo failed with exit status: 63
autoreconf: aclocal failed with exit status: 63
You should run git clean using the command:
shell$ git clean -xffd
Removing aclocal.m4
Removing autom4te.cache/
Removing config/libtool.m4
Removing config/ltmain.sh
Removing config/ltoptions.m4
Removing config/ltsugar.m4
Removing config/ltversion.m4
Removing config/lt~obsolete.m4
shell$ ./bootstrap
What does this bootstrap error mean?
Examples of errors due to old version of autotools:
shell$ ./bootstrap Downloading and extracting contrib tarballs…photon…libcuckoo…jemalloc…libffi…DONE libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, `config’. libtoolize: copying file `config/ltmain.sh’ libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `config’. libtoolize: copying file `config/libtool.m4′ libtoolize: copying file `config/ltoptions.m4′ libtoolize: copying file `config/ltsugar.m4′ libtoolize: copying file `config/ltversion.m4′ libtoolize: copying file `config/lt~obsolete.m4′ configure.ac:39: error: Libtool version 2.4.6 or higher is required config/libtool.m4:46: LT_PREREQ is expanded from… configure.ac:39: the top level autom4te: /usr/bin/m4 failed with exit status: 63 aclocal: error: echo failed with exit status: 63 autoreconf: aclocal failed with exit status: 63
or:
config/hpx_contrib_jemalloc.m4:4: error: m4_defn: undefined macro: _m4_divert_diversion config/hpx_contrib_jemalloc.m4:4: the top level autom4te: /usr/bin/m4 failed with exit status: 1 aclocal: autom4te failed with exit status: 1 autoreconf: aclocal failed with exit status: 1

This means your auto tools are outdated. Setup the auto tools as explained in the previous section and update your PATH environment variable to use the new auto tools.

How do I bootstrap HPX-5?

The latest autotools can be installed by using setup_autotools.sh in scripts folder. The script takes an path to install autotools to. After installing autotools, be sure to update your PATH variable. Only after installing the latest autotools should the user run ./bootstrap.

Bootstrapping

NOTE: ./bootstrap should not be used with release tarballs.

HPX provides a bootstrap script in its build. In the HPX directory run the bootstrap script using:
shell$ ./bootstrap
Downloading and extracting contrib tarballs…photon…libcuckoo…jemalloc…libffi…DONE
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, ‘config’.
libtoolize: copying file ‘config/ltmain.sh’
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, ‘config’.
libtoolize: copying file ‘config/libtool.m4’
libtoolize: copying file ‘config/ltoptions.m4’
libtoolize: copying file ‘config/ltsugar.m4’
libtoolize: copying file ‘config/ltversion.m4’
libtoolize: copying file ‘config/lt~obsolete.m4’
configure.ac:53: installing ‘config/ar-lib’
configure.ac:32: installing ‘config/compile’
configure.ac:33: installing ‘config/config.guess’
configure.ac:33: installing ‘config/config.sub’
configure.ac:46: installing ‘config/install-sh’
configure.ac:51: installing ‘config/missing’
contrib/dlmalloc/Makefile.am: installing ‘config/depcomp’
parallel-tests: installing ‘config/test-driver’

Bootstrap is a bash script that generate the initialization required to create a configure script when using GNU autotools. This calls the autoreconf.

Building without MPI enables the SMP bootstrapper. This allows you to run HPX on a single node. However, if you launch HPX with np > 1 then either MPI / Photon transport should be enabled. This is detailed in the answers to subsequent questions.

What are the software requirements for building HPX-5?

Compiling HPX requires that you have several software packages installed. The table below lists the required packages. The package column is the usual name for the software package that HPX depends on. The Version column provides “known to work” versions of the package. The Notes column describes how HPX uses the package and provides other details.

Compilers
C99 compatible compiler gcc Tested with 4.6.4, 4.8.4, 4.9.2, 5.1.0, 5.2.0 (Some features, specifically the use of the external tbbmalloc allocator and AGAS, are not supported under 4.6.4)
clang Tested with 3.6, 3.5 and earlier are known to be broken
icc Tested with 15.0.1, 16.0
Build Systems
GNU Make 3.81+ Makefile/build processor GNU Autoconf 2.6.9 Configuration script builder (Only required if building from the git version of HPX5)
GNU Automake 1.15 Aclocal macro generator (Only required if building from the git version of HPX5) GNU M4 1.4.17 Macro processor for configuration (Only required if building from the git version of HPX5)
Libtool 2.4.6 Shared library manager (only required for developer builds)
Pkg-config
Library Dependencies
MPI (optional) 1.6.3+ Tested with 1.6.3, 1.6.5, 1.8.1, 1.8.4, 1.10.0 MPICH 3.0.4, mvapich2/2.0b (Stampede)
Doxygen (optional) 1.8.7+ Required to build the documentation
Other included dependencies Photon, Jemalloc(If available, tbbmalloc may be used instead of jemalloc), Hwloc, Uthash, libffi

NOTES:

  1. HPX can build and run successfully without any network backend, but at present, MPI or Photon is required for networking by HPX-5.
  2. If you want to make changes to the configure scripts, you will need GNU autoconf (2.69), M4(1.4.17), automake(1.15), libtool-2.4.6.
How do I get a copy of the most recent source code?

Source code access is detailed here.

How can I download the most recent release of HPX-5?

The most recent version of HPX-5 can be downloaded from here.

Job Launching

Why am I seeing MPI warnings?

On many systems, HPX uses MPI to bootstrap itself, and variations in the local MPI configuration (via either a module or system install) may generate warnings about available network interfaces, memory, and process affinity. Most MPI warnings will not impact HPX operation but please consult the relevant documentation for your MPI installation to fully understand the scope of the warning message. For example, below is a common warning that occurs with recent OpenMPI versions and Mellanox HCAs:

WARNING: It appears that your OpenFabrics subsystem is configured to only allow registering part of your physical memory. This can cause MPI jobs to run with erratic performance, hang, and/or crash.

This may be caused by your OpenFabrics vendor limiting the amount of physical memory that can be registered. You should investigate the relevant Linux kernel module parameters that control how much physical memory can be registered, and increase them to allow registering all physical memory on your machine.

See this Open MPI FAQ item for more information on these Linux kernel module parameters:
http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
Local host:
Registerable memory: 32768 MiB
Total memory: 131043 MiB

Your MPI job will continue, but may be behave poorly and/or hang.

Such warnings can be resolved by following the instructions linked in the warning message itself, or by contacting the system administrator to advise on appropriate configuration or set of parameters that can suppress the messages.

How do I tell HPX-5 how many threads per locality to use?

HPX5 automatically tries to determine the number of cores that you have launched each locality on, and uses that number as the default number of worker threads to use. This is normally what you want, with one exception. HPX5 does not currently compute the right number of cores set during an aprun launch (the -d flag to aprun), so it ends up with one worker by default.

In order to run correctly, you should explicitly pass –hpx-threads=[ppn] to your application to override its default behavior. This will always work.

How do I tell HPX-5 how many cores per locality to use?

The short answer is that HPX-5 is a library, not a job launcher. We generally try to respect the job layout that you generate using your launcher, so the first thing you need to know how to do is to:

control the layout of your job from your launcher (numactl, mpirun, aprun, etc).:

$numactl -C 0,2,4 ./examples/countdown 1 –hpx-log-level=default –hpx-log-at=0
LIBHPX<0,-1>: (hpx.c:hpx_init:179) HPX running 3 worker threads on 3 cores
LIBHPX<0,-1>: (network.c:network_new:96) SMP network initialized
locality: 0, thread: 0, count: 1
locality: 0, thread: 0, count: 0
terminating.
$

$mpirun -np 1 -map-by node:PE=3 ./examples/countdown 1 –hpx-log-level=default –hpx-log-at=0
LIBHPX<0,-1>: (hpx.c:hpx_init:179) HPX running 3 worker threads on 3 cores
LIBHPX<0,-1>: (network.c:network_new:96) SMP network initialized
locality: 0, thread: 0, count: 1
locality: 0, thread: 0, count: 0
terminating.
$

aprun -n 1 -N 1 -d 3 ./examples/countdown 1 –hpx-log-level=default
LIBHPX<-1,-1>: (boot.c:boot_new:80) bootstrapped using PMI.
LIBHPX<0,-1>: (cpu.c:system_get_affinity_group_size:71) saw 1 cores from set 1
LIBHPX<0,-1>: (hpx.c:hpx_init:178) HPX running 1 worker threads on 1 cores
LIBHPX<0,-1>: (network.c:network_new:96) SMP network initialized
LIBHPX<0,0>: (worker.c:worker_start:582) worker 0 starting on core 0
locality: 0, thread: 0, count: 1
locality: 0, thread: 0, count: 0
terminating.
Application 394633 resources: utime ~0s, stime ~1s, Rss ~8668, inblocks ~327, outblocks ~41
$

 

But wait, HPX-5 didn’t doesn’t report the expected output for the aprun example!

The nitty-gritty is that HPX-5 determines the number of cores by computing the size of the current process’ thread affinity map. This works well normally, but does not work for aprun which is stingier about the affinity map. The only impact of this computation is that HPX-5 will set the wrong number of default threads for an aprun launch, so users should explicitly sepecify the –hpx-threads flag.

HPX5 does not modify the affinity of the pthreads that it spawns, so using your launcher correctly will allow you to lay out your localities as you want.

How do I know how many cores and threads per locality HPX-5 is using?

The most reliable way to know how many cores and threads each HPX-5 locality is using is to configure with –enable-debug CFLAGS=”-O0 -g” and then run your application with –hpx-log-level=default and –hpx-log-at=0 option.

It will print out some information about your run from locality 0, including the core/thread information.
$ mpirun -np 2 -map-by node:PE=4 ./examples/countdown 1 –hpx-log-level=default –hpx-log-at=0
LIBHPX<0,-1>: (hpx.c:hpx_init:179) HPX running 4 worker threads on 4 cores
LIBHPX<0,-1>: (network.c:network_new:96) PWC network initialized
locality: 0, thread: 0, count: 1
locality: 0, thread: 0, count: 0
terminating.

Known Bugs and Limitations

How do I run make check on NERSC’s Babbage?
To use make check on Babbage, set the variable TESTS_CMD when configuring HPX-5 as following:
export TESTS_CMD=’/usr/bin/timeout 20m mpirun.mic -n 2 -hostfile micfile.${SLURM_JOB_ID} -ppn 1′
When building HPX-5 applications on NERCS’s Babbage, I get errors about missing libraries. How do I fix this?

This problem is specific to Babbage and furthermore only happens when linking against a shared version of HPX5. One workaround is of course to use a static version of HPX5 instead.

Alternatively, you can set LDFLAGS as following when configuring (if using autotools) or building your application:
LDFLAGS=”-mmic -Wl,-rpath-link,/global/homes/j/jayaajay/hpx_babbage/HPX5/lib -Wl,-rpath-link,/global/babbage/nsg/opt/intel
/composerxe/composer_xe_2015.1.133/compiler/lib/mic,
-Wl,-rpath-link,/opt/intel/compilers_and_libraries_2016/linux/tbb/lib/mic”
When building HPX-5 on Cray with PrgEnv-intel (icc) I see missing header or build errors. How do I fix this?
The default versions of gcc loaded on some Cray systems with the Intel PrgEnv is outdated. Load a more recent gcc module, for example:
module load gcc/4.9.3

Check your compute cluster’s documentation for further information regarding GNU/C99/C++11 support and the available Intel programming environments.

 

 

When building HPX-5 I see errors about not finding ffi.h. How do I fix this?

This is similar to the previous question. Normally, the HPX-5 build system uses the system version of libffi, a library on which it depends, and if it cannot find a system version it uses the contrib version that is included with the HPX-5 distribution. On some systems the HPX-5 build system tries to use the system version of libffi when it should not. When building then it will give errors about finding ffi.h. This can be worked around by adding –with-libffi=contrib to the configure line when configuring HPX-5.

When building HPX-5 I see errors about not finding hwloc.h. How do I fix this?

Normally, the HPX-5 build system uses the system version of hwloc, a library on which it depends, and if it cannot find a system version it uses the contrib version that is included with the HPX-5 distribution. On some systems the HPX-5 build system tries to use the system version of hwloc when it should not.

When building then it will give errors about finding hwloc.h:
fatal error: hwloc.h: No such file or directory

This can be worked around by adding –with-hwloc=contrib to the configure line when configuring HPX-5.

HPX-5 only uses one thread per locality by default using the aprun launcher.

HPX-5 typically computes the number of cores per locality and launches one worker thread per core. Unfortunately HPX-5 does not currently compute the right number of cores set during an aprun launch (the -d flag to aprun), so it ends up with one worker by default.

In order to run correctly, you should explicitly pass –hpx-threads=[ppn] to your application to override its default behavior.

I am seeing segmentation faults and other errors when configuring with enable-jemalloc. Are there bugs in my application?

HPX-5 relies on a specific development version of jemalloc that we distribute. This is an older development version of jemalloc that has some stability issues in some applications, currently for unknown reasons. If you experience problems when using –enable-jemalloc, we recommend to try without it, or use a stable release of jemalloc using one of the techniques recommended by jemalloc developers (e.g., LD_PRELOAD): https://github.com/jemalloc/jemalloc/wiki/Getting-Started.

Are there any restrictions when running with AGAS?

Yes. Currently, AGAS can only run with the ISIR network.

Supported Systems

Does HPX-5 support execution in heterogeneous environments?

Our threads are fully POSIX-capable. HPX-5 leverages distributed GPU and co-processors (Intel Xeon Phi) through experimental OpenCL support.

Does HPX-5 support parallel debuggers

HPX-5 supports LGDB on cray machines. Detailed information is provided in the debugging section in the user guide.

Does HPX-5 support 64 bit environments?

Yes, you should also be able to use HPX-5 on 64 bit architectures and operating systems with no difficulty.

Is HPX-5 thread safe?

Yes.

What Photon HCA and backend selections does HPX-5 support?

The Photon default is to use the first detected IB device and active port. This behavior can be overridden with the following environment variable:

 export HPX_PHOTON_IBDEV=”mlx4_0″ 

 

This string also acts as a device filter. For example,

 export HPX_PHOTON_IBDEV=”mlx4_0:1+qib0:0″ 

 

will have Photon prefer device mlx40 and port 1 but will also use qib0 port 0 if detected. Device names can be retrieved with ibvdevinfo on systems with IB Verbs support. If HPXPHOTONIBDEV is set to be blank, Photon will try to automatically select the right device. (These parameters can also be set at run time using the command line option –hpx-photon-ibdev=<…>.)

The Photon default backend is also set to verbs. On BigRed2, or any other Cray system with a uGNI-supported interconnect, set the following environment variable:

 export HPX_PHOTON_BACKEND=ugni 

 

OR, use –hpx-photon-backend=ugni as a command line flag. The list of supported sizes can be obtained as follows:

 module avail craype-hugepages 
What network interconnects does HPX-5 support?

Note that if you are building with Photon, the libraries for the given network interconnect you are targeting need to be present on the build system. The two supported interconnects are InfiniBand (libibverbs and librdmacm) and Cray’s GEMINI and ARIES via uGNI (libugni). You may build with IBV and/or uGNI support on a workstation where the development packages are installed, but launching an HPX application with Photon requires that the actual network devices be present so that the library can initialize. HPX-5 2.0.0 includes experimental support for sockets and psm networking through the included libfabrics package, which may have additional dependencies on your system. Libfabric is enabled by default, but if photon detects a “native” backend, like verbs or ugni, it will use that by default only falling back to libfabric/sockets if neither is detected.

If you build with Photon and/or MPI on a system without networking, you may still use the SMP transport to run applications that are not distributed.

What bootstrap networks does HPX-5 support?

During startup HPX-5 relies on a simple bootstrap network interface to for all-to-all and all-reduce communication amongst its localities. At runtime HPX-5 relies on its high performance point-to-point network (either the PWC network running over the Photon transport or the Isend/Irecv network running over the MPI transport)the bootstrap network is neither used nor progressed at runtime.

HPX-5 supports bootstrapping with either an MPI or PMI-based network interface. Bootstrap networks are available based on the HPX-5 configuration options –enable-pmi and –enable-mpi. If both MPI and PMI are available, then HPX-5 prefers PMI, falling back to MPI if PMI initialization fails.

What network transports does HPX-5 support?

HPX-5 includes two network transport options: MPI and Photon. HPX can be built with one, both or none of the transports depending on application needs. Each transport is runtime configurable.

What allocators does HPX-5 support?

HPX-5 mainly supports jemalloc and tbbmalloc. To use TBBMalloc add –disable-jemalloc –enable-tbbmalloc in the configure option and add intel module. TBBMalloc support requires a C++11 compiler. Use TBBROOT to set the path to the root of Intel’s TBB installation. We also have experimental support to Doug Lea’s malloc for GAS allocation.

What hardware platform does HPX-5 support?

HPX-5 supports all the common platforms that the operating systems listed in the previous question support.