Building ASCAR Pilot

Note

Pilot is currently in the alpha test stage. It may crash or produce wrong results even we are trying very hard to avoid these problems by writing extensive test suits and testing it in as many situations as possible. If you suspect something is wrong please don’t hesitate to let us know through Slack or users mailing list.

Note

This section is intended for developers or those users who have a need to compile Pilot from source code. Most regular users just need to install our official Pilot binary, see Install Pilot.

Supported OSes

Our current roadmap is to support the latest versions of the popular flavors of UNIX/Linux, including Mac OS X. If you need to run Pilot on Windows or other embedded systems, please send us an email on the users mailing list.

Currently we support Pilot on the following operating systems:

  1. Red Hat Enterprise Linux / CentOS 7 (gcc 4.8.5 and gcc 7.2.1 from devtoolset-7)

The following OSes are supported on a best effort basis:

  1. Mac OS X Version 10.11 El Capitan (Apple LLVM version 7.3.0 (clang-703.0.29))
  2. Ubuntu 16.04 LTS x86-64 (gcc 5.3.1-14ubuntu2)

Pilot should work on most Linux systems that have a C++ compiler that can compile Boost. Users have reported successful build on Arch and Fedora. Please feel free to send an email to the users mailing list to share your experience if you have tested Pilot on other systems.

Requirements

Pilot requires a modern C++ compiler, CMake, ncurses, and the boost C++ library. Python development library is needed if you need to build Python support.

It also uses libcdk for the text UI and gtest (Google Test) for running test cases. Both libraries will be automatically downloaded and compiled when you run make for the first time.

Additionally, if you want to plot the results from some sample programs, you will need Python 2.* and matplotlib.

Set Up Your System

Red Hat Enterprise Linux / CentOS 7

You can set up the development environment manually, using Vagrant, or using Ansible. Setup Your Development Environment Using Ansible on CentOS 7 is easy and fast, and that’s what our developers use to set up our own development machines. But please be warned that the Ansible playbook alters your system by installing RPM packages. It also installs latest version Boost to /opt, which may not be what you like. It should be used with care in complex environments.

If you don’t have a dedicated host, you can easily develop or build Pilot in a VM using Vagrant (see Setup Your Development Environment Using Vagrant on CentOS 7).

Finally for best control, you can set up your development environment manually (see Manually Setup Development Environment on CentOS 7), it is not hard and shouldn’t take more than 30 minutes.

Setup Your Development Environment Using Ansible on CentOS 7

To use Ansible, you should add your host to Ansible’s inventory file and run:

ansible-playbook -l YOUR_HOST build-env/centos7-x64/bootstrap-pilot-builder.yaml

Please refer to Ansible’s documentation for Ansible-specific instructions.

Setup Your Development Environment Using Vagrant on CentOS 7

Vagrant is a good tool for quickly setting up a development environment in a virtual machine:

cd build-env/centos7-x64
vagrant up

Our Vagrantfile by default uses a large amount of memory and many CPUs (we use it for fast nightly build). Feel free to trim it down to meet your machine’s configuration by tweaking the v.memory and v.cpus line in the Vagrantfile.

After that you can just vagrant ssh into the VM and clone Pilot from source, install your tools, and do development or build.

Manually Setup Development Environment on CentOS 7

First enable the EPEL and SCL repository, and install development tools:

yum install epel-release centos-release-scl
yum install bc cmake3 gcc-c++ git graphviz ncurses-devel lua-devel readline-devel patch

Next we need to install a recent version of Boost (RHEL7 shipped with old Boost 1.53.0). Download the latest Boost source and unpack it to, say, ~/boost_1_61_0.

cd ~/boost_1_61_0; ./bootstrap.sh --prefix=/usr/local/
./b2 -j4 variant=release link=static install
sudo ./b2 install

Optional packages for compiling documentation, doing tests, and running evaluation scripts: doxygen valgrind numpy python-matplotlib

Mac OS X

  1. For Mac, make sure you have the latest Xcode and command line development tools installed.
  2. Install the required tools. If you are using MacPort you can install the following packages:
sudo port install cmake boost graphviz ncurses lua readline

The following Python packages are not needed for the function of Pilot, but they are needed if you’d like to run the test and evaluation scripts in the lib/test directory: numpy matplotlib.

The Boost library from MacPort does not build some static libraries. So if you plan to build a statically linked pilot binary (e.g. for packaging and distributing to other Mac users), please install the boost C++ library from the source. The following steps are tested with [Boost 1.59.0](http://www.boost.org/users/history/version_1_59_0.html):

  1. download the latest Boost source and unpack it to, say, ~/boost_1_59_0
  2. cd ~/boost_1_59_0; ./bootstrap.sh --prefix=/usr/local/
  3. ./b2 -j4 variant=release link=static
  4. sudo ./b2 install

Ubuntu

Install the following packages:

sudo apt-get install git cmake libboost-all-dev g++ graphviz libncurses5-dev liblua5.3-dev libreadline-dev

Optional packages for compiling documentation, doing tests, and running evaluation scripts: doxygen python-matplotlib valgrind python-numpy. You can also install dh-make if you need to build the deb package.

Building Pilot

mkdir build
cd build
cmake ..
make

And you can run test cases:

ctest -R unit_test

The above command only runs unit tests and should be fast.

or run all test cases:

make test

Some of the lengthy functional test cases need a few minutes and a couple of GB disk space in /tmp.

You can add -DCMAKE_BUILD_TYPE=Debug to step 3 if you need to disable the compiler optimization for debugging purposes. The default build type is RelWithDebInfo, which enables both optimization and debug info.

If you need to use a specific boost installation, add -DBOOST_ROOT=/path/to/install/boost to step 3.

If you need to build statically linked libraries, which will not require libboost libraries, add -DBoost_USE_STATIC_LIBS=ON to step 3.

Build Options

The following are configuration options that are supported by Pilot’s CMake builder. To set an option, use: -DOPT=VAL. For example, on Linux (CentOS 7) to use the system’s Python 2.7 you can set -DWITH_PYTHON=ON -DPYTHON_LIBRARY=/usr/lib64/libpython2.7.so -DPYTHON_INCLUDE_DIR=/usr/include/python2.7. If you are using Python 3 from CentOS Software Collection, you can set -DWITH_PYTHON=ON -DPYTHON_LIBRARY=/opt/python3.5/lib/libpython3.5.so -DPYTHON_INCLUDE_DIR=/opt/python3.5/inc. On macOS you can set -DWITH_PYTHON=ON -DPYTHON_LIBRARY=/System/Library/Frameworks/Python.framework/Versions/2.7/Python -DPYTHON_INCLUDE_DIR=/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7.

Option Description Default Introduced
CMAKE_BUILD_TYPE Build type RelWithDebInfo 0.1
WITH_PYTHON Build Python binding (you may also want to set the following three options) [1] OFF 0.2
PYTHON_LIBRARY The path of the libpython library Auto detect 0.2
PYTHON_INCLUDE_DIR The path of pyconfig.h [2] Auto detect 0.2
BOOST_PYTHON_MODULE The name of the Boost.Python module [3] python 0.2

Footnotes

[1]To enable building the Pilot Python binding, Boost.Python and libpython are needed and their versions have to match (e.g., Boost.Python3 has to be used if you use libpython3.x). Pilot uses the python interpreter as designated by shell PATH, and tries to find the location of libboost-python.so and libpython automatically. If the detection fails or if the detected components have mismatched versions or if you want to specify the libraries to use, set the three options that follows in the table.
[2]Pilot uses CMake’s find_library to search for libpython automatically. If you have multiple Python instances on your system you can set these two PYTHON_* options and point them to the correct libpython file and include path. Note that PYTHON_LIBRARY has to point to the actual dynamic library while PYTHON_INCLUDE_DIR points to a path. You also need to make sure that running python from the command line invokes the Python interpreter of the same version of your libpython; importing a module that is linked to a different version of libpython would lead to crash. You can do so by tweaking the order of directories in PATH or using a Python virtualenv. Refer to the documentation of your shell and Python documentation for details.
[3]This name specifics what Boost.Python library Pilot should link to. The default value is python which means the library called libboost-python.so should be used. On Ubuntu Linux libboost-python.so is a symlink to either libboost-python-py2*.so or libboost-python-py3*.so. This library has to match the version of libpython and Python interpreter you want to use. For instance, you need to set it to python-py35 if you want to build the Python binding for Python 3.5 on Ubuntu 16.04 LTS; you need to set it to python3 if you are using Python 3.* and a vanilla Boost library.

FAQ

Error: undefined reference to boost::python::detail::init_module()

Python 2 and 3 use different init_module(). If you see a link error message like this, it is likely that your Boost library is compiled to use a different version of Python than the libpython Pilot is trying to link to.