Compilation and Installation Tutorial¶

Assumptions¶

In this tutorial, we will go beyond the Quickstart to walk through compiling and installing QDYN into your home directory from scratch, starting from a fresh, minimal Ubuntu 20.04 installation. You should be logged in as a regular (non-root) user with sudo permissions, have a clean git checkout of the QDYN repository at ~/qdyn, and have a terminal with a bash shell open to your home directory, with 5-10 GB of free space.

Installing Compilation Prerequisites¶

System packages¶

First, we will need some prerequisites. The following are all the packages that are required to compile and install QDYN on Ubuntu:

sudo apt-get update
sudo apt-get install -y build-essential gfortran libblas-dev liblapack-dev python git wget texlive-latex-base texlive-fonts-recommended texlive-fonts-extra texlive-latex-extra

If you do not have sudo access, the system administrator will have to install these prerequisites.

Specifically, we now have:

The gfortran compiler. This is the “standard” free Fortran compiler. Using a commercial compiler such as Intel Fortran (ifort) will result in faster runtimes, but we don’t always have access to one.
The blas and lapack libraries. These are very widely used Fortran libraries for basic linear algebra. QDYN depends on a handful of third-party libraries, most of which are included as source code in the QDYN repository and will be compiled together with QDYN. Blas and Lapack are the only two external library dependencies. These two libraries are so central to scientific computing that many platforms or commercial compilers (e.g., the Intel Fortran compiler ifort) include their own implementations (e.g., the “Intel MKL”)
Gnu make (from the build-essential package). QDYN is a big library with a lot of modules. Compiling it “by hand” (run gfortran for every module with the right flags, and then archiving the compiling and linking the modules into a library a set of utilities) wouldn’t really be feasible. This is a problem that make solves: It reads rules defined in a Makefile to figure out which files need to be compiled with which options.
LaTeX, which is required for generating some parts of the documentation
QDYN’s compilation involves several Python scripts. Most importantly, there is a configure script in the root QDYN folder that we will use to “bootstrap” the installation (among other things, it generates the Makefile). Any version of Python will do.

The python package in Ubuntu 20.04 still installs the obsolete Python 2.7. This will work fine, but you could also install Python 3 with

sudo apt-get install -y python3 python-is-python3

After this, the default python executable will point to Python 3 instead of Python 2.

Conda (Miniforge)¶

While older versions of Python are OK for bootstrapping a minimal QDYN installation, for generating the documentation, running tests, or working through any of the examples, you will need a recent Python 3 and several third-party Python libraries such as numpy, scipy, and qutip.

Warning

Never install third-party libraries into your system Python. Always use a separate Python installation and/or some sort of virtual environment.

Here, we will install Miniforge into our home directory, which provides the conda package manager. Conda, in turn, can install many versions of Python, “scientific” Python packages, and other tools. Having it in the home directory provides isolation from the rest of the system. Also, in general, it is an excellent way to install an up-to-date Python on a system where you don’t have root/sudo access. However, it will take up 3-5 GB of space in your home directory.

wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
sh Miniforge3-Linux-x86_64.sh -b

The -b is for “batch mode”. You can leave it out for an interactive installation.

After the installation finishes, you can remove the installer

rm Miniforge3-Linux-x86_64.sh

Note

The conda package manager could also be installed via the official Anaconda distribution (a full conda-based Python distribution managed by Anaconda Inc.) or Miniconda (a more minimal installer for Anaconda).

Miniforge is similar to Miniconda, but it uses the community-managed conda-forge as the default “channel” (the package repository) instead of the channel managed by Anaconda Inc that is the default for Anaconda/Miniconda. Some of the Python packages that help with working with QDYN, most notably QuTiP, are only available on the conda-forge channel. Since mixing packages from the default Anaconda and the conda-forge channels can cause some problems, Miniforge is a good option, with a better default configuration and support for more platforms.

.bashrc¶

We need to configure bash to be able to find the conda executable and the default conda environment, which provides our “default” Python executable.

./miniforge3/bin/conda init bash

which will modify your .bashrc file to set the $PATH (an environment variable with a list of folders where bash searches for executables when your run a command). If you used the interactive Miniforge installer, it might have already asked you to do this initialization.

You will have to restart your terminal for these settings to take effect. Alternatively, to initialize conda once, in the current shell, run

eval "$(~/miniforge3/bin/conda shell.bash hook)"

Note

Conda modifies your command line prompt to show the name of the currently active conda environment. If you find this annoying, you can deactivate it.

You should also make sure that the ~/bin directory is on your $PATH, by adding

export PATH=$HOME/bin:$PATH

at the bottom of your .bashrc. Again, you will have to restart your terminal, or type the above command into your current shell.

The ~/bin directory is generally a good location for user scripts or simple binaries, and it is also the location where we will install the QDYN utilities, see below.

QDYN Installation¶

Having installed all the prerequisites for compiling QDYN, we now continue with the actual compilation and installation of QDYN into the home directory, as the basis for running any of the QDYN utilities or for developing Fortran programs that use QDYN.

We assume a clean git-checkout of the QDYN repository in the ~/qdyn folder.

cd qdyn

Supported Fortran Compilers¶

QDYN is routinely used with and regularly tested against the following Fortran compilers:

ifort 19.1.2
ifort 14.0.3
ifort 12.0.0
GNU Fortran 10.2.1
GNU Fortran 8.3.0
GNU Fortran 7.5.0
GNU Fortran 4.8.5
NAG Fortran Compiler Release 6.1(Tozai)

configure¶

Inside the qdyn folder, you will see a configure script. This is what “bootstraps” the compilation of QDYN and controls compiler options and where the compiled library should be installed. Running it without arguments,

./configure

prepares the compilation of QDYN with default options.

The script will also create a conda environment in the ./venv subfolder that contains a recent version of Python 3 and all the third-party libraries and tools required for developing, testing, and installing QDYN. Even if we don’t do any QDYN development, we still need this for generating the documentation.

The configure script will print a summary at the end of its run. In our case, this will look something like this:

PREFIX         : /home/user
LIB_INSTALL_DIR: /home/user/lib/qdyn
MOD_INSTALL_DIR: /home/user/lib/qdyn/mod
BIN_INSTALL_DIR: /home/user/bin
DOC_INSTALL_DIR: /home/user/lib/qdyn/doc
DAT_INSTALL_DIR: /home/user/lib/qdyn
VENV_PYTHON: ./venv/bin/python
PREP           :
FC             : gfortran
FFLAGS         : -O2 -fopenmp -ffree-line-length-0
LIBS           : -lqdyn -llapack -ldfftpack -llbfgsb -larpack -lblas -lwigxjpf -lfastwigxj
FEATURES       : no-check-cheby, no-check-newton, no-parallel-ham, parallel-oct, backtraces, no-debug, no-no-ipo
EXTLIBFFLAGS   : -O2 -fopenmp -ffree-line-length-0
TEST COMMAND   : $(VENV_PYTHON) -m pytest --tb=short --maxfail=10 --durations=5 --doctest-modules --cov-config=qdynpylib.coveragerc --cov=qdyn --cov-report= --cov-append -n auto -m "not multicore" ./tests

The *_DIR folders are the locations where the compiled QDYN will be installed. By default, these are all subdirectories of PREFIX, which by default is your home directory (user is your username in this example). You can change the default prefix by setting a $PREFIX environment variable in your .bashrc, e.g., to $HOME/local if you want to have an extra layer of separation.

The VENV_PYTHON is the conda/Python environment that will be used for QDYN development (like the TEST COMMAND shown above). In this case, it is the environment in the local venv folder that the configure script set up.

FC is the Fortran compiler we will be using, and FFLAGS and LIBS are options for the compiler. QDYN also includes several third-party libraries such as arpack that are included with the QDYN source code, in the ./ext_libs/ folder. These will be compiled with the EXTLIBFFLAGS options.

The FEATURES select particular features of the QDYN library (based on precompiler flags). In this case, the flags mostly indicate that we’re compiling QDYN without various check and debug features that would help with debugging at the expense of running more slowly.

All of these settings can be changed by calling configure with command-line options. Have a look at

./configure --help

to see what is available.

make¶

One of the things that running configure did was to generate a Makefile.

Thus, we can now run make to compile the QDYN library.

make -j4

The -j4 flag is optional but speeds up the compilation by running up to 4 processes in parallel.

Once the compilation finishes, you will see a file libqdyn.a in the QDYN root folder, as well as *.a files for all the “external” libraries that are included with Fortran, e.g., libarpack.a.

Warning

You cannot run make before running ./configure

make install¶

We install QDYN into the home directory with

make install

This first performs some additional compilations, specifically of the Utilities in the ./utils subfolder and of the QDYN documentation in ./doc/_build.

Finally, all of the compiled files are copied into the folders selected by the configure script. That is, with the default settings, you will find e.g. qdyn_prop_traj and qdyn_optimize in ~/bin, libqdyn.a (and the other *.a files) in ~/lib/qdyn/, and the documentation, mod-files and (unit conversion) data files in specific sub-folders of ~/lib/qdyn/.

Note

Run make help to see a documented list of available make targets. For example, make doc will generate the documentation and make uninstall will undo the effects of make install.

make clean¶

After the compiled QDYN has been installed, we can clean up the checkout folder. Running

make clean

will remove all the “compilation artifacts”, such as the compiles .o files for each module and the .a library files.

Running

make distclean

goes one step further to remove the files generated by the ./configure script. This includes the ./venv folder with the Python environment used for development and the Makefile. You will have to re-execute ./configure before you can run make again.

Verifying the Installation¶

We test the success of the installation by running

qdyn_version

The qdyn_version utility was installed in the ~/bin folder, so the above will only work if the ~/bin/ is included in the $PATH environment variable in the .bashrc.

The output of qdyn_version will show the git commit hash of the compiled QDYN version, the compilation options, and the compilation date and time.

You are now ready to use the installed QDYN library in your own programs.

For more information on how to install QDYN on different systems or with different options, see the Compilation and Installation How-To.

Setting Up Jupyter¶

The Jupyter notebook server¶

Jupyter notebooks are a powerful environment in which to work with QDYN. Notebooks are web-based documents that interleave markdown text and code cells. The output of code cells, e.g., visualizations created with matplotlib, are shown within the notebook. The Python scientific ecosystem makes it easy to perform pre-processing and data analysis. QDYN also comes with “QDYN-pylib”, a library to enable Python workflows around QDYN.

Of course, using Python or notebooks is entirely optional: If you don’t know Python, you can stick to writing Fortran programs on top of QDYN. If you do decide to use Jupyter notebooks, install the jupyter program and related packages into the “base” conda environment we set up earlier:

conda install -n base -y ipython pandoc matplotlib black jupyterlab jupyter_contrib_nbextensions nbdime jupytext ipyparallel ipywidgets jupyterlab_code_formatter

The QDYN project kernel¶

In Jupyter’s architecture, code cells get send to an IPython “kernel” that is entirely separate from the jupyter process. In fact, the kernel can be in a completely separate environment. Here, we will create a “QDYN” project kernel in an isolated conda environment.

From inside ~/qdyn, run

conda env create --name QDYN --file ./doc/sources/examples/environment.yml

This creates an new environment based on the the following environment.yml file:

environment.yml¶

name: QDYN
channels:
    - conda-forge
dependencies:
    - python=3.8
    - pip
    - black
    - isort
    - numpy
    - matplotlib
    - scipy
    - sympy
    - ipython
    - ipykernel
    - ipywidgets
    - pandas
    - pandoc
    - qutip
    - pip:
        - weylchamber
        - krotov

You could create at copy of this file at an arbitrary location, edit it to your needs, and use it with conda env create for a custom setup.

Once the environment has been created, we can register it as a kernel for Jupyter:

conda run --name QDYN python -m ipykernel install --user --name QDYN

This makes the ipykernel from the QDYN conda environment available to any Jupyter server: we will be able to run jupyter notebook or jupyter lab from the base environment and select a “QDYN” kernel. The code cells in the notebook will then run inside the QDYN environment, with access to all the packages in that environment, without the environment having to be activated.

You can check that the installation of the kernel succeeded with

jupyter kernelspec list