Current projects

art advanced i/o with HDF5

Jan 2016 through October 2017

All High Energy Physics (HEP) experiments rely on event processing software systems to manage algorithms and data from detectors and simulations. The purpose of this research project is to produce a prototype software system for HEP event processing that demonstrates an architecture and a design that will be capable of scaling to greater than 100K cores and efficiently moving event data through these new high-performance computing platforms in forms necessary for algorithmic work. We propose moving to a distributed memory, multi-process design to gain back memory space across the application, and to eliminate all I/O to physical storage, except at specialized aggregation points. The primary objective of this project is to produce a prototype software system suitable for moving HEP experiment event data through multiple processing stages in an exascale-class computing facility. In particular, work will concentrate on demonstration of two critical components of a complete event-processing system: (a) high-performance I/O to a parallel filesystem, and (b) communication of event data through high-performance node interconnects (rather than through the filesystem), between processes of one application running across hundreds or thousands of nodes—a distributed program. Thus the research plan is organized into three parts:

  1. Production of a proof-of-concept I/O implementation for a selection of realistic HEP simulation and experimental data.
  2. Production of a proof-of-concept implementation for a distributed event data product store, suitable for handling these same data on high-core-count, low-memory-per-core computing hardware.
  3. Studying the scaling performance of the resulting systems, iterating on the architecture and design choices to optimize the system.


October 2015 – present

The overall goal of this project is to enable a portion of the g-2 simulation with art to run on Mira at ALCF and on future HPC platforms. Within the context of muon g-2, there will be several simulation processing scenarios that can be usefully implemented on HPC machines. Running any of them on HPC resources can help muon g-2 complete important computing tasks from now until the time of detector commissioning.

art multi-threading


The goal of this project is to enhance the art framework to allow multithreaded operation such that multiple events can be processed concurrently, allowing saving of memory by sharing non-event data between threads. We will share effort with the CMSSW developers, to leverage the progress they have made in doing the equivalent for CMS. This work includes providing new framework module types that allow module-by-module migration to the new system, as well as providing guidance to experiments on how to convert their modules and algorithm code to be thread-friendly.

CMS multi-threaded framework

 2010 – 2017

To scale CMS HEP data processing to effectively use the growing number of CPU cores per machine requires a change of the CMS data processing framework from using only one CPU core per process to multiple cores. This project has three parts. The first part is to explore different C++ libraries to allow concurrent processing of data and evaluate their use for constructing a HEP data processing framework. The second part is to use a chosen technology, in this case Intel’s Thread Building Blocks, as the basis for re-engineering the existing CMS data processing framework to utilize many CPU cores in one process. The third part is to modify existing CMS code, that is executed by the framework, to be efficient when using multiple CPU cores. This third part also entails developing tools to find thread-safety problems and inefficiencies as well as educating the CMS developers on best coding practices when developing code in a multi-core environment.

Big Data

2016 – 2017

The aim of our big data project is to understand the role of big data technologies such as Spark and others on HPC platforms for HEP data processing tasks (non-traditional HPC), and to define the role of incorporating exascale-capable visualization tools for algorithm development and visual debugging. Our HEP use cases will be from CMS and LArTPC experiments. The goals of our project are:

  • Bring HPC and exascale tools (e.g. Paraview for visualization and HDF5 as data format) into big data technologies.
  • Work with science drivers from the CMS and LArTPC based experiments.

We are currently working with the CMS big data science project as our first use case; we have provided tools for converting CMS bacon files (in ROOT) to the HDF5 format, and setup to load and analyze the converted data using Spark on NERSC (Edison, Cori). Our code and recent talks are available in a GitHub repository.

BigData Express

1/2016 – 6/2017

BigData Express aims to provide Scheduable, Predictable, and High-performance data transfer service for DOE large-scale science computing facilities (LCF, NERSC, US-LHC computing facilities, etc.) and collaborators. This project is funded by DOE ASCR Network Research program.

Containers using Docker and Shifter


Over the past few years, container technology has become increasingly promising as a means to seamlessly make our software available across a wider range of platforms. Docker containers provide a way to “wrap up a piece of software in a complete filesystem that contains everything it needs to run”. In combination with Shifter, such containers provide a way to run software developed and deployed on “typical” HEP platforms (such as SLF 6, in common use at Fermilab and on OSG platforms) on HPC facilities at NERSC. Providing distributions through container technology looks to be an excellent way to download and run from a complete and preconfigured software distribution. We already need to provide two external groups with a way to confnigure and run our software to produce simulated LAr detector data. Our current plan is to provide these groups with containers to generate the data they need.


2013 – 2018

GeantV is a R&D partnership with CERN incorporating Intel Xeon Phi, GPU, and multi-core with plans for integration with experiment frameworks such as CMSSW and art. GeantV’s aim is to develop a community supported, open-source, next generation particle transport code for HEP integrating both detailed and fast simulation physics models and transport algorithms, optimised for the emerging parallel and vector architectures and is an opportunity for a partnership between Particle Physics and Scientific Computing. In the next years we are expecting an increase on one side of the needs for detector simulation, and on the other in the parallelism of the hardware, widening the gap between the needs and the available means. GeantV’s main purpose is to sustainably reduce this gap.

Detector simulation is one of the most CPU intensive tasks in modern HEP.  While its importance for the design of the detector and the estimation of the efficiency is ever increasing, the amount of events that can be simulated is often constrained by the available computing resources. For example, simulation helps HEP experiments design the optimal detector, for the best physics at a given cost. More information is available at:

MDTM project 

September 2015 – current

Existing data movement tools are still bound by major inefficiencies when running on multicore systems. To address these inefficiencies and limitations, DOE’s Advanced Scientific Computing Research (ASCR) office has funded Fermilab to work on the Multicore-Aware Data Transfer Middleware (MDTM) project. MDTM aims to accelerate data movement toolkits on multicore systems.

ParaView advanced visualization

2016 – 2017

ParaView is a visualization framework developed in collaboration as a community project. Common elements in visualization applications include: display event data with annotations, display detector in background and interact with display. ParaView is a scientific visualization application with rich capabilities. Argonne has a visualization group that makes heavy use of ParaView.



Event generators, such as Pythia, are a critical component of interpreting data at the energy and intensity frontiers. Many parameters are needed as an input to describe the complex structure of particle collisions. The goal of this project is to exploit the computing power of Mira to perform an extensive parameter scan to maximize the agreement between benchmark data and the generator predictions.

WireCAP project


WireCAP is a novel packet capture engine for commodity network interface cards (NICs) in high-speed networks. It provides lossless zero-copy packet capture and delivery services by exploiting multi-queue NICs and multicore architectures. In addition, WireCAP implements a packet transmit function that allows captured packets to be forwarded, potentially after being analyzed in flight. Thus, WireCAP can be used to support middlebox-type applications. WireCAP now supports Intel ixgbe-based 10GE NICs. WireCAP on Intel 40GE NICs should be released very soon. 

For more information on any of these projects, please contact the CS researcher email list at