Motivation

In today's world, numerous everyday systems are controlled and monitored remotely by electronic devices. Sensors and components in cars communicate with a central computer that runs diagnostics and controls power steering, torque in the wheels, and other functions.  Rocket engines measure pressure and temperature, and run control algorithms that tune the engine to reach a target output force. Smart doors in buildings, smart HVAC units, all have sensors and control interfaces that communicate to a central computer or diagnostic system.

There are a few software programs that can control and monitor a complex system with sensors and control interfaces. Software like Ignition, which is a SCADA software (Supervisory Control And Data Acquisition), is commonly used in industrial applications as it's designed to work with common industrial protocols such as MODBUS or PROFIBUS, such as in factories or power plants. Another popular choice, LabVIEW, is commonly used in research lab settings for its ability to acquire large amounts of data and the fact that it's designed to interface directly with National Instruments systems.

These programs, however, all have their shortcomings. 

I witnessed the downsides of the modern data acquisition and control programs listed above firsthand while on the Texas A&M Sounding Rocketry Team. As a member of GSE (Ground Support Equipment) for 2 years, and then as the manager for my 3rd year, I noticed time and resources being wasted on trying to re-learn the LabView and custom driver systems that previous years had built, and the entire projects eventually became abandoned. 

People would then build their own data acquisition and control software, with a custom graphical user interface (GUI) and backend control, or use a Data Acquisition software, and a project-specific hardware control box, or a mix of both. 

After a year spent building the custom software, coding the GUI, and getting it to work properly with the current control and data acquisition systems, if the project changed at all, the entire GUI would need to be re-coded, and more drivers would need to be written and in extreme cases, the entire codebase would need to be re-designed to accept data and send commands to systems it was not entirely designed for.

If specific control hardware was built, ie, a box with switches and a button to actuate certain valves, the hardware would be abandoned as soon as the project changed or the person who built it left and could not troubleshoot it anymore.

Either way, it generally turned out to be a mess, with either the software not being easy to add to or change, or controls and data acquisition systems being separated, or time being wasted on rebuilding systems every year, when the project changes, or whoever built it leaves.

What is DARTWIC?

DARTWIC (Data Acquisition for Rocket Testing With Integrated Controls) was built to address all the drawbacks mentioned above. It is a lightweight program, written in C++ for speed, has a convenient GUI that requires no coding to visualize system data and controls, and is built with modularity and flexibility in mind, by completely exposing the memory of the program in C++ functions and loops, and by exposing a network API, so you can decide how data is gathered and controlled.

Since all real-time data is stored in memory, data throughput is massive, on the order of 12 million operations per second (with 8 threads). Because the program is thread-safe, you can run chunks of code in parallel, and linearly increase throughput as you increase threads (as computer hardware allows).

Software Architecture

While I will refrain from going into too much detail into the specific architecture of each component, below is a summary of what each component does and its purpose in the overall architecture.

DARTWIC is a monorepo, containing a few different software projects put together to create a complete controls and data acquisition platform.

• Engine: The main project that combines RAPID, CAESAR, and TEMPEST to create a unified controls and data acquisition platform.

• Interface: The user interface that is used to interact with DARTWIC. Written in ReactJS. Contains a frontend web server UI and a desktop app that is built using Electron.

• RAPID: Real-time Access Parallel In-memory Database. A database that is highly performant in multi-threaded environments, focused on data acquisition, automation, and control applications. DARTWIC uses this to store data and commands called "channels", directly in memory for fastests realtime access, and records historical data to disk. Channels have many different features, such as mapped channels that propagate their value to, scaling and offset values for converting sensor units to engineering units, buffer size settings to calculate rolling averages and standard deviations, and much more. 

• CAESAR: Command Authority Engine for Sensitive Automated Resources. Commanding engine for DARTWIC. This contains the threaded loops that users can create in DARTWIC, and will also later implement command authority triplication by allowing multiple RAPID or DARTWIC nodes to be triplicated, and voting on the final command in the threaded loops (planned).

• TEMPEST: Transaction & Event Messaging Platform for Encrypted System Telemetry. A message router that routes all data and commands to and from the DARTWIC node. Client libraries, TAARS, and the interface app will use this to connect to a DARTWIC node.

• TAARS (PLANNED): Telemetry AI Assistant for Remote Systems. An AI assistant that connects to a DARTWIC node as a client, and can perform certain actions, such as commanding channels, analyzing historical data, etc.

• ARGUS (PLANNED): Automated Recording of General User and System Events. An event logger & log storage engine that records all notable events in DARTWIC, and provides an interface for storing lower-rate data than RAPID deals with (for example, console messages). Designed to log all operations that pass through TEMPEST, process-level messages (like print streams to console), and to store Console Messages.

• SDK: Software Development Kits for embedding functionality into DARTWIC. Includes creating additional user interface components via the ReactJS SDK (planned), or C++ device drivers via the Modules SDK's.

• Modules: A collection of device drivers and user interface components created using the DARTWIC SDK.

• Clients: Client libraries for connecting to a DARTWIC node. Planned libraries include Python, C++, and ReactJS.

DARTWIC is best described as a test and mission operations software. Think a combination of InfluxDB for data storage, and Grafana/ Ignition for visualization, all run by an engine built around real-time control and data acquisition applications, written in C++.

DARTWIC's superpower is that it is designed around a server-client architecture. The intention is to allow for extreme system scaling, as traditional databases do. 

Have too many sensors to gather data from? Add another DARTWIC node to your network. Want to triplicate your control algorithms? Add 2 more DARTWIC nodes to your system, link them, and define voting criteria.

DARTWIC is also intended to support multiple operating systems and applications. Deploy a ground station on a Windows machine with an asynchronous RAPID database for the highest possible throughput, or deploy a flight software node on a real-time Linux system with a synchronous RAPID database for real-time and deterministic control.

 Lastly, DARTWIC is designed to be interfaced from anywhere. Want to use the built-in Interface Webserver app? No problem. Want to build your own application instead? Just use DARTWIC's client libraries to send and receive data/ events. 

Applications can span from Aerospace Test to Satellite Mission Operations to Industrial Automation to Spacecraft Control.

Interface

The Interface App is a ReactJS project that connects to DARTWIC via the TEMPEST client library and presents a human interface for interacting with a DARTWIC system. Below, I will show some cool features of the app. **Some photos are slightly outdated as the interface layout is always improving over time

Database Searcher

Below is the Database Searcher, which shows the channels in the database central to DARTWIC. The user can search, update, or view any channel data in the program in real time.

Schematics

A schematic is an infinite canvas where you can visualize systems using "nodes". Nodes are visual components that can be dragged and dropped onto the canvas and configured to display certain data, command a channel once clicked, etc.

 Current schematic nodes include graphs, parameter displays, control displays, buttons, sliders, switches, tanks, valves, etc. 

A very useful feature, as you can generate graphical user interfaces that visualize entire rocket systems in little to no time at all, with no programming.

Automation Scripting

One of the best features in DARTWIC is the integrated scripting. With this, the user can create automation scripts in real time, without having to build any object code, or restart the server. 

The scripting is python based with a parser to inject custom syntax. 

The script below controls an autofill sequence used to fill a rocket engine run tank.

The user can create control loops, specify the target frequency of the loop, start or stop the loops, request a test operator prompt, hold the script, abort, send messages to the console, etc.

Event Console

DARTWIC also has an event console. It is used to display statuses and important messages to all connected users. 

The console exposes errors, warnings, aborts, and info messages, which are basic forms of messages to display statuses to the user.

  The console also exposes functional events, which include holds and prompts. When a hold is called in code, the program waits at the executed line. The user then needs to release the hold to continue the program. Prompts function exactly as holds, except they require a value to be submitted to release, and upon release, the value is passed to the line that triggered the prompt.

  Console Event statuses are automatically propagated to the schematic node that uses the channel (tag - I am in the process of changing the naming convention)  that exists in the event message.

Resources & Interface Layout

The interface provides an easy way to view resources stored on the server. The program has a sidetab that shows the resource categories, and upon clicking, reveals the file structure and files. Different resources can be opened, renamed, deleted, created, etc., all from the client.

 The interface also allows multiple resources to be displayed at once.

Module System

To develop custom device drivers that are embedded into the DARTWIC process, the user can use the Module Software Development Kit.

 The module SDK provides a C++ interface for writing custom C++ code to interact with any device. An example below shows a custom MODBUS TCP client module that we use to interact with MODBUS devices.

Upon creating a module, the user needs to put their build module project into the modules registry on the DARTWIC server. They then will be able to select that module type when creating a new module instance on the interface app.

Once they create the module instance, they can configure parameters to build the module. Once done, they can save the config, and a new module instance is built.

The UI shown when creating a module is easily configurable using a YAML file when shipping the module.

Below is an example of a custom module. DARTWIC exposes an API, that exposes some basic functions for interacting with RAPID channels, Console Events, and CAESAR control loops.

In the constructor of MyModule, the "device_name" parameter is retrieved and printed to the terminal, a loop is created, and in the loop, a channel is queried and printed.

Client Libraries

For outside programs to interface with DARTWIC, client libraries have been created. These connect to TEMPEST, which is the communication platform that powers DARTWIC.

Client libraries are a work in progress, but TEMPEST exposes a very basic client for receiving telemetry, and sending "operations" which are commands bound to internal DARTWIC processes.

An example below shows a TempestClient being created using the bridge client package. This is used in frontend applications to communicate with the bridge server, on backend servers, which then communicates to TEMPEST. A bridge is needed because TEMPEST communicates in Raw TCP. 

However, TEMPEST is built around a protocol-agnostic core, and web safe versions of communication will be added later.

Using DARTWIC

Now that we have described the motivation,  purpose, and architecture of DARTWIC, I can show you some awesome things I have built with DARTWIC that is not possible with any of the traditional applications (at least not in the time it took to build them)

Below is a project that took me about a day to build. It is a comprehensive testing suite for testing a hybrid rocket engine.

It contains:

• 1000Hz Data collection and control via LabJack T7 ADC

• 60Hz data collection via a custom arduino scale running a MODBUS server

• Schematics to view all systems (LabJack, Pad Scale, Engine and ground fluids)

• Control interface with autofill panel, and ignition sequence panel

  • Autofill prompts user for fill target.

• Mission clock with mission states triggered at certain times

  • Autofill triggered at T-20, and Ignition sequence at T-0:40

• Error and warning messages for out-of-bounds sensors or disconnected devices via Console Events

• Safety checks for accidental ignition triggering, or fill triggering, and panel armings.

Conclusion

DARTWIC has been motivated by the issues and limitations of old, outdated software in today's industrial automation and testing industry. I have used DARTWIC in practice, and it saves months of software development by introducing a unified control and data acquisition platform, eliminating the need of having to build testing and mission operations systems from scratch.